Skip to main navigation Skip to search Skip to main content

Deobfuscation of virtualization-obfuscated code through symbolic execution and compilation optimization

  • Mingyue Liang
  • , Zhoujun Li*
  • , Qiang Zeng
  • , Zhejun Fang
  • *Corresponding author for this work
  • Beihang University
  • Temple University
  • CNCERT/CC

Research output: Chapter in Book/Report/Conference proceedingConference contributionpeer-review

Abstract

Virtualization-obfuscation replaces native code in a binary with semantically equivalent and self-defined bytecode, which, upon execution, is interpreted by a custom virtual machine. It makes the code very difficult to analyze and is thus widely used in malware. How to deobfuscate such virtualization obfuscated code has been an important and challenging problem. We approach the problem from an innovative perspective by transforming it into a compilation optimization problem, and propose a novel technique that combines trace analysis, symbolic execution and compilation optimization to defeat virtualization obfuscation. We implement a prototype system and evaluate it against popular virtualization obfuscators; the results demonstrate that our method is effective in deobfuscation of virtualization-obfuscated code.

Original languageEnglish
Title of host publicationInformation and Communications Security - 19th International Conference, ICICS 2017, Proceedings
EditorsSihan Qing, Dongmei Liu, Chris Mitchell, Liqun Chen
PublisherSpringer Verlag
Pages313-324
Number of pages12
ISBN (Print)9783319894997
DOIs
StatePublished - 2018
Event19th International Conference on Information and Communications Security, ICICS 2017 - Beijing, China
Duration: 6 Dec 20178 Dec 2017

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume10631 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349

Conference

Conference19th International Conference on Information and Communications Security, ICICS 2017
Country/TerritoryChina
CityBeijing
Period6/12/178/12/17

Keywords

  • Compilation optimization
  • Deobfuscation
  • Symbolic execution
  • Virtualization obfuscation

Fingerprint

Dive into the research topics of 'Deobfuscation of virtualization-obfuscated code through symbolic execution and compilation optimization'. Together they form a unique fingerprint.

Cite this