Divergence detector: A fine-grained approach to detecting VM-awareness malware

Chia Wei Hsu, Fan Syunshih, Chi Weiwang, Shiuh-Pyng Shieh

Research output: Contribution to conferencePaper

3 Scopus citations

Abstract

Virtualized execution has become an effective mechanism to analyze malware in a dynamic way. To conceal its malicious behaviors, VM-aware malware probes the execution environment for analysis-resistance. These malware programs hide their malicious behaviors if they are launched in a virtual machine (VM). VM awareness becomes a barrier for malware analysis due to the concealment of malicious behaviors. In this paper, we discover that uncertain factors have significant influence on the effectiveness of malware detection. To cope with the problems, a new VM-aware detection scheme, namely Divergence Detector, is proposed to address the swindle of the evolved malware. Unlike conventional schemes, the Divergence Detector reduces the uncertain factors at instruction level, and can detect the divergence of multi-execution traces across heterogeneous virtual machines. The proposed Divergence Detector is implemented across the three commonly used VM platforms, that is, QEMU, Bochs and Xen. It compares the code coverage of the execution traces on various VM platforms to discover the deviation of behavior, thereby precisely detecting the VM-awareness. We will formally predict the effectiveness of Divergence Detector by constructing a mathematic model, which shows the maximum false positive rate is exponentially decreased with respect to the number of multi-executions. Representative samples utilizing seven types of commonly used VM-aware techniques were also employed for evaluation. The evaluation results indicate that the maximum false positive rate complies with our prediction. The uncertain factors play the major role in the VM-awareness detection. To reduce uncertain factors causing false positives, a method is proposed for VM-aware detection. The Divergence Detector can also enable the identification of new types of malware since the benign programs do not need to be aware of execution environment.

Original languageEnglish
Pages80-89
Number of pages10
DOIs
StatePublished - 9 Sep 2013
Event7th International Conference on Software Security and Reliability, SERE 2013 - Gaithersburg, MD, United States
Duration: 18 Jun 201320 Jun 2013

Conference

Conference7th International Conference on Software Security and Reliability, SERE 2013
CountryUnited States
CityGaithersburg, MD
Period18/06/1320/06/13

Keywords

  • Malware
  • VM-awareness
  • Virtual Mashine

Fingerprint Dive into the research topics of 'Divergence detector: A fine-grained approach to detecting VM-awareness malware'. Together they form a unique fingerprint.

Cite this