Three-phase behavior-based detection and classification of known and unknown malware

Ying-Dar Lin, Yuan Cheng Lai*, Chun Nan Lu, Peng Kai Hsu, Chia Yin Lee

*Corresponding author for this work

Research output: Contribution to journalArticlepeer-review

12 Scopus citations


To improve both accuracy and efficiency in detecting known and even unknown malware, we propose a three-phase behavior-based malware detection and classification approach, with a faster detector in the first phase to filter most samples, a slower detector in the second phase to observe remaining ambiguous samples, and then a classifier in the third phase to recognize their malware type. The faster detector executes programs in a sandbox to extract representative behaviors fed into a trained artificial neural network to evaluate their maliciousness, whereas the slower detector extracts and matches the LCSs of system call sequences fed into a trained Bayesian model to calculate their maliciousness. In the third phase, we define malware behavior vectors and calculate the cosine similarity to classify the malware. The experimental results show that the hybrid two-phase detection scheme outperforms the one-phase schemes and achieves 3.6% in false negative and 6.8% in false positive. The third-phase classifier also distinguishes the known-type malware with an accuracy of 85.8%.

Original languageEnglish
Pages (from-to)2004-2015
Number of pages12
JournalSecurity and Communication Networks
Issue number11
StatePublished - 25 Jul 2015


  • Behavior analysis
  • Malware classification
  • Malware detection
  • Sandbox
  • System call

Fingerprint Dive into the research topics of 'Three-phase behavior-based detection and classification of known and unknown malware'. Together they form a unique fingerprint.

Cite this