Abstract
An operating system kernel is the core of system software which is responsible for the integrity and operations of a conventional computer system. Authors of malicious software (malware) have been continuously exploring various attack vectors to tamper with the kernel. Traditional malware detection approaches have focused on the codecentric aspects of malicious programs, such as the injection of unauthorized code or the control flow patterns of malware programs. However, in response to these malware detection strategies, modern malware is employing advanced techniques such as reusing existing code or obfuscating malware code to circumvent detection.
In this dissertation, we offer a new perspective to malware detection that is different from the code-centric approaches. We propose the data-centric malware defense architecture (DMDA), which models and detects malware behavior by using the properties of the kernel data objects targeted during malware attacks. This architecture employs external monitoring wherein the monitor resides outside the monitored kernel to ensure tamper-resistance. It consists of two core system components that enable inspection of the kernel data properties.
First, an external monitor has a challenging task in identifying the data object information of the monitored kernel. We designed a runtime kernel object mapping system which has two novel characteristics: (1) an un-tampered view of data objects resistant to memory manipulation and (2) a temporal view capturing the allocation context of dynamic memory. We demonstrate the effectiveness of these views by detecting a class of malware that hides dynamic data objects. Also, we present our analysis of malware attack behavior targeting dynamic kernel objects.
Second, in addition to the mapping of kernel objects, we present a new kernel malware characterization approach based on kernel memory access patterns. This approach generates signatures of malware by extracting recurring data access patterns specific to malware attacks. Moreover, each memory pattern in the signature represents abstract data behavior; therefore, it can expose common data behavior among malware variants. Our experiments demonstrate the effectiveness of these signatures in the detection of not only malware with signatures but also malware variants that share memory access patterns.
Our results utilizing these approaches in the defense against kernel rootkits demonstrate that the DMDA can be an effective solution that complements code-centric approaches in kernel malware defense.
