ACSAC Test of Time Award for "Detecting Kernel-Level Rootkits Through Binary Analysis"

One of my first papers, Detecting Kernel-Level Rootkits Through Binary Analysis, was selected for the ACSAC Test of Time Award at this year’s conference! Thank you ACSAC!

The awards ceremony was led by Jeremy Epstein, who managed the not-insignificant feat of wrangling Chris, Giovanni, and I into the same Zoom webinar to ask some thought-provoking questions about this work.

A lot of time has passed since this paper was first published, and unsurprisingly the security landscape has also undergone a drastic transformation. I was a first-year PhD student in Dick Kemmerer’s Reliable Systems Lab at the time, though I had already interned in the lab for a year at that point. People were extremely worried about Internet flash worms, firewalls were unknown to most, Microsoft CEO Bill Gates had only recently penned his Trustworthy Computing memo identifying security as something to be concerned about, stack cookies were hot new tech, ROP had yet to be invented, “SSL Everywhere” was a pipe dream, the commercialization and militarization of exploitation had yet to hit the mainstream, and virtually nobody was seriously investigating binary analysis. Indeed, there was at the time and for many years after lingering questions about why one might want to analyze binary programs when static and dynamic analysis of source programs is so much easier.

Yet today, binary analysis is taken for granted as a crucial capability for finding vulnerabilities in third-party code, detecting malware on endpoints, and instrumenting already-distributed programs with novel run-time defenses. So, it’s intensely gratifying to be recognized as having been one of the first movers in this space.

There are multiple innovative aspects of this paper. First off, binary analysis of untrusted Linux kernel modules was an unconventional but pragmatic design choice given the realities of how Linux kernel rootkits were and are distributed by attackers. But in addition, we were not simply performing some scan over the raw bytes comprising the tested rootkits.¹ Rather, we developed an analysis to identify malicious behavior, applying what was essentially a simple form of dynamic symbolic execution to check for potentially dangerous accesses to important kernel data structures that rootkits would hook in order to hide their presence from user-space detectors. The key insight motivating that design choice was that while the syntax of attacks can be obfuscated, their semantics are much more difficult to hide. That is, a malicious rootkit could use essentially limitless variations of concrete sequences of instructions (or bytes) to implement its kernel hooking. However, there are only a limited number of ways for the rootkit to interpose on VFS operations, rendering the detection problem tractable and reliable from a behavioral standpoint. Indeed, in this way this work presaged a long string of publications on behavioral malware detection using dynamic analysis as a way to mitigate static obfuscation techniques ranging from simple unpacking to strong encryption.²

In any case, binary analysis and dynamic symbolic execution remain to this day important open research areas in security. If you happen to be thinking about a PhD and are interested in working on these (and related) problems, please feel free to reach out!