Lead Microsoft Engineer Kevin Sheldrake Brings Sysmon to Linux
4 - 7 min read
Nov 22, 2021
Thanks to Kevin Sheldrake, co-author of Sysmon for Linux from Microsoft for working with us on this article.
Seven years after Microsoft Sysinternals released Sysmon – a system monitoring tool for Windows that reports key system activity via the event log – we were very proud to release the Linux version of the same tool at the Sysinternals @25 celebration event. (You can still catch up on the talks by clicking on the event link.)
Unlike the Linux versions of ProcDump and ProcMon, which were complete rewrites for the Linux platform, Sysmon For Linux is a direct port of the Windows code. This means that the Linux version uses the exact same configuration loader and event filtering engine that are familiar and comfortable to existing Sysmon users.
Built on eBPF
The key enabling technology that allowed Sysmon For Linux to come to life is eBPF. Originally standing for Enhanced Berkeley Packet Filter, eBPF is a way to run user-specified and user-controlled code inside the Linux kernel, hooked onto any number of the numerous attachment points. This means that software developers can write applications, such as Sysmon, where one part resides in the application-centric ‘userland’ (where everything initiated by a user runs, from your favourite editor to the systemd daemon that starts up and controls Linux); and the other part resides inside the Linux kernel, triggered by events, and able to collect kernel information, which it transmits outside to the other part.
The Windows version of Sysmon comprises a service and a driver: the driver uses supported kernel callouts and I/O monitoring capabilities to generate the events, which it communicates to the service that filters the events as per the supplied configuration, before writing them out to the Windows event log. On Linux, we replaced the driver with eBPF programs, ported the service, and wrote the events to Syslog instead.
To enable the widest compatibility with the many distributions and versions of Linux, we abstracted the eBPF architecture into a library called SysinternalsEBPF, which allows Sysmon to focus on the task of system observability. This library – which is also available as open source and will soon become the basis for ProcMon For Linux – loads and controls the eBPF programs, and, crucially, informs them how to navigate the Linux kernel internals.
Each version of the Linux kernel, and each configuration of it, lays out its data structures in slightly different ways. The standard eBPF approach to locating information within them is called BPF Type Format (BTF) but, whereas eBPF is usually enabled in common modern distributions, enabling BTF often requires users to reconfigure and recompile the kernel themselves. To avoid this disruption, SysinternalsEBPF comes with a database of common kernel layouts, a discovery process that finds the layout itself, and a fallback kernel module that can be compiled (but not actually loaded) to identify the required details precisely. SysinternalsEBPF, and therefore Sysmon, can then work on systems without requiring a custom kernel or the installation of kernel symbols packages.
Safe and Sound
Traditionally, interacting with the Linux kernel would require the creation of a kernel module, compiled for each specific version. The benefit of eBPF over kernel modules is the safety provided by the eBPF verifier and virtual machine. All eBPF programs are first checked by a verifier, built into the kernel itself, which ensures that programs do not read or write arbitrary memory, and that they exit within a specified number of machine code operations. This prevents kernel crashes and threads from hanging.
The eBPF programs run inside a virtual machine (implementing BPF machine code) that limits access to kernel APIs to a minimal set. This prevents eBPF programs from causing arbitrary effects in the kernel and ensures that Sysmon is safe to run.
Instead of affecting the kernel itself, Sysmon’s eBPF programs simply transmit their event data to the userland part of Sysmon, via the perf ring buffer. As events are received, they are processed, filtered and output to the Syslog.
An Eventful Experience
The heart of Sysmon is the events it can generate. New event types – including Linux-specific ones – will be developed over time, but Sysmon was launched with the following:
- Sysmon State Change
- Sysmon Configuration Change
- Process Create
- Process Terminate
- File Create
- File Delete
- Network Connection
- Raw Access Read
- Process Access
Sysmon creates these events, like many other tracing programs, by attaching to system call tracepoints. When userland programs need access to resources – such as creation and deletion of files, network sends and receives, or launch and termination of processes – they make their requests by initiating a system call (syscall) to the kernel. Each syscall, and the entire syscall architecture, have tracepoints on their entry and exit points that eBPF programs can be attached to. Here, our telemetry programs get run whenever the specific syscalls are triggered.
To avoid Time Of Check, Time Of Use (TOCTOU) race conditions, we obtain our event details (where possible) from the kernel data structures, rather than the syscall parameters specified by userland applications. This handles adversarial programs that attempt to deceive system monitors by switching the contents of their parameter buffers between the time the kernel reads them, and the time the tracing program reads them. By obtaining the details from the kernel data structures themselves, Sysmon reports the actual files, command lines, and network traffic, rather than the ones the sneaky programs want to pretend they were accessing.
Filtering and Output
As I mentioned at the beginning of this article, we ported the configuration loading and event filtering parts of Sysmon from Windows to Linux. If you already use Sysmon and know how to write configuration files (see 2847054 and sysmon modular for examples) then you can make configuration files for Sysmon For Linux in the exact same way; and you can also rely on the filtering engine to treat your configuration as you’d expect.
In terms of output, Sysmon For Linux generates events using the same XML schema as the Windows version, the only difference being that the events are sent to Syslog instead of the Windows event log. If your Security Information and Event Management system (SIEM) can already process Sysmon XML events from Windows, then it can probably also already process events from Linux, if you just set it to collect from Syslog instead.
Go Get It!
Sysinternals Linux tools are open source and available on GitHub as source, as well as in Microsoft repositories as DEBs and RPMs. All the details to obtain the pre-built versions, and also to build from source, are in the relevant GitHub repositories, where you can also raise issues and submit your own pull requests. Sysinternals for Linux (including ProcMon and ProcDump) benefit from community involvement and are part of Microsoft’s commitment to open source projects. Go and browse the source code and read the documentation at:
We hope that you find Sysmon For Linux useful and effective, and that it fills a functionality gap in your system monitoring on Linux.
About the Author
Kevin Sheldrake is a penetration tester and researcher who started working in the technical security field in 1997. Over the years, Kev has been a developer and systems administrator of ‘secure’ systems, an infosec policy consultant, a penetration tester, a reverse engineer, and an entrepreneur who founded and ran his own security consulting company. He currently works for Microsoft, porting Sysinternals Sysmon from Windows to Linux.
He has a Masters degree, is a Chartered Engineer and, in the past, has been a CHECK Team Leader, a CISSP and held CLAS.
Kev has presented at 44CON, Troopers, DeepSec, EMF Camp, DEFCON 4420, 441452 and 441392 on RFID crypto (Cracking HiTag2 Crypto); EMF Camp, DEFCON 4420 and 441452 on hacking embedded devices (Inside our Toys); presented on building debuggers for embedded devices at Securi-Tay (Phun with Ptrace()); and also presented a lengthy take down on the use of NLP in Social Engineering at DEFCON 4420 (Social Engineering LIES!). He is published in PoC||GTFO 0x18 for the write up of his "Exploits With Scratch" talk (44con, EMF Camp).
