Windows Events - Part 2 of 3

Part 1 of this Windows Events series covered the basics & how tedious it can be to turn on native Process Auditing. Now let’s look at Sysmon, how to gain useful process-activities insights for FREE! Part 1 is rather easy-peasy for some but this round will be more 🧠 involved. Please refer to Cyber Security in 7 weeks contents page for background materials!

What to learn & WHY it matters…

What	Why
Part 1 - Process (running instance of a program) Auditing using native Windows audit events.	By default, some of these audit events are not enabled. While we may be able to see foreground apps that we interact with, many processes which include malware like backdoor or trojan horses, are running mostly in the background.
Part 2 - Free Microsoft Sysmon events	Tell us what a process is doing without the pain of configuring Windows policy. I will also related with the 3 types of Code-Execution & TTP models as shared in my Mental Model series.
Part 3 - Event Tracing for Windows	Preparation for intermediate (using FreeEDR) & advanced topics like developing your own App-Control agent in C# with free Visual Studio.

Outline & Objectives

Install Sysmon to get started. I will explain Sysmon w.r.t TTP models (both MITRE’s ATT&CK & my Attack-Life-Cycle approach) & cover lasting value propositions for most Cyber-Security Operations Centre job scopes (from host sensor development, event-transport to backend analysis).

Repeat Notepad exercise from Part 1 to save file; to experience the difference between native Process Audit & Sysmon. (Reuse steps from Part 1 to view Sysmon events from Event Viewer app)

Use Powershell to start a hidden Notepad process & view events; to illustrate some common offensive evasions.

Use Powershell to write all ProcessCreates & NetworkConnects into a file; how many distinct processes are network-bound?

Ways to transport events to backend because it is required for centralised monitoring & response. I will also cover pros & cons of various event-transport approaches. When we don’t get it right at the early phase, we will be screwed later when attackers can either silence selectively, or grind your event-ingestion to a halt (been there, done that)!

📢

Join my Telegram group https://t.me/+iWZ6l-x5jGE1ZDc9 & be notified of new contents!

Chasing After “Bad” is ENDLESS

This series is NOT about HOW to figure out “bad” because it is an INFINITE set:

This series helps you to zoom into that much smaller set in blue, because when you don’t, you will just end up like everyone else, chasing after endless “downstream” indicators like that hamster in a THREAT-MILL. Yes, countless FREE tools to obfuscate, mutate or to change malicious binaries or scripts so quickly that there’s really no way to keep up with so call Detection Engineering chasing After-Math indicators. In 2013, David Bianco wrote this conceptual model of Pyramid of Pain:

It is painful because the problem-space is infinite & you don’t know what you don’t know. What I am about to share is to deal with endpoint intrusion at Upstream!

Upstream vs Downstream Efforts

📢

Dan Heath’s “Upstream” best seller (published March 2020) explores the psychological forces that push us downstream—including “problem blindness,” which can leave us oblivious to serious problems in our midst. And Heath introduces us to the thinkers who have overcome these obstacles and scored massive victories by switching to an upstream mindset…

Many organisations are still of the view that Cyber Security is not their problem… For those who think it is, from a buyers’ perspective, it’s easier to explain & justify payments for some services or products that keep the “bad” out, with recurring subscriptions, feeds, products, services & so on. From the sellers’ perspective, it is also a natural evolution & continuity that started off from chasing after “bad-file” (after the first major Internet Worm hit, Anti-Virus became big businesses), to the apex of pyramid; Tactics, Techniques & Procedures (now we have lots of stories of shadowry Threat Actors groups like some comic series).

📢

We need to get out of this endless loop of chasing after “bad” & proactively shift LEFT… because attackers are already ahead, “poisoning” Upstream through supply side infiltration & tampering, to get to more Downstream victims.

So when I say, hey! Let’s focus on the legit set of network-bound processes & there will be fewer problems. It’s only natural to be skeptical because most are so used to chasing after bad things, bad news...

Observing TTP

It all boils down to these 3 layers within any complex systems (minus the humans): (1) Hosts/endpoints/devices (2) Software including Software-as-a-Service, & (3) Networks between the 2 earlier layers which connect users to software & services. This series is specific to Windows, a widely attacked layer. Concepts apply across OSes.

In case you haven’t read my Cyber Defense Mental Models part 2, here’s typically how things would connect:

Part 3 of Cyber Defense Mental Models, I also illustrated How do “Bad” Codes get in?:

Examples of left path (—Yes—> Vulnerabilities e.g. Remote Code Execution or RCE) are recent MS Exchange vulnerabilities, Log4J & even as silly as leaving sensitive information on poorly configured web-servers that list out nicely for attackers’ picking. These are what I termed as Low-Hang Fruits you need to fix or patch ASAP. I will also cover later part within Cyber Security in 7 Weeks series, how to mitigate against such known & Zero-day vulnerability exposures.

The following will focus on the right path (earlier diagram), delivering bad-stuff to victims & the aftermath of Code-Execution (visualisation below):

Code-Executions starts immediately after Payload Delivery, to perform the next offensive sub-objective, with calling-home or back to a remote Command & Control infrastructure (servers that listen for homing network calls) as the most commonly observed behavior for many endpoint-zone infiltration (yes, including IoT devices). Stage 2 & 3 refers to my RINSE (& repeat Recon, Infiltrate Networks, Spread & Execute-Objectives) Tactical (sub-objectives) Attack-Life Cycle across all kinds of logical attacks:

The triangles within the cycle refers to the Necessary & Sufficient conditions of Cyber Physical attacks as covered by part 1 of my Cyber Defense Mental Models. Apart from Adversaries-in-the-Middle types of attacks, ALL other logical attacks will involve Code-Execution for getting either into the systems or data (e.g. SQL injection to app layers) directly.

I developed this approach independently while I was working on a (Singapore) National Cyber Security Operations Center, way before I encountered MITRE ATT&CK. Although the tactical aspect looks somewhat similar, but my objectives are quite different from MITRE’s, which is trying to enumerate “bad” while my approach is rationalising what, why & how to disrupt cost-effectively.

Please go through the WHATs & WHYs of each tactic here before we proceed. Since Code-Execution is necessary for various offensive sub-objectives, it is important to observe what a process is doing. To explore and learn at your own pace, I strongly recommend Sysmon.

Sysmon Crash Course

Earlier on with Part 1 of Windows Events series, you might have experienced the TEDIUM (in Singlish we call it TROUBLESOME lah!) of just configuring Windows to figure out which process started. See what install a single Sysmon driver software can tell us about a process, https://www.blackhillsinfosec.com/a-sysmon-event-id-breakdown/ Table-1 (from a mind-map I did long ago…):

ProcessCreate (EventID 1) provides details related to Payload Delivery, Run or Execution (e.g. parameters used, privilege..), Installation of backdoor aka Persistence, calling to Remote C2… so on so forth offensive tactics, as highlighted in the horizontal red-box within Table-1, equivalent to the graph visualization earlier that shows Code-Execution linking to various other tactics.

For the latest schema & filtering info, check out SwiftOnSecurity sysmon-config github project! Each event type e.g. ProcessCreate below has various attributes/fields & comments that highlight certain technical considerations.

Install Sysmon…

Download Zip file from https://learn.microsoft.com/en-us/sysinternals/downloads/sysmon

Download https://raw.githubusercontent.com/SwiftOnSecurity/sysmon-config/master/sysmonconfig-export.xml to desktop

Start an admin Cmd.exe (search bar, type cmd, right-click Command Prompt row, run-as admin)

Follow the instruction under Usage section of Sysmon download page. Assuming all files are at the same directory, traverse to the directory with admin Cmd.exe session & run:


sysmon64 -i sysmonconfig-export.xml

Use Event Viewer…

And again you can view them with Event Viewer either by traversing to the correct event-channel, or create a custom filter (since there’s a lot of clicking):

🧠

Recap from Mental Model Part 3, a tactical objective can be realised with many techniques. Running EXEs, loading DLLs, exploits that run shell-code… so on so forth, that are specific techniques for Windows. The blue set within the earlier Venn-diagram are existing programs running in your Windows OS, e.g. Powershell which you will use later. Others like browser, email... are vectors or means to deliver “bad-stuff” to victims. Malware is nothing more than programs coded to perform specific malicious actions that are NOT part of your system (from the red-set) BEFORE the delivery & infection! And since malicious binaries & scripts can be mutated & obfuscated infinitely, it is painful for defenders to chase after such files, along with various indicators as stated in the Pyramid of Pain. An example of intersection between our finite legit programs & infinite bad, would be Powershell. It is built-in a scripting language that does whatever .NET framework can do on your Windows! The exercises later will strengthen your understanding & appreciation of Code-Execution. And for what? So that we can disrupt TTPs (write your own agent during Part 3)!

Table-1 vertical red-box highlights that Install Payload or Backdoor (column) has the highest count of event types. That means we have lots of information but it is also the “noisest” area to monitor since lots of benign processes are also writing files. Nonetheless, it is still a good idea that a beginner to go through the exercises to examine ProcessCreate (ID 1) with an associated activity FileCreate (ID 11).

Observe Notepad Writing a File

Make sure your Sysmon service is working & that you can drill into Sysmon/Operational with Event Viewer before proceeding. Use what you learn in Part 1, start a Notepad, write a file & save it anywhere you wish. Assuming you have done the exercise, it is most likely that you took two separate efforts to locate these 2 event-types. It is also tedious to scroll & figure out since different Event IDs are mixed together. To make life easier, create a custom view like the following. Notice there are two IDs defined 1, 11:

For the benefit of those who don’t even want to move your mouse cursors ;) I have put them side by side:

⚠️

The focus here is NOT about the steps or details of which field is for what (at least for now), but HOW TO link events together using the field ProcessGuid. ProcessId is a number that can be re-used by different processes, ProcessGuid ensures you can “correlate” (which is a wrong use of the term btw, the correct concept is attribution) or link (attributing) activities TO THE CORRECT process. So think of this like a Database schema, except the “tables” are not normalised (notice lots of repeated field-values) & it is for a good reason that I will cover in final part of this Windows Events series.

For this simple exercise, we can simply say (or interpret the data-set as): Notepad.exe wrote a TargetFilename: ….lol.txt.

Fetch Events with Get-WinEvent (pls use a bigger screen for this)

Instead of scrolling with mouse & UI, start an administrative (search for Power.., right-click, Run-as Admin) Powershell:

The earlier animation shows how you can use custom view Filter XML directly into Powershell cmdlet Get-WinEvent’s -filterXml parameter:


[xml] $qry = @"
<QueryList>
   <Query Id="0" Path="Microsoft-Windows-Sysmon/Operational">
     <Select Path="Microsoft-Windows-Sysmon/Operational">*[System[(EventID=1 or EventID=11)]]</Select>
   </Query>
 </QueryList>
"@

Press enter after "@, then change the ProcessGuid (*{4b8c0f54-…} * ) below before you copy, paste & then press Enter to run:


# this will likely take a LONG time... another shorter way in later exercise!
get-winEvent -filterXml $qry | where-object {$_.Message -like "*{4b8c0f54-b1a2-6388-c963-b85500000000}*" } | Format-List -Property RecordId,TimeCreated,ID,LevelDisplayName,Message

# alternatively use notepad.exe, may fetch > 1 instances, tally against ProcessGuid or ProcessId from Event Viewer or Task Manager respectively...
get-winEvent -filterXml $qry | where-object {$_.Message -like "*notepad.exe*" } | Format-List -Property RecordId,TimeCreated,ID,LevelDisplayName,Message

It may take a while depending how many accumulated events within your Windows. Event Viewer app is also using the same filter XML query under the hood to fetch event-records. The filter XML block retrieves both ProcessEvent (ID = 1) OR FileCreate (ID = 11). You can change 2nd EventID to something else like NetworkConnect, which will be a later exercise. For more details of various event types, it is worthwhile to read the valuable comments within https://github.com/SwiftOnSecurity/sysmon-config/blob/master/sysmonconfig-export.xml

Hidden Notepad…

Close all existing Notepad, start a non-admin Powershell console. Copy below, paste into console then press Enter.


Start-Process -WindowStyle hidden -FilePath notepad.exe

Fire up Task Manager, locate Notepad.exe:

Run-as Admin to start Powershell session. Copy below, paste & press Enter:


# similar to earlier exercise but removed `or EventID = 11`
[xml] $qry = @"
<QueryList>
   <Query Id="0" Path="Microsoft-Windows-Sysmon/Operational">
     <Select Path="Microsoft-Windows-Sysmon/Operational">*[System[(EventID=1)]]</Select>
   </Query>
 </QueryList>
"@

# note that -maxEvents fetches latest, not oldest... this is the red-box within earlier screen-shot
get-winEvent -filterXml $qry -maxEvents 100 |where-object {$_.Message -like "*notepad.exe*" } | Format-List -Property RecordId,TimeCreated,ID,LevelDisplayName,Message

Observations & Significance

There’s no Notepad to be seen immediately after Start-Process. Notice that -WindowStyle hidden option, it is a feature of Windows.

Any event field(s) that indicates a hidden or visible notepad.exe?
But why go through all that trouble?

If a user can’t see it, then likely s/he won’t be closing it. It is going to stay around until Windows shuts down. This is a common approach to create a long-live process that will be subsequently injected with malicious codes.

Look at the ParentImage field of earlier Notepad exercise & compare with this. Notice that earlier on, it should be Explorer.exe as parent because you used Windows GUI but this round, Powershell is the parent process that started Notepad.exe. Imagine yourself working within a Cyber Security monitoring operation that collects all these data into a backend for review, like what you did. Which pair of Parent-Child relationship would you find suspicious & why? 🧠 Based on Least-Effort Principle, it is unlikely that any legit users will go through the trouble of starting a Powershell, then use it to start a Notepad just to read or write a few lines! But what if there was a way to spoof the Parent-Process, such that all these audit data reflected Explorer.exe as parent?

Are there any fields that captured the Powershell command (Start-Process ….) used earlier? Unfortunately NO, it is a lot like your part 1 experience that Process Audit is off by default, so is Powershell auditing! Powershell is a high-level scripting language that does whatever .NET Framework can do, & that includes ability to use low-level functions that can alter what is being reported: Parent-Child relationship & CommandLine string that you’ve seen!

🔥

When compound with the fact that even audit events can’t tell us whether a Notepad.exe (or whatever legit app-process) is hidden or not, after manipulation of Parent-Child process relationship (turned to Explorer.exe → Notepad.exe instead of Powershell.exe → Notepad.exe) & CommandLine, there’s limited remote observability & detection. From my Cyber Defense Mental Model part-3 on Three types of Code-Execution, this exercise is about Type 2 - Abuse of System Features (intersection between the blue & red sets shown earlier), from the hidden windows style, even spoofing of Parent-PID is a feature as shared by Didier Stevens in 2009. There are links at the end of the article for further explorations.

Out of the 3 Types of Code-Execution, why start with Type 2 & nothing much about Portable Executable file-based (Type 1) malware or the sensational headline-types of Exploits (Type 3, nowadays vulnerabilities have very marketable names)?

Abusing Features Against Defenders is Effective Evasion

🧠

The most effective (& often cheapest) strategy is to use legitimate features against defenders.

Why?

Compared to exploits that may be OS & App version-specific, system features are quite portable across multiple types of Windows (servers vs clients) or even Linux (.NET core… write once re-use anywhere)!

Was it hard to Start-Process with hidden window-style? No! Easy, cheap & effective.

Anti-virus or whatever Host-controls companies may block Type-1 or 3 at a file or even network-packet level, but blocking hosts’ system features may incur a fair bit support issues. Again, attractive for threat actors when system commands are usually allowed & unblocked!

Since auditing are OFF’ed by default, but even when enabled, we saw gaps we’ve seen so far, it’s NOT just a matter of NOT blocked, but most often attacks will also carrying on (from one host to another, aka Lateral Movement) undetected!

Don’t believe me blindly, read up https://attack.mitre.org/tactics/TA0005/ & see or count how many are related to system features, or due to the design & implementation of OS features that allow such techniques!

⚠️

This entry is NOT going to make you an expert “Threat Hunter”… The industry makes it sound cool or fashionable to “Hunt Evil”, but the reality of burn-out is real! The whole point of this series is to convince you that by focusing on the legit set of Network-Bound processes (earlier Venn-diagram), there are DETERMINISTIC ways to evaluate process’s legitimacy & discover NEW system-abuse methods!

How many Network-Bound processes…

Are running on your machine? I thank you for keeping up with me! This is an important but straight-forward exercise to establish some facts or evidence. Again use admin-Powershell:


# EventID = 3 => NetworkConnect, execution may take a while...
Get-WinEvent -FilterHashtable @{logname="Microsoft-Windows-Sysmon/Operational";id=3} | Format-List -Property RecordId,TimeCreated,ID,LevelDisplayName,Message > [Environment]::GetFolderPath("Desktop")\networkconnect.txt

All Sysmon NetworkConnect events collected so far since the time of installation of Sysmon driver are written to your desktop in that file.

Do find out:

How many distinct (non-repeated) Image (full path to the stored program) are there?

For each distinct Image, note their CommandLine, exactly the same, varying slightly or very different?

IntegrityLevel (need to link with ProcessCreate from step 2) & User of the process as this pertains to the tactical aspect of Privilege-Escalation.

You will realise it is very tedious & error prone when done manually with your eye-balls. But imagine you can do this! This is part of FreeEDR backend, a multi-modal (graph, key-value, SQL) database.

From the 33 distinct Image, we drill into NetworkConnect…

Transport Events to Backend

Finally... before we can event query like what I did earlier, we first need to send events back to a backend so that it gets into some of database. I will cover a few common approaches & compare Pros vs Cons as per outlined earlier:

Approaches	+Pros	-Cons
Windows Event Forwarding	1. Built into Windows environments, nothing much to install but more of configuration. 2. Handles VPN, RAS, and DirectAccess scenarios well and will reconnect and send any accumulated backlog of events when the connection to the WEF Collector is re-established. 3. Kerberos (mutual) authentication and encryption are performed regardless if HTTP or HTTPS is selected.	1. Can be quite complex to configure, there are Pull vs Push options.. especially for large-scale environment, experience & knowledge are necessary to scale! 2. Depending on how you setup Windows Event Collector, it can be a Single-Point-of-Failure.
Syslog	1. Most commonly available logging transport mechanism for network devices like switches, firewall, security-appliance to Linux OSes. 2. Easy to configure, put it the syslog collect IP or hostname, port number if custom, done!	1. Windows does not support Syslog emission or collection out-of-the-box. You need to install some software agents at both sender & receiver. 2. Often no authentication or transport security 3. Easy to DoS syslog-collector when network segmentation is done wrongly since there is usually NO event delivery optimisation or rate-control
Software Agent-based e.g. Elastic Beats, Nxlog	Multiple OSes, most of the +Pros like the above, other than need to install something	1. Some devices cannot install agents. 2. Need to patch or update software agents.

Recap

You covered:

Fundamentals not just in theory, but also hands-on exercise to look at key process attributes & some activities.

Difference between native Process Audit & Sysmon event data-sets & how they related to 2 TTPs (MITRE ATT&CK & my RINSE TTP).

Starting a hidden Notepad process & view events to get the main idea behind offensive evasions.

Capturing all ProcessCreates & NetworkConnects into a file to find out how many distinct processes are network-bound?

Pros & cons of various event-transport approaches.

📖

Everything so far is setting you up for the final part of this Windows Events series, which requires some programming background to look into Event Tracing for Windows. If you don’t have, take this opportunity to learn some .NET programming. Conversely, you may already have some programming or computer science background, & it will be interesting to apply existing knowledge for system monitoring & protection!

Why settle for observing when we can disrupt attacks?