Recently I was challenged with analyzing different evtx files and now I want to make my workflow to accomplish this mission published.
In this HowTo we will parse our EVTX Files with an Rust Parser to transform the files into JSON Files. Those will then be delivered to an Logstash instance where we can play with the Logstash JSON Filter to manipulate the Elasticsearch Output.
Preparation
There is a really fast EVTX Parser written in Rust – get it here: https://github.com/omerbenamram/evtx
Installation
(Based on a Ubuntu System)
sudo apt install cargo cargo install evtx export PATH=$HOME/.cargo/bin:$PATH
Processing
There is a github repository which contains different EVTX Files for testing. If you don’t have some own EVTX, this might be a good starting point: https://github.com/sbousseaden/EVTX-ATTACK-SAMPLES
Let’s see what we can get:
evtx_dump EVTX-ATTACK-SAMPLES/Command\ and\ Control/DE_RDP_Tunnel_5156.evtx
Record 1
<?xml version="1.0" encoding="utf-8"?>
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
<System>
<Provider Name="Microsoft-Windows-Eventlog" Guid="{fc65ddd8-d6ef-4962-83d5-6e5cfe9ce148}">
</Provider>
<EventID>1102</EventID>
...
...
Ok. That parser is able to extract single records from an Windows Eventlog File and it also have some other output formats:
-o, --format <output-format> Sets the output format [default: xml] [possible values: json, xml, jsonl]
So – as soon as we will parse the EVTX with the option evtx_dump -o jsonl --dont-show-record-number EVTX-ATTACK-SAMPLES/Command\ and\ Control/DE_RDP_Tunnel_5156.evtx we will get a nicely formated JSON output wich is able to be consumed with logstash.
To find the files in Batch mode I’m using a alternate find utility from https://github.com/sharkdp/fd.
For the next command it’s needed to CWD into the directory where the EVTX Files resides.
fd -e evtx -x evtx_dump -o jsonl '{}' -f '{.}.json'
Elasticsearch
I’ll assume you already have a ELK (Elasticsearch/Logstash/Kibana) up and running and know those basics.
For the very basic start I’ll use the following config to just test the first imports, explain advantages and disadvantages and we will improve the things together.
# Logstash Config
input {
file {
id => "evtx_dump"
path => [ "/logs/evtx_json/*.json" ]
type => "evtx_dump"
sincedb_path => "/dev/null"
max_open_files => 16384
start_position => "beginning"
}
}
filter{
if [type] == "evtx_dump" {
json{
source => "message"
}
}
}
output {
elasticsearch {
index => "logstash-%{[type]}"
hosts => ["http://elasticsearch:9200"]
}
}
We will receive elastic-documents like this:
logstash_1 | {
logstash_1 | "@timestamp" => 2020-06-28T08:45:31.917Z,
logstash_1 | "Event" => {
logstash_1 | "EventData" => {
logstash_1 | "FilterRTID" => 0,
logstash_1 | "RemoteUserID" => "S-1-0-0",
logstash_1 | "DestPort" => "55139",
logstash_1 | "SourcePort" => "1900",
logstash_1 | "ProcessID" => 1588,
logstash_1 | "DestAddress" => "::1",
logstash_1 | "Protocol" => 17,
logstash_1 | "LayerName" => "%%14610",
logstash_1 | "LayerRTID" => 46,
logstash_1 | "Application" => "\\device\\harddiskvolume1\\windows\\system32\\svchost.exe",
logstash_1 | "RemoteMachineID" => "S-1-0-0",
logstash_1 | "SourceAddress" => "ff02::c",
logstash_1 | "Direction" => "%%14592"
logstash_1 | },
logstash_1 | "System" => {
logstash_1 | "Provider" => {
logstash_1 | "#attributes" => {
logstash_1 | "Guid" => "54849625-5478-4994-A5BA-3E3B0328C30D",
logstash_1 | "Name" => "Microsoft-Windows-Security-Auditing"
logstash_1 | }
logstash_1 | },
logstash_1 | "Correlation" => nil,
logstash_1 | "TimeCreated" => {
logstash_1 | "#attributes" => {
logstash_1 | "SystemTime" => "2019-02-13T18:05:21.597104Z"
logstash_1 | }
logstash_1 | },
logstash_1 | "Opcode" => 0,
logstash_1 | "Task" => 12810,
logstash_1 | "Level" => 0,
logstash_1 | "Computer" => "PC01.example.corp",
logstash_1 | "EventRecordID" => 227953,
logstash_1 | "Keywords" => "0x8020000000000000",
logstash_1 | "Channel" => "Security",
logstash_1 | "Version" => 1,
logstash_1 | "Execution" => {
logstash_1 | "#attributes" => {
logstash_1 | "ProcessID" => 4,
logstash_1 | "ThreadID" => 56
logstash_1 | }
logstash_1 | },
logstash_1 | "EventID" => 5156,
logstash_1 | "Security" => nil
logstash_1 | },
logstash_1 | "#attributes" => {
logstash_1 | "xmlns" => "http://schemas.microsoft.com/win/2004/08/events/event"
logstash_1 | }
logstash_1 | },
logstash_1 | "message" => "{\"Event\":{\"#attributes\":{\"xmlns\":\"http://schemas.microsoft.com/win/2004/08/events/event\"},\"EventData\":{\"Application\":\"\\\\device\\\\harddiskvolume1\\\\windows\\\\system32\\\\svchost.exe\",\"DestAddress\":\"::1\",\"DestPort\":\"55139\",\"Direction\":\"%%14592\",\"FilterRTID\":0,\"LayerName\":\"%%14610\",\"LayerRTID\":46,\"ProcessID\":1588,\"Protocol\":17,\"RemoteMachineID\":\"S-1-0-0\",\"RemoteUserID\":\"S-1-0-0\",\"SourceAddress\":\"ff02::c\",\"SourcePort\":\"1900\"},\"System\":{\"Channel\":\"Security\",\"Computer\":\"PC01.example.corp\",\"Correlation\":null,\"EventID\":5156,\"EventRecordID\":227953,\"Execution\":{\"#attributes\":{\"ProcessID\":4,\"ThreadID\":56}},\"Keywords\":\"0x8020000000000000\",\"Level\":0,\"Opcode\":0,\"Provider\":{\"#attributes\":{\"Guid\":\"54849625-5478-4994-A5BA-3E3B0328C30D\",\"Name\":\"Microsoft-Windows-Security-Auditing\"}},\"Security\":null,\"Task\":12810,\"TimeCreated\":{\"#attributes\":{\"SystemTime\":\"2019-02-13T18:05:21.597104Z\"}},\"Version\":1}}}",
logstash_1 | "path" => "/logs/evtx_json/DE_RDP_Tunnel_5156.json",
logstash_1 | "@version" => "1",
logstash_1 | "type" => "evtx_dump",
logstash_1 | "log.host" => "fdd3449b48a6"
logstash_1 | }
Or as it would shown in Kibana:
The problem here is, that all @timestampfields are set to the Date/Time of the import and not to the Date/Time of the Log Event itself.
Of course we could now tweak the Kibana Index Pattern to correct this but I would like to dive deeper into the logstash processing and mutating direct on the ingested data to get into somewhat of clean data state.
filter{
if [type] == "evtx_dump" {
json{
source => "message"
}
# Remove message field if json parser was successful
if "_jsonparsefailure" not in [tags] {
mutate {
remove_field => [ "message" ]
}
}
# Extract timestamp from Event Time Created System Time
if [Event][System][TimeCreated][#attributes][SystemTime] {
date {
match => [ "[Event][System][TimeCreated][#attributes][SystemTime]" , "ISO8601" ]
target => "@timestamp"
}
}
}
}
This filter will give us Documents like this:
logstash_1 | {
logstash_1 | "log.host" => "fdd3449b48a6",
logstash_1 | "path" => "/logs/evtx_json/DE_RDP_Tunnel_5156.json",
logstash_1 | "@timestamp" => 2019-02-13T18:05:24.601Z,
logstash_1 | "type" => "evtx_dump",
logstash_1 | "@version" => "1",
logstash_1 | "Event" => {
logstash_1 | "EventData" => {
logstash_1 | "Protocol" => 17,
logstash_1 | "ProcessID" => 1588,
logstash_1 | "Application" => "\\device\\harddiskvolume1\\windows\\system32\\svchost.exe",
logstash_1 | "Direction" => "%%14592",
logstash_1 | "LayerRTID" => 46,
logstash_1 | "DestAddress" => "::1",
logstash_1 | "RemoteUserID" => "S-1-0-0",
logstash_1 | "RemoteMachineID" => "S-1-0-0",
logstash_1 | "FilterRTID" => 0,
logstash_1 | "SourcePort" => "1900",
logstash_1 | "LayerName" => "%%14610",
logstash_1 | "SourceAddress" => "ff02::c",
logstash_1 | "DestPort" => "55139"
logstash_1 | },
logstash_1 | "System" => {
logstash_1 | "Task" => 12810,
logstash_1 | "Computer" => "PC01.example.corp",
logstash_1 | "Security" => nil,
logstash_1 | "EventID" => 5156,
logstash_1 | "Execution" => {
logstash_1 | "#attributes" => {
logstash_1 | "ThreadID" => 56,
logstash_1 | "ProcessID" => 4
logstash_1 | }
logstash_1 | },
logstash_1 | "Opcode" => 0,
logstash_1 | "TimeCreated" => {
logstash_1 | "#attributes" => {
logstash_1 | "SystemTime" => "2019-02-13T18:05:24.601424Z"
logstash_1 | }
logstash_1 | },
logstash_1 | "Correlation" => nil,
logstash_1 | "Level" => 0,
logstash_1 | "Provider" => {
logstash_1 | "#attributes" => {
logstash_1 | "Guid" => "54849625-5478-4994-A5BA-3E3B0328C30D",
logstash_1 | "Name" => "Microsoft-Windows-Security-Auditing"
logstash_1 | }
logstash_1 | },
logstash_1 | "Keywords" => "0x8020000000000000",
logstash_1 | "Version" => 1,
logstash_1 | "Channel" => "Security",
logstash_1 | "EventRecordID" => 227957
logstash_1 | },
logstash_1 | "#attributes" => {
logstash_1 | "xmlns" => "http://schemas.microsoft.com/win/2004/08/events/event"
logstash_1 | }
logstash_1 | }
logstash_1 | }
Update: Lowercase all the keys
# https://github.com/omerbenamram/evtx/issues/109 evtx_dump security.evtx -o jsonl | jq -r 'walk(if type == "object" then with_entries(.key|=ascii_downcase) else . end)'