[windows.powershell_operational] Handle ContextInfo containing multi-line values #15055
Description
Integration Name
Windows [windows]
Dataset Name
windows.powershell_operational
Integration Version
3.0.1
Agent Version
9.1.0
Agent Output Type
elasticsearch
Elasticsearch Version
9.1.0
OS Version and Architecture
unknown
Software/API Version
No response
Error Message
field [winlog.event_data.ContextInfo] does not contain value_split [[:=]]
Event Original
{
"winlog": {
"event_data": {
"ContextInfo": " Severity = Informational\n Host Name = ConsoleHost\n Host Version = 5.1.26100.4652\n Host ID = 11111111-1111-1111-1111-111111111111\n Host Application = powershell.exe -ExecutionPolicy Restricted -Command\n some multline\n\n powershell script\n\n $foo=\"bar\"\n\n Write-Host 'Final result:',$foo\n\n Engine Version = 5.1.26100.4652\n Runspace ID = 11111111-1111-1111-1111-111111111111\n Pipeline ID = 1\n Command Name =\n Command Type = Script\n Script Name =\n Command Path =\n Sequence Number = 28\n User = WINNET\\SYSTEM\n Connected User =\n Shell ID = Microsoft.PowerShell"
}
}
}
What did you do?
Ingesting event ID 4103 where the executed command was PowerShell with a multi-line -command value.
What did you see?
The kv processor fails to split the ContextInfo field.
What did you expect to see?
The splitting logic for ContextInfo should be more robust. Likely a script processor needs to be used. I started PoC with (it still needs work).
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": "Pipeline for Windows Powershell/Operational events",
"processors": [
{
"script": {
"lang": "painless",
"if": "ctx.winlog?.event_id == \"4103\"",
"source": "String text = ctx.winlog.event_data.ContextInfo;\nString[] lines = /\\n/.split(text);\nMap result = new HashMap();\nString currentKey = null;\nList currentValue = new ArrayList();\n\nfor (String line : lines) {\n // TODO: the old kv handled ' : ' and ' = '.\n if (line.startsWith(' ') && line.contains(' = ')) {\n // Save previous key-value if exists\n if (currentKey != null) {\n StringBuilder sb = new StringBuilder();\n for (int i = 0; i < currentValue.length; i++) {\n if (i > 0) sb.append(\"\\\\n\");\n sb.append(currentValue[i]);\n } \n String valueStr = sb.toString().trim();\n result.put(currentKey, valueStr);\n }\n\n // Start new key-value\n String[] parts = / = /.split(line);\n currentKey = parts[0].trim();\n currentValue = new ArrayList();\n if (parts.length > 1) {\n currentValue.add(parts[1]);\n } else {\n currentValue.add('');\n }\n } else if (currentKey != null) {\n // Continue multiline value\n currentValue.add(line);\n }\n}\n\n// Save the last key-value\nif (currentKey != null) {\n StringBuilder sb = new StringBuilder();\n for (int i = 0; i < currentValue.length; i++) {\n if (i > 0) sb.append(\"\\\\n\");\n sb.append(currentValue[i]);\n }\n String valueStr = sb.toString().trim();\n result.put(currentKey, valueStr);\n}\n\n// Add parsed fields to the document\nfor (Map.Entry entry : result.entrySet()) {\n field('winlog.event_data.'+entry.getKey()).set(entry.getValue());\n}"
}
}
]
},
"docs": [
{
"_source": {
"winlog": {
"event_id": "4103",
"event_data": {
"ContextInfo": " Severity = Informational\n Host Name = ConsoleHost\n Host Version = 5.1.26100.4652\n Host ID = 11111111-1111-1111-1111-111111111111\n Host Application = powershell.exe -ExecutionPolicy Restricted -Command\n some multline\n\n powershell script\n\n $foo=\"bar\"\n\n Write-Host 'Final result:',$foo\n\n Engine Version = 5.1.26100.4652\n Runspace ID = 11111111-1111-1111-1111-111111111111\n Pipeline ID = 1\n Command Name =\n Command Type = Script\n Script Name =\n Command Path =\n Sequence Number = 28\n User = WINNET\\SYSTEM\n Connected User =\n Shell ID = Microsoft.PowerShell"
}
}
}
}
]
}
This produces:
{
"winlog": {
"event_id": "4103",
"event_data": {
"Runspace ID": "11111111-1111-1111-1111-111111111111",
"User": "WINNET\\SYSTEM\\n Connected User =",
"Engine Version": "5.1.26100.4652",
"Host ID": "11111111-1111-1111-1111-111111111111",
"Severity": "Informational",
"Host Name": "ConsoleHost",
"Shell ID": "Microsoft.PowerShell",
"Sequence Number": "28",
"Command Type": "Script\\n Script Name =\\n Command Path =",
"Pipeline ID": "1\\n Command Name =",
"Host Version": "5.1.26100.4652",
"Host Application": "powershell.exe -ExecutionPolicy Restricted -Command\\n some multline\\n\\n powershell script\\n\\n $foo=\"bar\"\\n\\n Write-Host 'Final result:',$foo\\n",
"ContextInfo": " Severity = Informational\n Host Name = ConsoleHost\n Host Version = 5.1.26100.4652\n Host ID = 11111111-1111-1111-1111-111111111111\n Host Application = powershell.exe -ExecutionPolicy Restricted -Command\n some multline\n\n powershell script\n\n $foo=\"bar\"\n\n Write-Host 'Final result:',$foo\n\n Engine Version = 5.1.26100.4652\n Runspace ID = 11111111-1111-1111-1111-111111111111\n Pipeline ID = 1\n Command Name =\n Command Type = Script\n Script Name =\n Command Path =\n Sequence Number = 28\n User = WINNET\\SYSTEM\n Connected User =\n Shell ID = Microsoft.PowerShell"
}
}
}Anything else?
This reproduced the error and is based on the current pipeline.
POST _ingest/pipeline/_simulate
{
"pipeline": {
"description": "Pipeline for Windows Powershell/Operational events",
"processors": [
{
"kv": {
"description": "Split Event 4103 event data fields.",
"field": "winlog.event_data.ContextInfo",
"target_field": "winlog.event_data",
"field_split": "",
"trim_key": " \n\t",
"trim_value": " \n\t",
"value_split": "[:=]",
"if": "ctx.winlog?.event_id == '4103'"
}
}
]
},
"docs": [
{
"_source": {
"winlog": {
"event_id": "4103",
"event_data": {
"ContextInfo": " Severity = Informational\n Host Name = ConsoleHost\n Host Version = 5.1.26100.4652\n Host ID = 11111111-1111-1111-1111-111111111111\n Host Application = powershell.exe -ExecutionPolicy Restricted -Command\n some multline\n\n powershell script\n\n $foo=\"bar\"\n\n Write-Host 'Final result:',$foo\n\n Engine Version = 5.1.26100.4652\n Runspace ID = 11111111-1111-1111-1111-111111111111\n Pipeline ID = 1\n Command Name =\n Command Type = Script\n Script Name =\n Command Path =\n Sequence Number = 28\n User = WINNET\\SYSTEM\n Connected User =\n Shell ID = Microsoft.PowerShell"
}
}
}
}
]
}