pySigma-backend-openobserve

[juju4]

Hello,
I am working on a backend for openobserve which is sql-based (https://github.com/juju4/pySigma-backend-openobserve/).

Working on a kunai pipeline and I have a questions:

• I need to mark some fields as non-existing like 'Initiated' in network rules or ParentCommandLine in process ones.
  How to do that? if unset or set to empty value, it keeps default field name
• To add an expression depending on rule type, it's the rule_conditions? for example, to say network are type=connect
• For full-text search, openobserve has a FTS option but only for fields configured for it.
  As an alternate option, is there a way to match that to a given field depending on logsource?
  For example, product=linux and service=sshd could translate to journald unit=ssh and journald MESSAGE field (body_message in my setup).
  Same question for a default value. if product=linux, use journald MESSAGE unless more granular match.

Is there a community review process? right place to discuss it?
Don't see any pointers on https://sigmahq.io/docs/digging-deeper/backends.html

[thomaspatzke]

Hi!

* I need to mark some fields as non-existing like 'Initiated' in network rules or ParentCommandLine in process ones.
  How to do that? if unset or set to empty value, it keeps default field name

For this purpose you should use the DropDetectionItemTransformation as done in this pipeline.

* To add an expression depending  on rule type, it's the rule_conditions? for example, to say network are type=connect

The condition can be added with a AddConditionTransformation, like done here. I think rule type means log source in your context. Then this can be achieved with rule conditions as done in the same example few lines below.

* For full-text search, openobserve has a FTS option but only for fields configured for it.
  As an alternate option, is there a way to match that to a given field depending on logsource?
  For example, product=linux and service=sshd could translate to journald unit=ssh and journald MESSAGE field (body_message in my setup).
  Same question for a default value. if product=linux, use journald MESSAGE unless more granular match.

Generally you can use the FieldMappingTransformation to map fields to others. From pySigma 1.0 it will also be possible to map keyword searches to particular fields. Generally, you can use the rule condition LogSourceCondition to match on the log source specified in the rule.

Is there a community review process? right place to discuss it? Don't see any pointers on https://sigmahq.io/docs/digging-deeper/backends.html

There's no formal review process for backends. I do some quick review when a backend/pipeline project is added to the plugin directory to ensure nothing harmful or disfunctional is added here and less focused on quality. Feel free to reach out in the discussion, I'm happy to support with a deeper review if wanted!

[juju4]

Thanks @thomaspatzke ! I would happily take a review. Still some underlying backend issue (Alert import) ongoing.
I added a jupyter notebook as a base demo/documentation https://github.com/juju4/pySigma-backend-openobserve/tree/main/docs.

On other points
For initiated, code added but I believe the pipeline py part is not used when calling sigma cli sigma convert -t openobserve -p ../pySigma-backend-openobserve/sigma/pipelines/openobserve/kunai.yml ../sigma/rules/linux/process_creation/proc_creation_lnx_doas_execution.yml so how to drop it in this case?
Same for AddConditionTransformation

[thomaspatzke]

You shouldn't use YAML-based pipeline definitions in a backend project. Everything can also be expressed in the Python pipeline defintion. There's an autodiscovery for Python pipelines in the processing pipeline resolver that should find kunai_pipeline and expose it with the pipeline identifier kunai (derived from the function name). From the CLI it can then be used with -p kunai.

[juju4]

yaml definition was a good starter. python pipeline updated and working with example in docs/ notebook.

it was not working for file_event type. tried to add logsource_linux_file_event like the logsource_windows_file_event but nok as not defined. only logsource_linux_file_create in https://github.com/SigmaHQ/pySigma/blob/main/sigma/pipelines/common.py#L274. worked fine if change in rule logsource.category from file_event to file_create. either one of the update.

Comments welcomed. Thanks

[thomaspatzke]

These functions are just shortcuts, you can also define the LogsourceCondition directly in your pipeline.