Proposal: new modifier case sensitive

[vandir]

As stated in the specification, string values are case sensitive by default. I think it could be helpful to have a new modifier (insensitive or just i) to search string values for case insensitive matching, that is something like this:

detection:
    rule1:
        fieldA|i: value_a
    condition: rule1

It could also be chained with re modifier to simulate the presence of flag re.IGNORECASE , and obiusly with contains, startswith, endswith, utf16le and utf16be. In many of my use cases, the presence of such modifier is critical.

If you think this could be useful I can add it and make a PR.

[Neo23x0]

The specification states the opposite. All string values are always case-INsenstive. Only Regex (|re) values a treated case-sensitive and thus represent a workaround when you want strings to match case-sensitive: https://github.com/SigmaHQ/sigma-specification/blob/main/Sigma_1_0_2.md#general

[vandir]

I don't know how I read exactly the opposite of what was written (and I've even read it several times 🤦‍♂️), sorry for that, I've updated the title.

Anyway the problem still remains for anyone needs a way to express both search matching (sensitive and insensitive) with no or partial (i.e. Elasticsearch) regular expression support . The logic is the opposite of what proposed in my first message: the new modifier could be something like |sensitive.

I'm writing a custom backend (which I wish to publish in the next weeks, after internal testing) to convert Sigma rules to Elasticsearch (pure DSL) queries. In Elastic the field type flattened supports case sensitive and insensitive matching but no regular expressions. This type is critical for many of our indices.

[frack113]

We already had a discussion about this a few months ago.
It was found that it could be useful to have this type of modification but it would represent 0.001% of the cases.
For example, an APT that would use a valid name but with a different cast, like if you put in system32 a "Conhost.exe" next to "conhost.exe".
But it can already be detected by other means.
So this is a lot of work and be a source of incomprehension for almost no gain.

The idea is not to modify the sigma specifications to fit a particular product.

It is a elasticsearch backend trouble.
The elasticsearch backend need a way to select the mapping according to the modifier used.

[vandir]

We already had a discussion about this a few months ago. It was found that it could be useful to have this type of modification but it would represent 0.001% of the cases. For example, an APT that would use a valid name but with a different cast, like if you put in system32 a "Conhost.exe" next to "conhost.exe". But it can already be detected by other means.

I understand what you are saying and I agree that the use case is not that widespread. But a high level query language that relies on regular expressions to support such a basic search it's quite strange for at least two reasons:

• Even when supported, regex are expensive
• There are products which don't support regular expressions.

So this is a lot of work and be a source of incomprehension for almost no gain.

I disagree here, because:

1. It's not a lot of work. A simple and clean solution would be to add a new modifier sensitive which populate a new member field case_sensitivity on ConditionFieldEqualsValueExpression class. Then we could split convert_condition_field_eq_val_str to convert_condition_field_eq_val_str_sensitive and convert_condition_field_eq_val_str_insensitive and let the specific backends to decide how to convert the condition. The modification thus described is simple and clean and I think it's now the best time to do it because Sigma 2.0 is currently on progress, and, at the moment, there are few backends which support pySigma.

2. It's not a source of incomprehension since a new modifier is not a breaking change and I think it's clear what it does.

It is a elasticsearch backend trouble. The elasticsearch backend need a way to select the mapping according to the modifier used.

The issue is about the fact that we miss a modifier to let the backend decide which search to apply. Moreover, the backend you quoted does not support case INsensitive searches and pySigma does not notify in any way this fact.

Anyway this is just a proposal because we are investing on writing a new backend for Elasticsearch that fix the limitations of the project you have linked, but we have very large (with many nested and flattened fields) indices and we need to support both searches to fully integrate Sigma in our processes.

[Neo23x0]

that relies on regular expressions to support such a basic search

We have 2000+ rules in the repo and AFAIK not a single rule making use of regular expressions to make sure the search is case-sensitive.

I'd say, we are fine.

We've always followed a minimalistic approach and introduced new features or extended the specification whenever we noticed a general demand for it.

In Elastic the field type flattened supports case sensitive and insensitive matching but no regular expressions. This type is critical for many of our indices.

How many rules that required case-sensitive matching do you maintain?
Could you give us an example?

I ask because I wouldn't want to maintain a modifier that is only theoretically relevant.

[vandir]

I wanted to drop a quick message to say thanks for adding the functionality I suggested here. It's awesome to see it implemented now!

I appreciate that you and the team took the time to consider my idea, even though it was initially marked as unnecessary. While I'm a bit bummed that it wasn't given a chance earlier, I'm still happy to see it included in the project.