feat: ESQL query validation against Elastic cluster #4955
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
traut
commented
Aug 1, 2025
| @@ -1432,15 +1432,14 @@ def get_packaged_integrations( | |||
| # if both exist, rule tags are only used if defined in definitions for non-dataset packages | |||
| # of machine learning analytic packages | |||
|
|
|||
| rule_integrations = meta.get("integration", []) | |||
| if rule_integrations: | |||
| for integration in rule_integrations: | |||
There was a problem hiding this comment.
simple style fix, replacing if condition with a more robust default value condition via
rule_integrations = meta.get("integration") or []
traut
commented
Aug 1, 2025
| @@ -1754,7 +1753,7 @@ def parse_datasets(datasets: list[str], package_manifest: dict[str, Any]) -> lis | |||
| else: | |||
| package = value | |||
|
|
|||
| if package in list(package_manifest): | |||
| if package in package_manifest: | |||
traut
commented
Aug 1, 2025
traut
commented
Aug 1, 2025
detection_rules/rule_validators.py
Outdated
|
|
||
| log(f"Got query columns: {', '.join(query_column_names)}") | ||
|
|
||
| # FIXME: validate the dynamic columns |
There was a problem hiding this comment.
The columns returned from the cluster must be validated against the input mapping, and the dynamic fields checked for validity.
|
at the moment (before any field validation) the test marks 33 rules out of 75 as invalid. The tests were executed against a vanilla local The many errors are most probably because of the bugs in the code, so I expect the number of invalid rules to go down after those are fixed. full log |
|
Updated to include initial dynamic field validation. This will parse the schema(s) for dynamic fields and perform some initial formatting check. It checks if the field has a proper prefix as described in #4909, and if the field is based on a field that is present in the schema. However, additional validation will be needed if we want to validate the proper types for ES|QL function and operator return values. https://www.elastic.co/docs/reference/query-languages/esql/esql-functions-operators Additionally, a number of the errors seen in the above testing are due to schema updates that do not have the required fields. For instance. Next steps are:
Note after discussion with @Mikaayenson we determined that the sub-field of the dynamic query does not need to have ecs enforcement here. E.g. For |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Pull Request
Issue link(s):
Summary - What I changed
How To Test
.detection-rules-cfg.yml) or from the environment variablesChecklist
bug,enhancement,schema,maintenance,Rule: New,Rule: Deprecation,Rule: Tuning,Hunt: New, orHunt: Tuningso guidelines can be generatedmeta:rapid-mergelabel if planning to merge within 24 hoursContributor checklist