Is your feature request related to a problem?
A detector can potentially be configured with thousands of rules. Rules are used to construct the query that's executed by the detector to generating finds/alerts. If a query becomes too large, the execution of the query could fail with ParseException. E.g.,
ParseException[Query string length exceeds max allowed length 32000 (search.query.max_query_string_length); actual length: 32920];
Increasing the indices.query.bool.max_clause_count and search.query.max_query_string_length cluster settings can resolve those errors, but this is not a good long-term solution as large queries can negatively impact cluster performance (e.g., causing out-of-memory errors by consuming large amounts of heap memory, increasing search latency by taking a long time to parse and complete large queries, etc.). This would likely require scaling up/out the cluster to compensate for the performance issues. This is a limitation of opensearch itself, not so much the plugin.
What solution would you like?
It would be helpful for the plugin to intelligently warn when a detector query is becoming too large so that users can take appropriate action (e.g., adjust the 2 cluster settings mentioned above, auditing rules to make them more succinct/efficient, create more detectors with fewer rules, etc.).
This could be accomplished through the plugin UI as part of a dry-run process when creating/updating a detector.
It would also be helpful for the detectors to generate error alerts that call out the parse exception.
Do you have any additional context?
Ref# V2112755895
Is your feature request related to a problem?
A detector can potentially be configured with thousands of rules. Rules are used to construct the query that's executed by the detector to generating finds/alerts. If a query becomes too large, the execution of the query could fail with
ParseException. E.g.,Increasing the
indices.query.bool.max_clause_countandsearch.query.max_query_string_lengthcluster settings can resolve those errors, but this is not a good long-term solution as large queries can negatively impact cluster performance (e.g., causing out-of-memory errors by consuming large amounts of heap memory, increasing search latency by taking a long time to parse and complete large queries, etc.). This would likely require scaling up/out the cluster to compensate for the performance issues. This is a limitation of opensearch itself, not so much the plugin.What solution would you like?
It would be helpful for the plugin to intelligently warn when a detector query is becoming too large so that users can take appropriate action (e.g., adjust the 2 cluster settings mentioned above, auditing rules to make them more succinct/efficient, create more detectors with fewer rules, etc.).
This could be accomplished through the plugin UI as part of a dry-run process when creating/updating a detector.
It would also be helpful for the detectors to generate error alerts that call out the parse exception.
Do you have any additional context?
Ref# V2112755895