Add Support for Multiple Kibana Security Detection Rule Types

[Copilot]

Implements comprehensive support for Kibana Security Detection Rules with multiple rule types beyond the initial query rules.

Fixes #523

Changes Made

Core Infrastructure

• Analyze current implementation and comments
• Replace !IsNull() && !IsUnknown() pattern with utils.IsKnown()
• Use utils.ListTypeAs and utils.ListValueFrom for list operations
• Move functions to be receivers on SecurityDetectionRuleData struct
• Use clients.CompositeIdFromStrFw for ID parsing
• Add proper UUID validation to prevent panics
• Re-use Read logic in Create and Update instead of parsing response directly
• Address all specific comment suggestions
• Test and validate changes
• All tests pass and linting is clean
• Documentation generated successfully

Multi-Rule Type Support

• Schema Enhancement: Updated schema to support all 8 detection rule types (query, eql, esql, machine_learning, new_terms, saved_query, threat_match, threshold) with their specific fields
• Rule Creation: Implemented create logic for all rule types with type-specific field handling
• Rule Updates: Implemented update logic for query and EQL rules (others show helpful error messages)
• Rule Reading: Enhanced read logic to handle query and EQL rules with improved error messages for unsupported types
• Helper Functions: Added utility functions for ID extraction and rule type handling across different rule structures

Rule Type Specific Fields Added

• EQL Rules: tiebreaker_field support
• Machine Learning Rules: anomaly_threshold, machine_learning_job_id
• New Terms Rules: new_terms_fields, history_window_start
• Saved Query Rules: saved_id support
• Threat Match Rules: threat_index, threat_query, threat_mapping, threat_filters, threat_indicator_path, concurrent_searches, items_per_search
• Threshold Rules: threshold configuration with field, value, and cardinality settings
• Common Fields: timeline_id, timeline_title, threat (MITRE ATT&CK framework support)

Current Status

• Fully Supported: Query and EQL rules (create, update, read)
• Create Only: ESQL, ML, New Terms, Saved Query, Threat Match, and Threshold rules
• Remaining Work: Update and read operations for the 6 remaining rule types

This implementation provides a solid foundation for all Kibana Security Detection rule types while maintaining backward compatibility and following established patterns in the codebase.

---

✨ Let Copilot coding agent set things up for you — coding agent works faster and does higher quality work when set up for your repo.

[cla-checker-service]

💚 CLA has been signed

[nick-benoit]

The trickiest part of this PR is sorting out rule_id vs id in the update case: https://www.elastic.co/docs/api/doc/kibana/operation/operation-updaterule

Update a detection rule using the rule_id or id field. The original rule is replaced, and all unspecified fields are deleted.

The api gives back this error if both are included:

│ API returned status 400: {"message":["both \"id\" and \"rule_id\" cannot exist, choose one or the other"],"status_code":400}```

If provided we will use the `rule_id` and unset the `id`. Otherwise we use the `id`.

[tobio]

Pretty excellent first pass here, added a bunch of comments around using some shared utils and how we've generally tried to structure things but this is looking good.

IMO we should cover all the rule types in this PR, but I could be convinced they can be added in a follow up. We do need to cover them to consider the issue 'done' though.

[tobio]

Looking at the acceptance test failures:

The 7.17 test is failing because an expected index doesn't exist yet. I guess it wouldn't be hard to create another 7.x specific test which also first created the index, but I'm not sure it's worthwhile given it's 7.x.

The 8.10 test looks like we're getting a different response from Kibana in that version. Definitely worth investigating, but given 8.10 is 2 years old I don't think it's unreasonable to skip the test in 8.10 and mention in the docs that the resource isn't compatible with 8.10.

[nick-benoit]

@copilot Please support all detection rule types defined in https://www.elastic.co/docs/api/doc/kibana/group/endpoint-security-detections-api

This will require adding updating the schema in internal/kibana/security_detection_rule/schema.go and corresponding methods in internal/kibana/security_detection_rule/models.go (eg toCreateProps, etc). Additionally instead of always using SecurityDetectionsAPIQueryRuleCreateProps to create and SecurityDetectionsAPIQueryRuleUpdateProps to update we will need choose the appropriate query rule type based on the type of rules we are using.

[Copilot]

@copilot Please support all detection rule types defined in https://www.elastic.co/docs/api/doc/kibana/group/endpoint-security-detections-api

This will require adding updating the schema in `intern...

I'm implementing support for all detection rule types. I've updated the schema to support all 8 rule types (query, eql, esql, machine_learning, new_terms, saved_query, threat_match, threshold) with their specific fields, and I'm working on updating the models to handle creation and updates for each type. This is a significant change that requires updating the create, update, and read logic for all rule types. I'll commit the progress once the implementation is complete and tested.

[tobio]

LGTM with one minor nit. Happy to get those addressed in a follow up PR though if you wanted to get this merged/released.

[tobio]

The following few functions return diags that will always be empty. We should likely just remove the return value altogether.