Add compaction, rules, dynamic configurations, nativeSpec

[jmcwilliams-te]

Description

I have a Druid cluster managed with this operator and wanted to change compaction settings, retention rules, dynamic configurations, and define ingestion specs with native kubernetes manifests instead of json to enable further configuration with Infrastructure as Code.

These settings are incorporated into existing Druid and DruidIngestion CRs. Common API logic has been moved into a common module in pkg/ for access across the operator. This includes URL building and Auth logic.

Compaction settings and dynamic configurations are updated based on GET requests to the Druid API rather than storing expected settings in the status section of cluster objects. This ensures settings are properly applied even if the Druid cluster state changes or if the Druid cluster doesn't properly persist the values.

Documentation has been added for the new features mentioned but also for the existing authentication configuration logic.

---

This PR has:

• been tested on a real K8S cluster to ensure creation of a brand new Druid cluster works.
• been tested for backward compatibility on a real K*S cluster by applying the changes introduced here on an existing Druid cluster. If there are any backward incompatible changes then they have been noted in the PR description.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added documentation for new or modified features or behaviors.

---

Key changed/added files in this PR

• apis/druid/v1alpha1/druidingestion_types.go
• controllers/druid/druid_controller.go
• controllers/druid/dynamic_config.go
• controllers/ingestion/reconciler.go
• pkg/druidapi/druidapi.go
• pkg/util/util.go

[AdheipSingh]

Thank you, this is really great and helpful.

[AdheipSingh]

where ever emitEventGeneric is called can you use constants.
Refer to https://github.com/datainfrahq/druid-operator/blob/master/controllers/druid/interface.go#L25