[Draft] Kibana config overwrites

[ruflin]

This changes comes from the fact that there is a very fast pace on new Kibana config options popping up. Some are feature flags, others are for advanced settings to be overwritten. Last, there are also credentials for connectors that need to be used from times to times. When building Kibana from source, kibana.dev.yml can be used, but I would really like to try some of these features in elastic-package and always have it enabled. My current go to solution was to build my own elastic-package binary, but that is not sustainable.

This PR introduces the option to put a kibana-custom.yml configuration file into the profile. These configs are appended to what is already configured.

Opening this in draft mode, as I'm not fully happy yet with the change, here a few points:

• Initially I had a config option that needed to be enabled to load the kibana-custom.yml file. I don't think this is needed, so I changed it to always be picked up if the file exists
• Naming of the file: Should we call it kibana.dev.yml or similar?
• Usage of template variables: That is part of what the AI generated and it is kind of nice that template variables can be used too, but is it overkill? Do we really need that?
• Having this, will make some of the hard coded configs obsolete? Do we eventually need this for other services too?

Before doing a full code review, lets discuss these points.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 0c4b7a8

History

• 💔 Build #6193 failed 1349eb6

[jsoriano]

This PR introduces the option to put a kibana-custom.yml configuration file into the profile. These configs are appended to what is already configured.

The overall idea sounds good to me, I think we could make this part of the compose provider.

I would still add a recommendation in the docs about requesting first-level support for options that are frequently modified in the config. Maybe we should expose something for feature flags for example.

Initially I had a config option that needed to be enabled to load the kibana-custom.yml file. I don't think this is needed, so I changed it to always be picked up if the file exists

+1, I see this as an advanced configuration, and users should know what they do when they add this file 🙂

Though I would show a warning if this file exists, in case a user is having issues with the custom configuration and forgets about having it.

Naming of the file: Should we call it kibana.dev.yml or similar?

kibana-custom.yml sounds good to me, but maybe kibana.dev.yml is better as is the name already used in other places. Not a strong opinion, as you prefer.

Usage of template variables: That is part of what the AI generated and it is kind of nice that template variables can be used too, but is it overkill? Do we really need that?

Yeah, I wonder if we need templating at all in kibana-custom.yml. I think we can start without templating, and add it later if needed.
Other option would be to append the file to the template before resolving it, and have the same functions as the original template has.

Having this, will make some of the hard coded configs obsolete? Do we eventually need this for other services too?

We could add this also at least for elasticsearch and elastic-agent, but lets start with kibana and we can decide later about other services.