[Draft] Kibana config overwrites

docs/howto/custom_kibana_config.md

@@ -0,0 +1,185 @@
+# Custom Kibana Configuration
+
+This document explains how to add custom configuration to Kibana when using elastic-package.
+
+## Overview
+
+You can provide additional Kibana configuration that will be appended to the base configuration generated by elastic-package. This allows you to:
+
+- Override default settings
+- Add custom plugins configuration
+- Set up custom security settings
+- Configure additional features
+
+## Setup
+
+1. Create your custom configuration file:
+   ```bash
+   # Create ~/.elastic-package/profiles/default/kibana-custom.yml
+   touch ~/.elastic-package/profiles/default/kibana-custom.yml
+   ```
+
+2. Add your custom configuration:
+   ```yaml
+   # Example custom configuration
+   logging.loggers:
+     - name: plugins.security
+       level: debug
+
+   server.customResponseHeaders:
+     X-Custom-Header: "MyValue"
+
+   # Template variables are supported
+   xpack.security.enabled: {{ if eq .security_enabled "true" }}true{{ else }}false{{ end }}
+   ```
+
+## Template Support
+
+Your custom configuration supports the same templating as the base configuration:
+
+- `{{ fact "kibana_version" }}` - Kibana version
+- `{{ fact "username" }}` - Elasticsearch username  
+- `{{ fact "password" }}` - Elasticsearch password
+- `{{ fact "apm_enabled" }}` - APM enabled flag
+- `{{ fact "self_monitor_enabled" }}` - Self monitoring enabled flag
+- All other facts available in the base template
+
+### Available Template Variables
+
+The following template variables are available in your custom configuration:
+
+| Variable | Description | Example |
+|----------|-------------|---------|
+| `kibana_version` | Version of Kibana | `8.11.0` |
+| `elasticsearch_version` | Version of Elasticsearch | `8.11.0` |
+| `agent_version` | Version of Elastic Agent | `8.11.0` |
+| `username` | Elasticsearch username | `elastic` |
+| `password` | Elasticsearch password | `changeme` |
+| `kibana_host` | Kibana host URL | `https://kibana:5601` |
+| `elasticsearch_host` | Elasticsearch host URL | `https://elasticsearch:9200` |
+| `fleet_url` | Fleet server URL | `https://fleet-server:8220` |
+| `apm_enabled` | APM enabled flag | `true`/`false` |
+| `logstash_enabled` | Logstash enabled flag | `true`/`false` |
+| `self_monitor_enabled` | Self monitoring flag | `true`/`false` |
+
+## Configuration Precedence
+
+1. Base elastic-package configuration (from template)
+2. Your custom configuration (appended)
+
+Since YAML allows duplicate keys and Kibana processes them in order, your custom settings will override base settings with the same key.
+
+## Examples
+
+### Enable Debug Logging
+```yaml
+logging.loggers:
+  - name: root
+    level: debug
+  - name: plugins.fleet
+    level: trace
+```
+
+### Custom Security Settings
+```yaml
+xpack.security.session.idleTimeout: "8h"
+xpack.security.session.lifespan: "24h"
+xpack.security.authc.providers:
+  basic.basic1:
+    order: 0
+```
+
+### Development Features
+```yaml
+server.rewriteBasePath: false
+server.dev.basePathProxyTarget: 3000
+
+# Enable experimental features
+xpack.fleet.enableExperimental:
+  - customIntegrations
+  - agentTamperProtectionEnabled
+```
+
+### Custom UI Settings
+```yaml
+uiSettings:
+  overrides:
+    "theme:darkMode": true
+    "dateFormat": "YYYY-MM-DD HH:mm:ss.SSS"
+    "discover:sampleSize": 1000
+```
+
+### Template Usage Example
+```yaml
+# Use template variables in your custom config
+server.name: kibana-{{ fact "kibana_version" }}
+server.customResponseHeaders:
+  X-Kibana-Version: "{{ fact "kibana_version" }}"
+
+# Conditional configuration based on features
+{{ if eq (fact "apm_enabled") "true" }}
+elastic.apm.active: true
+elastic.apm.serverUrl: "http://fleet-server:8200"
+elastic.apm.environment: "development"
+{{ end }}
+
+# Version-specific configuration
+{{ if semverLessThan (fact "kibana_version") "8.0.0" }}
+# Legacy configuration for older versions
+xpack.monitoring.ui.container.elasticsearch.enabled: true
+{{ else }}
+# Modern configuration for newer versions
+monitoring.ui.container.elasticsearch.enabled: true
+{{ end }}
+```
+
+## Troubleshooting
+
+### Configuration Not Applied
+- Verify the `kibana-custom.yml` file exists in your profile directory
+- Check that the YAML syntax is valid
+- Ensure the file is in the correct location: `~/.elastic-package/profiles/{profile_name}/kibana-custom.yml`
+
+### Template Errors
+- Use `{{ fact "variable_name" }}` syntax for template variables
+- Ensure template functions like `semverLessThan` are used correctly
+- Check the elastic-package logs for template parsing errors
+
+### Kibana Startup Issues
+- Validate your custom configuration against Kibana documentation
+- Start with minimal changes and add complexity gradually
+- Check Kibana logs for configuration validation errors
+
+## Profile-Specific Configurations
+
+You can have different custom configurations for different profiles:
+
+```bash
+# Development profile with debug settings
+~/.elastic-package/profiles/development/kibana-custom.yml
+
+# Production profile with optimized settings  
+~/.elastic-package/profiles/production/kibana-custom.yml
+```
+
+Custom configuration is automatically detected and applied for each profile when the `kibana-custom.yml` file exists.
+
+## Best Practices
+
+1. **Start Simple**: Begin with basic configuration changes and add complexity gradually
+2. **Use Comments**: Document your custom configurations for future reference
+3. **Test Changes**: Verify that Kibana starts successfully after configuration changes
+4. **Version Compatibility**: Use template conditions for version-specific settings
+5. **Backup Configurations**: Keep copies of working configurations before making changes
+
+## Limitations
+
+- Custom configuration is appended to the base configuration (not merged at the YAML structure level)
+- Template processing errors will prevent stack startup
+- Some Kibana settings may require specific ordering or dependencies
+
+## Related Documentation
+
+- [Kibana Configuration Settings](https://www.elastic.co/guide/en/kibana/current/settings.html)
+- [elastic-package Profiles](../README.md#profiles)
+- [Stack Configuration](../README.md#stack-configuration)

internal/stack/kibana_config.go

@@ -0,0 +1,107 @@
+// Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+// or more contributor license agreements. Licensed under the Elastic License;
+// you may not use this file except in compliance with the Elastic License.
+
+package stack
+
+import (
+	"bytes"
+	"fmt"
+	"html/template"
+	"io"
+	"os"
+
+	"github.com/elastic/go-resource"
+
+	"github.com/elastic/elastic-package/internal/profile"
+)
+
+// kibanaConfigWithCustomContent generates kibana.yml with custom config appended
+func kibanaConfigWithCustomContent(profile *profile.Profile) func(resource.Context, io.Writer) error {
+	return func(ctx resource.Context, w io.Writer) error {
+		// First, generate the base kibana.yml from template
+		var baseConfig bytes.Buffer
+		baseTemplate := staticSource.Template("_static/kibana.yml.tmpl")
+		err := baseTemplate(ctx, &baseConfig)
+		if err != nil {
+			return fmt.Errorf("failed to generate base kibana config: %w", err)
+		}
+
+		// Write base config to output
+		_, err = w.Write(baseConfig.Bytes())
+		if err != nil {
+			return fmt.Errorf("failed to write base kibana config: %w", err)
+		}
+
+		// Check if custom config file exists
+		customConfigPath := profile.Path(KibanaCustomConfigFile)
+		customConfigData, err := os.ReadFile(customConfigPath)
+		if os.IsNotExist(err) {
+			return nil // No custom config file, that's fine
+		}
+		if err != nil {
+			return fmt.Errorf("failed to read custom kibana config: %w", err)
+		}
+
+		// Add separator comment
+		_, err = w.Write([]byte("\n\n# Custom Kibana Configuration\n"))
+		if err != nil {
+			return fmt.Errorf("failed to write custom config separator: %w", err)
+		}
+
+		// Process custom config as template
+		customTemplate, err := template.New("kibana-custom").
+			Funcs(templateFuncs).
+			Parse(string(customConfigData))
+		if err != nil {
+			return fmt.Errorf("failed to parse custom kibana config template: %w", err)
+		}
+
+		// Create template data from resource context facts
+		templateData := createTemplateDataFromContext(ctx)
+
+		err = customTemplate.Execute(w, templateData)
+		if err != nil {
+			return fmt.Errorf("failed to execute custom kibana config template: %w", err)
+		}
+
+		return nil
+	}
+}
+
+// createTemplateDataFromContext creates template data from resource context
+// This function extracts commonly used facts and makes them available to templates
+func createTemplateDataFromContext(ctx resource.Context) map[string]interface{} {
+	data := make(map[string]interface{})
+
+	// List of facts that should be available in custom templates
+	factNames := []string{
+		"kibana_version",
+		"elasticsearch_version",
+		"agent_version",
+		"username",
+		"password",
+		"kibana_host",
+		"elasticsearch_host",
+		"fleet_url",
+		"apm_enabled",
+		"logstash_enabled",
+		"self_monitor_enabled",
+		"kibana_http2_enabled",
+		"logsdb_enabled",
+		"elastic_subscription",
+		"geoip_dir",
+		"agent_publish_ports",
+		"api_key",
+		"enrollment_token",
+	}
+
+	// Extract facts from context
+	for _, factName := range factNames {
+		if value, found := ctx.Fact(factName); found {
+			data[factName] = value
+		}
+	}
+
+	return data
+}

internal/stack/resources.go

@@ -34,6 +34,9 @@ const (
 	// KibanaConfigFile is the kibana config file.
 	KibanaConfigFile = "kibana.yml"
 
+	// KibanaCustomConfigFile is the custom kibana config file.
+	KibanaCustomConfigFile = "kibana-custom.yml"
+
 	// LogstashConfigFile is the logstash config file.
 	LogstashConfigFile = "logstash.conf"
 
@@ -189,7 +192,21 @@ func applyResources(profile *profile.Profile, stackVersion string) error {
 	resourceManager.RegisterProvider("file", &resource.FileProvider{
 		Prefix: stackDir,
 	})
-	resources := append([]resource.Resource{}, stackResources...)
+	// Create kibana resource with custom config support
+	kibanaResource := &resource.File{
+		Path:    KibanaConfigFile,
+		Content: kibanaConfigWithCustomContent(profile),
+	}
+
+	// Replace the kibana resource in stackResources with our custom one
+	resources := make([]resource.Resource, 0, len(stackResources))
+	for _, res := range stackResources {
+		if file, ok := res.(*resource.File); ok && file.Path == KibanaConfigFile {
+			resources = append(resources, kibanaResource)
+		} else {
+			resources = append(resources, res)
+		}
+	}
 
 	// Keeping certificates in the profile directory for backwards compatibility reasons.
 	resourceManager.RegisterProvider(CertsFolder, &resource.FileProvider{