[Feature Proposal] Helm Chart Overhaul and Update Documentation for Helm v3

[heisenbergs-uncertainty]

Apache StreamPipes version

dev (current development state)

Affected StreamPipes components

Backend, Processing Elements, Connect, UI

What happened?

tldr: Helm Chart is in need of a major overhaul.

Issue:

Helm Chart is 2 years old, based on Helm v3, and is not in a state that would be acceptable in Production Environments. It requires a lot of tinkering to work out of the box on a local setup and is missing lots of functionality.

Proposal:

I’d like to propose major updates to the Helm chart located in the installer/k8s directory of the Apache StreamPipes repository. These updates aim to improve functionality, configurability, and alignment with Helm and Kubernetes best practices, enhancing deployment flexibility and supporting production use cases. Below is a detailed breakdown of the proposed changes, their motivation, implementation steps, expected outcomes, and additional context.

Proposed Changes

1. Refactor values.yaml with Helm v3 Best Practices

   • Restructure the file to enhance readability.
   • Add detailed comments to clarify configuration options and usage.

2. Dependencies Update

   • Replace simple deployment files with proper Helm chart dependencies for external software components, improving maintainability and alignment with Helm conventions.

3. Add Kafka Kraft Capability

   • Integrate support for Kafka Kraft to leverage its advanced features within StreamPipes deployments.

4. Enhance Configurability

   • Introduce extensive and detailed configuration options in values.yaml with sensible default values to cater to both new and experienced users.
   • Expand configuration options for chart dependencies, allowing finer control over their behavior.

5. Improve Security

   • Implement security best practices, such as:
     • Running containers as non-root users.
     • Applying Pod Security Standards to ensure compliance with Kubernetes security guidelines.

6. Enhance Scalability

   • Optimize the chart to better handle scaling of StreamPipes components and dependencies, supporting larger and more complex deployments.

7. Update Documentation

   • Update the Helm chart README with comprehensive usage instructions, examples, and explanations of new features to improve user adoption and experience.

8. Add Testing

   • Develop a test suite to validate the chart's functionality, configurability, and reliability, ensuring robustness across various deployment scenarios.

9. Improve Extensibility

   • Add the ability to configure connections to external instances of dependencies, allowing users to use existing services instead of deploying them via the chart. This reduces deployment overhead and increases flexibility.

How to reproduce?

helm install streampipes streampipes/installer/k8s

Expected behavior

The updated Helm chart will offer:

• Improved readability and maintainability through a refactored values.yaml.
• Seamless integration with external software via Helm chart dependencies.
• Enhanced Kafka capabilities with Kraft support.
• Extensive configuration options with defaults that simplify usage for new users while allowing advanced customization.
• Strengthened security posture through best practices.
• Better scalability for handling larger deployments.
• Comprehensive documentation and examples for easier adoption.
• Robust testing to ensure reliability.
• Flexibility to connect to external services, reducing deployment overhead.

Additional technical information

Environment

• Development and Testing: Minikube, Kind, Rancher-Desktop

Development Setup

• M3 Macbook/aarch64
• docker
• kubernetes

Are you willing to submit a PR?

Yes, I am willing to submit a PR!