This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.
This helm chart is a lightweight way to configure and run our official Logstash docker image
- This repo includes a number of example configurations which can be used as a reference. They are also used in the automated testing of this chart
- Automated testing of this chart is currently only run against GKE (Google Kubernetes Engine).
- The chart deploys a statefulset and by default will do an automated rolling update of your cluster. It does this by waiting for the cluster health to become green after each instance is updated. If you prefer to update manually you can set
updateStrategy: OnDelete - It is important to verify that the JVM heap size in
logstashJavaOptsand to set the CPU/Memoryresourcesto something suitable for your cluster - We have designed this chart to be very un-opinionated about how to configure Logstash. It exposes ways to set environment variables and mount secrets inside of the container. Doing this makes it much easier for this chart to support multiple versions with minimal changes.
logstash.ymlconfiguration files can be set either by a ConfigMap usinglogstashConfiginvalues.ymlor by environment variables usingextraEnvsinvalues.yml, however Logstash Docker image can't mix both methods as defining settings with environment variables causeslogstash.ymlto be modified in place while using ConfigMap bind-mount the same file (more details in this Note).
- Add the elastic helm charts repo
helm repo add elastic https://helm.elastic.co - Install it
helm install --name logstash elastic/logstash
- Clone the git repo
git clone [email protected]:elastic/helm-charts.git - Install it
helm install --name logstash ./helm-charts/logstash
This chart is tested with the latest supported versions. The currently tested versions are:
| 6.x | 7.x |
|---|---|
| 6.8.5 | 7.5.0 |
Examples of installing older major versions can be found in the examples directory.
While only the latest releases are tested, it is possible to easily install old or new releases by overriding the imageTag. To install version 7.5.0 of Logstash it would look like this:
helm install --name logstash elastic/logstash --set imageTag=7.5.0
| Parameter | Description | Default |
|---|---|---|
antiAffinity |
Setting this to hard enforces the anti-affinity rules. If it is set to soft it will be done "best effort". Other values will be ignored. | hard |
antiAffinityTopologyKey |
The anti-affinity topology key. By default this will prevent multiple Logstash nodes from running on the same Kubernetes node | kubernetes.io/hostname |
extraContainers |
Templatable string of additional containers to be passed to the tpl function |
"" |
extraEnvs |
Extra environment variables which will be appended to the env: definition for the container |
[] |
extraInitContainers |
Templatable string of additional init containers to be passed to the tpl function |
"" |
extraVolumes |
Templatable string of additional volumes to be passed to the tpl function |
"" |
extraVolumeMounts |
Templatable string of additional volumeMounts to be passed to the tpl function |
"" |
image |
The Logstash docker image | docker.elastic.co/logstash/logstash |
imagePullPolicy |
The Kubernetes imagePullPolicy value | IfNotPresent |
imagePullSecrets |
Configuration for imagePullSecrets so that you can use a private registry for your image | [] |
imageTag |
The Logstash docker image tag | 7.5.0 |
extraInitContainers |
Templatable string of additional init containers to be passed to the tpl function |
"" |
httpPort |
The http port that Kubernetes will use for the healthchecks and the service. | 9600 |
labels |
Configurable labels applied to all Logstash pods | {} |
lifecycle |
Allows you to add lifecycle configuration. See values.yaml for an example of the formatting. | {} |
livenessProbe |
Configuration fields for the livenessProbe | failureThreshold: 3initialDelaySeconds: 300periodSeconds: 10successThreshold: 3timeoutSeconds: 5 |
logstashConfig |
Allows you to add any config files in /usr/share/logstash/config/ such as logstash.yml and log4j2.properties. See values.yaml for an example of the formatting. |
{} |
logstashJavaOpts |
Java options for Logstash. This is where you should configure the jvm heap size | -Xmx1g -Xms1g |
logstashPipeline |
Allows you to add any pipeline files in /usr/share/logstash/pipeline/. |
{} |
maxUnavailable |
The maxUnavailable value for the pod disruption budget. By default this will prevent Kubernetes from having more than 1 unhealthy pod in the node group | 1 |
nodeAffinity |
Value for the node affinity settings | {} |
nodeSelector |
Configurable nodeSelector so that you can target specific nodes for your Logstash cluster | {} |
persistence.annotations |
Additional persistence annotations for the volumeClaimTemplate |
{} |
persistence.enabled |
Enables a persistent volume for Logstash data | false |
podAnnotations |
Configurable annotations applied to all Logstash pods | {} |
podManagementPolicy |
By default Kubernetes deploys statefulsets serially. This deploys them in parallel so that they can discover each other | Parallel |
podSecurityContext |
Allows you to set the securityContext for the pod | fsGroup: 1000runAsUser: 1000 |
podSecurityPolicy |
Configuration for create a pod security policy with minimal permissions to run this Helm chart with create: true. Also can be used to reference an external pod security policy with name: "externalPodSecurityPolicy" |
create: falsename: "" |
priorityClassName |
The name of the PriorityClass. No default is supplied as the PriorityClass must be created first. | "" |
readinessProbe |
Configuration fields for the readinessProbe | failureThreshold: 3initialDelaySeconds: 60periodSeconds: 10successThreshold: 3timeoutSeconds: 5 |
replicas |
Kubernetes replica count for the statefulset (i.e. how many pods) | 1 |
resources |
Allows you to set the resources for the statefulset | requests.cpu: 100mrequests.memory: 1536Milimits.cpu: 1000mlimits.memory: 1536Mi |
schedulerName |
Name of the alternate scheduler | "" |
secretMounts |
Allows you easily mount a secret as a file inside the statefulset. Useful for mounting certificates and other secrets. See values.yaml for an example | [] |
securityContext |
Allows you to set the securityContext for the container | capabilities.drop:[ALL]runAsNonRoot: truerunAsUser: 1000 |
terminationGracePeriod |
The terminationGracePeriod in seconds used when trying to stop the pod | 120 |
tolerations |
Configurable tolerations | [] |
updateStrategy |
The updateStrategy for the statefulset. By default Kubernetes will wait for the cluster to be green after upgrading each pod. Setting this to OnDelete will allow you to manually delete each pod during upgrades |
RollingUpdate |
volumeClaimTemplate |
Configuration for the volumeClaimTemplate for statefulsets. You will want to adjust the storage (default 30Gi) and the storageClassName if you are using a different storage class |
accessModes: [ "ReadWriteOnce" ]resources.requests.storage: 1Gi |
rbac |
Configuration for creating a role, role binding and service account as part of this helm chart with create: true. Also can be used to reference an external service account with serviceAccountName: "externalServiceAccountName". |
create: falseserviceAccountName: "" |
In examples/ you will find some example configurations. These examples are used for the automated testing of this helm chart
To deploy a cluster with all default values and run the integration tests
cd examples/default
make
The recommended way to install plugins into our docker images is to create a custom docker image.
The Dockerfile would look something like:
ARG logstash_version
FROM docker.elastic.co/logstash/logstash:${logstash_version}
RUN bin/logstash-plugin install logstash-output-kafka
And then updating the image in values to point to your custom image.
There are a couple reasons we recommend this.
- Tying the availability of Logstash to the download service to install plugins is not a great idea or something that we recommend. Especially in Kubernetes where it is normal and expected for a container to be moved to another host at random times.
- Mutating the state of a running docker image (by installing plugins) goes against best practices of containers and immutable infrastructure.
This chart uses pytest to test the templating logic. The dependencies for testing can be installed from the requirements.txt in the parent directory.
pip install -r ../requirements.txt
make pytest
You can also use helm template to look at the YAML being generated
make template
It is possible to run all of the tests and linting inside of a docker container
make test
Integration tests are run using goss which is a serverspec like tool written in golang. See goss.yaml for an example of what the tests look like.
To run the goss tests against the default example:
cd examples/default
make goss