yq-eval

A Kubernetes Resource Model (KRM) function that evaluates yq expressions on Kubernetes manifests.

Overview

yq-eval allows you to transform Kubernetes resources using yq expressions defined in annotations. This provides a powerful way to manage environment-specific configurations and perform complex manifest manipulations that are difficult with Kustomize alone.

Key capabilities include:

• Full yq Expression Support: Leverage any yq v4 expression for sophisticated resource transformations.
• Dynamic Configuration: Inject values from ConfigMaps and Secrets as environment variables for easy string interpolation and configuration.
• Cross-Resource References: Modify a resource based on the properties of another using the $resourceList variable, enabling advanced automation.
• Non-Invasive by Design: Transformations are defined in annotations, keeping your base manifests clean and portable.

Quickstart

1. Create the manifests you want to transform

configmap.yaml – update fields inside a JSON blob:

apiVersion: v1
kind: ConfigMap
metadata:
  name: myapp
  annotations:
    yq-eval: | # Update JSON with env vars.
      .data."myapp.json" |= (fromjson
        | .database.host = strenv(DB_HOST)
        | .database.port = env(DB_PORT)
        | .cache.enabled = env(CACHE_ENABLED)
        | tojson)
data:
  myapp.json: |
    {
      "database": {
        "host": "localhost",
        "port": 5432
      },
      "cache": {
        "enabled": false
      }
    }

ingress.yaml – substitute environment variables anywhere in the manifest:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
  annotations:
    yq-eval: | # Substitute env vars in all string values.
      .. | select(tag == "!!str") |= envsubst(nu)
spec:
  rules:
    - host: myapp.${DOMAIN}
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: myapp
                port:
                  name: http
  tls:
    - hosts:
        - myapp.${DOMAIN}
      secretName: myapp-tls

2. Wire everything together with Kustomize

Create kustomization.yaml that generates the variables and runs the function:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - configmap.yaml
  - ingress.yaml

generatorOptions:
  disableNameSuffixHash: true # Do not append suffix hash to local-config.

configMapGenerator:
  - name: local-config
    literals:
      - DOMAIN=mydomain.example
      - DB_HOST=postgres.myapp.svc.cluster.local
      - DB_PORT=5432
      - CACHE_ENABLED=true
    options:
      annotations:
        config.kubernetes.io/local-config: "true" # Do not include local-config in output.

transformers:
 - | # Evaluate yq expression in each resource using env vars from local-config.
    apiVersion: t13a.com/v1alpha1
    kind: YqEval
    metadata:
      annotations:
        config.kubernetes.io/function: |
          container:
            image: ghcr.io/t13a/yq-eval:latest
      name: yq-eval
    envFrom:
      - configMapRef:
          name: local-config

3. Build with Kustomize

kustomize build --enable-alpha-plugins .

Result:

apiVersion: v1
data:
  myapp.json: |
    {
      "database": {
        "host": "postgres.myapp.svc.cluster.local",
        "port": 5432
      },
      "cache": {
        "enabled": true
      }
    }
kind: ConfigMap
metadata:
  name: myapp
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: myapp
spec:
  rules:
  - host: myapp.mydomain.example
    http:
      paths:
      - backend:
          service:
            name: myapp
            port:
              name: http
        path: /
        pathType: Prefix
  tls:
  - hosts:
    - myapp.mydomain.example
    secretName: myapp-tls

The JSON payload inside the ConfigMap and the Ingress hostname are both updated from the same set of environment variables. Switching environments is just a matter of changing the literals in local-config.

Examples

The following examples demonstrate common use cases for yq-eval. Each example is a self-contained, runnable Kustomize project.

• Quick Start: A minimal working example demonstrating basic variable substitution.
• Kustomize Base/Overlays: A complete example showing integration with a typical Kustomize base/overlays pattern.
• Ingress Hostname: Dynamically construct Ingress hostnames from environment variables.
• ConfigMap Data Transformation: Transform structured data (e.g., JSON) within a ConfigMap.
• Secret Data Manipulation: Update selected fields in a base64-encoded Secret.
• Cross-Resource References: Modify a resource based on another resource's properties (e.g., PV and PVC).
• Init Container Script: Safely substitute variables into shell scripts within Pods.

Usage

There are two ways to use yq-eval: as a container (recommended) or as a local script.

As a container (recommended)

This is the easiest and most portable way to use the function. Simply reference the container image in your function definition:

apiVersion: t13a.com/v1alpha1
kind: YqEval
metadata:
  annotations:
    config.kubernetes.io/function: |
      container:
        image: ghcr.io/t13a/yq-eval:latest
  name: yq-eval
envFrom:
  - configMapRef:
      name: local-config

As a local script

If you are developing the script or prefer to run it locally, you can use the exec function.

1. Clone the repository:

   git clone https://github.com/t13a/krm-function-yq-eval.git

2. Reference the script path:

   apiVersion: t13a.com/v1alpha1
   kind: YqEval
   metadata:
     annotations:
       config.kubernetes.io/function: |
         exec:
           # Adjust the path to the script
           path: /path/to/krm-function-yq-eval/src/yq-eval.sh
     name: yq-eval
   envFrom:
     - configMapRef:
         name: local-config

   Note: This method requires the --enable-exec flag in Kustomize.

Function configuration

The behavior of the yq-eval function is controlled by a custom resource of kind: YqEval. This resource is not deployed to the cluster but serves as configuration for the function.

apiVersion: t13a.com/v1alpha1
kind: YqEval
metadata:
  name: yq-eval
envFrom:
  - configMapRef:
      name: my-config
      namespace: my-namespace # Optional: defaults to any namespace
  - secretRef:
      name: my-secret
      namespace: my-namespace # Optional: defaults to any namespace
annotation:
  key: my-custom-annotation # Optional: defaults to `yq-eval`

Configuration fields

• envFrom (optional, array): A list of sources to load environment variables from.

  • configMapRef: Reference to a ConfigMap.
    • name (string): The name of the ConfigMap.
    • namespace (string, optional): The namespace of the ConfigMap. If omitted, only matches by name.
  • secretRef: Reference to a Secret.
    • name (string): The name of the Secret.
    • namespace (string, optional): The namespace of the Secret. If omitted, only matches by name. Data values are automatically Base64-decoded.

• annotation.key (optional, string): The annotation key that contains the yq expression.

  • Defaults to yq-eval.

Execution flow

The function processes the list of resources provided by Kustomize or kpt according to the following steps:

1. Load Environment Variables: The function first scans the input ResourceList to find the ConfigMaps and Secrets specified in the envFrom field of the function configuration. The key-value pairs from these resources are loaded as environment variables.

2. Identify Target Resources: It then iterates through all resources in the ResourceList, looking for resources that have the specified annotation (default: yq-eval).

3. Evaluate yq Expression: For each target resource found:

   • The yq expression from the annotation value is executed against that resource.
   • During evaluation, two special variables are available:
     • $resourceList: The entire ResourceList object, allowing for cross-resource lookups.
     • Environment variables loaded in step 1 are accessible via env() and envsubst().
   • The resource is replaced in-place with the result of the yq expression.

4. Clean Up Annotation: The annotation used for the transformation is removed from the resource to keep the final output clean.

5. Return Transformed Resources: The function outputs the final, modified ResourceList.

Tips and tricks

Using with for readability

When you need to apply multiple transformations, use the with operator to separate concerns:

annotations:
  yq-eval: |
    with(.. | select(tag == "!!str"); . |= envsubst(nu))
    | with(.spec.replicas; . |= tonumber)
    | with(.metadata.labels; .environment = env(ENVIRONMENT))

This is much more readable than chaining everything with | operators.

Working with structured data in ConfigMaps

You can parse, modify, and serialize JSON/YAML/TOML/XML data:

yq-eval: |
  with(.data."config.json"; . |= (fromjson | .key = "value" | tojson))
  | with(.data."config.yaml"; . |= (fromyaml | .key = "value" | toyaml))
  | with(.data."config.toml"; . |= (from_toml | .key = "value" | to_toml))
  | with(.data."config.xml"; . |= (from_xml | .key = "value" | to_xml))

Working with encoded data in Secrets

You can decode, modify, and re-encode base64-encoded data in Secrets:

yq-eval: |
  with(.data."myapp-secret.json"; . |= (
    sub("\n", "") # Remove newlines.
    | @base64d
    | fromjson
    | .key = "value"
    | tojson
    | @base64
    | sub("(.{70})", "${1}\n") # Re-insert newlines every 70 chars.
  ))

Conditional transformations

yq supports conditional updates using with and select:

yq-eval: |
  with(select(env(ENABLE_TLS) == "true"); .spec.tls = [{"hosts": [env(HOSTNAME)], "secretName": "tls-cert"}])

Limitations

• The container function requires the --enable-alpha-plugins flag in Kustomize. The exec function additionally requires --enable-exec.
• Secret values are base64 decoded automatically, which may not be desired in all cases.
• Cross-resource references via $resourceList can be complex for large resource sets.

Contributing

Contributions are welcome! Please feel free to submit issues and pull requests.

License

MIT License. See LICENSE for details.