Scaleway Cilium + Hubble TLS issues

[khaddict]

Scaleway Cilium + Hubble TLS issues

1. Missing TLS Certificate Generation (incorrect cronJob value)

When deploying Hubble with the scaleway-cilium-hubble chart, the
secret hubble-relay-client-certs is never created:

Warning  FailedMount  MountVolume.SetUp failed for volume "tls" :
secret "hubble-relay-client-certs" not found

Resolution:

In Scaleway's default values.yaml:

cilium:
  hubble:
    tls:
      auto:
        method: cronjob   # incorrect

The Cilium upstream chart requires:

method: cronJob   # correct (capital J)

Fixing the value to cronJob correctly triggers certificate generation.

---

2. hubble-relay cannot connect to hubble-peer

Once certificates are generated, hubble-relay fails to connect:

rpc error: code = Unavailable desc = connection error:
dial tcp 10.32.15.117:443: connect: connection refused

hubble-peer correctly exposes port 443 → 4244, but the Cilium agent never starts its Hubble gRPC server on port 4244 because the TLS certificates are not mounted in the cilium-agent (even though the volume exists).

Upstream Cilium mounts:

volumeMounts:
- name: hubble-tls
  mountPath: /var/lib/cilium/tls/hubble
  readOnly: true

This mount is missing from Scaleway's managed CNI DaemonSet.

Without it:

• certificates are not accessible,
• Hubble gRPC server does not start,
• hubble-relay cannot connect → connection refused.

---

3. Manual fix

You can patch the DaemonSet manually:

kubectl patch daemonset cilium -n kube-system   --type='json'   -p='[
    {
      "op": "add",
      "path": "/spec/template/spec/containers/0/volumeMounts/-",
      "value": {
        "name": "hubble-tls",
        "mountPath": "/var/lib/cilium/tls/hubble",
        "readOnly": true
      }
    }
  ]'

After this patch:

• cilium-agent loads certificates,
• Hubble gRPC server starts on port 4244,
• hubble-peer and hubble-relay become functional.

---

4. Required fixes for Scaleway

A. In scaleway-cilium-hubble values.yaml

Fix the TLS autogen method:

- method: cronjob
+ method: cronJob

B. In the Scaleway-managed Cilium DaemonSet

Add missing mount:

volumeMounts:
- name: hubble-tls
  mountPath: /var/lib/cilium/tls/hubble
  readOnly: true

This matches Cilium upstream behavior and is required for Hubble TLS to work.

---

5. Impact Summary

Without these fixes:

• Hubble relay fails,
• Hubble UI cannot connect,
• Hubble observability is non-functional on Scaleway-managed clusters.

With both fixes applied, Hubble works correctly on Cilium 1.34.2.

[khaddict]

#42

[nox-404]

The cronjob is already distributed, as for the missing secret we will rollout a fix this week.
This was a regression starting from 1.33.

[khaddict]

The cronjob is already distributed, as for the missing secret we will rollout a fix this week. This was a regression starting from 1.33.

Thanks for the quick reply.
What about the missing /var/lib/cilium/tls/hubble volume mount in the Cilium DaemonSet?
Without this mount, even if the hubble-relay-client-certs secret is created, Hubble can not start.
Regards,

[nox-404]

it's part of this week's fix

[nox-404]

Let's keep that issue open until we rollout the fix. You can then test it out and close it if it's fine by you.

[khaddict]

The first point is fixed, but the mount is still not present on the cilium DaemonSet which causes the issue:

If I patch it by myself with:

kubectl patch daemonset cilium -n kube-system   --type='json'   -p='[
    {
      "op": "add",
      "path": "/spec/template/spec/containers/0/volumeMounts/-",
      "value": {
        "name": "hubble-tls",
        "mountPath": "/var/lib/cilium/tls/hubble",
        "readOnly": true
      }
    }
  ]'

Everything works:

[nox-404]

you need to upgrade the cluster for it to take effect
(you can upgrade to the same version you are in to just trigger the manifest update)

[khaddict]

you need to upgrade the cluster for it to take effect (you can upgrade to the same version you are in to just trigger the manifest update)

I tried on a new cluster deployed this morning (1.34.2), not an old one.

[nox-404]

I'll try to look at it this week

[nox-404]

Hey,
I noticed we had a regression on that volume mount on 1.34.x with cilium in tunnel mode (default is vxlan), a patch will be pushed to prod next week.

Sorry for the delay 🙏

[khaddict]

@nox-404 Looks like it's fixed, thanks a lot for your time ! :)