cannot execute shell or view logs of pods that run on rancher cluster

[RobertoRamacciatiViseca]

Describe the bug

cannot execute shell or view logs of pods that run on rancher cluster

To Reproduce

Steps to reproduce the bug:

1. Go To a Pod
2. [Click on terminal/shell button or show log button
3. "Failed to connect…Press the enter key to reconnect." is printed

Environment (please provide info about your environment):

• Installation type: Windows executable
• Headlamp Version: version 0.29
• Other: k8s cluster is rancher
• kubectl logs with same kubeconfig works as expected

[sniok]

Hi, thanks for opening an issue! Could you provide logs from the console to see if there are any errors?

[jjaaskel]

I have this same issue (hence in WSL2 with Ubuntu 24.04), logs like this:

`server process stderr: 2025/03/07 19:21:18 http: proxy error: context canceled

server process stderr: 2025/03/07 19:21:18 http: proxy error: context canceled

server process stderr: 2025/03/07 19:21:19 http: proxy error: context canceled

server process stderr: {"level":"error","source":"/home/runner/work/headlamp/headlamp/backend/cmd/multiplexer.go","line":341,"error":"websocket: bad handshake","time":"2025-03-07T19:21:34+02:00","message":"dialing WebSocket"}
{"level":"error","clusterID":"tn-devint","UserID":"a62d79daf5ce9089","source":"/home/runner/work/headlamp/headlamp/backend/cmd/multiplexer.go","line":487,"error":"dialing WebSocket: websocket: bad handshake","time":"2025-03-07T19:21:34+02:00","message":"establishing cluster connection"}
{"level":"error","clusterID":"tn-devint","source":"/home/runner/work/headlamp/headlamp/backend/cmd/multiplexer.go","line":522,"time":"2025-03-07T19:21:34+02:00","message":"establishing cluster connection"}

server process stderr: {"level":"error","source":"/home/runner/work/headlamp/headlamp/backend/cmd/multiplexer.go","line":341,"error":"websocket: bad handshake","time":"2025-03-07T19:21:34+02:00","message":"dialing WebSocket"}
{"level":"error","UserID":"a62d79daf5ce9089","clusterID":"tn-devint","source":"/home/runner/work/headlamp/headlamp/backend/cmd/multiplexer.go","line":487,"error":"dialing WebSocket: websocket: bad handshake","time":"2025-03-07T19:21:34+02:00","message":"establishing cluster connection"}
{"level":"error","clusterID":"tn-devint","source":"/home/runner/work/headlamp/headlamp/backend/cmd/multiplexer.go","line":522,"time":"2025-03-07T19:21:34+02:00","message":"establishing cluster connection"}

server process stderr: {"level":"error","source":"/home/runner/work/headlamp/headlamp/backend/cmd/multiplexer.go","line":341,"error":"websocket: bad handshake","time":"2025-03-07T19:21:34+02:00","message":"dialing WebSocket"}
{"level":"error","clusterID":"tn-devint","UserID":"a62d79daf5ce9089","source":"/home/runner/work/headlamp/headlamp/backend/cmd/multiplexer.go","line":487,"error":"dialing WebSocket: websocket: bad handshake","time":"2025-03-07T19:21:34+02:00","message":"establishing cluster connection"}
{"level":"error","clusterID":"tn-devint","source":"/home/runner/work/headlamp/headlamp/backend/cmd/multiplexer.go","line":522,"time":"2025-03-07T19:21:34+02:00","message":"establishing cluster connection"}`

[carlosalbertorg]

Hi, Headlamp is amazing!

I have this same issue on rancher cluster to load terminal:

Failed to connect…

Press the enter key to reconnect.

And for and for show logs of pod too. Displays RECONNECT message.

Thanks for all!

[joaquimrocha]

We will disable the multiplexer for the upcoming version so it shall work in cases like this, and then re-enable in the main branch so we debug and fix it. Thanks.

[Adrianos712]

Hello @joaquimrocha,

I have the same kind of issue: headlamp respond 502 to every websocket connection from the frontend.
It is really obvious when using log or shell, but even for all watch request from the frontend I have a 502 in the console and the following log in headlamp pod:

http: proxy error: No valid id-token, and cannot refresh without refresh-token

which I can't find in the source code (probably external lib).
I took tcpdump traces between ingress and headlamp and 502 is sent by headlamp. Ingress seems to work well with webSockets

I'm using the in-cluster installation with multi-cluster kubeconfig. My clusters are EKS but I'm not sure it really matters.
I tried helm releases 0.29.1, 0.28.0 and 0.27.0 (read somewhere that from 6 jan there was a websocket refacto) and same behaviour for all.

Interesting clue: everything is working well with the /c/main/... routes but as soon as I try the multi-cluster routes /c/<cluster>/... 502appears.

[jjaaskel]

Still same issue with headlamp 0.30.0, neither checking logs of pods or opening shell to pod are working..

[sniok]

I've looked into this and it seems that there's an issue with the way we connect to websockets that's only affecting rancher clusters.
Kubectl uses "Authorization: Bearer token" but we use the Sec-Websocket-Protocol: base64.binary.k8s.io,base64url.bearer.authorization.k8s.io.token header (because we're creating connection on the frontend and can't set headers for the websocket handshake)

There's an old issue in the rancher repository that seems to align with this theory rancher/rancher#20153

Logs from my tests

With protocol authorization

curl --include \
     --no-buffer \
     --insecure \
     --http1.1 \
     --header "Connection: Upgrade" \
     --header "Upgrade: websocket" \
     --header "Host: example.com:80" \
     --header "Origin: http://example.com:80" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
     --header "Sec-WebSocket-Version: 13" \
     --header "Sec-Websocket-Protocol: base64.binary.k8s.io,base64url.bearer.authorization.k8s.io.REDACTED TOKEN" \
     "https://localhost:8443/k8s/clusters/local/api/v1/pods?watch=1&resourceVersion=86621"

HTTP/1.1 401 Unauthorized
Cache-Control: no-cache, no-store, must-revalidate
Content-Type: text/plain; charset=utf-8
X-Api-Cattle-Auth: false
X-Content-Type-Options: nosniff
Date: Fri, 28 Mar 2025 16:17:21 GMT
Content-Length: 15

not authorized

but with a Authorization header it seems to work

curl --include \
     --no-buffer \
     --insecure \
     --http1.1 \
     --header "Connection: Upgrade" \
     --header "Upgrade: websocket" \
     --header "Host: example.com:80" \
     --header "Authorization: Bearer REDACTED TOKEN" \
     --header "Origin: http://example.com:80" \
     --header "Sec-WebSocket-Key: SGVsbG8sIHdvcmxkIQ==" \
     --header "Sec-WebSocket-Version: 13" \
     --header "Sec-Websocket-Protocol: base64.binary.k8s.io" \
     "https://localhost:8443/k8s/clusters/local/api/v1/pods?watch=1&resourceVersion=86621"


HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: qGEgH3En71di5rrssAZTmtRTyFk=
Sec-WebSocket-Protocol: base64.binary.k8s.io

kubectl works because it uses Authorization header

so I'm not sure what's the fix for this yet but I'll look more into this

[sniok]

Okay I found a possible solution, we need to use Authorization header and also replace user agent header because rancher has CORS-like check for websocket connections.

This is specific to rancher clusters and ideally of course protocol auth should be added on rancher side, and cors should be configurable, but since kubectl works we should also make headlamp work with this