Difference between tsh kube and tsh proxy kube

[Ezzahhh]

Not sure if it was appropriate to create an issue instead of a discussion: #49622

I'm observing an interesting difference when connecting through Teleport with tsh kube and tsh proxy kube when in conjunction with the k9s utility. Specifically, I get a lot of context deadline exceeded especially with the subjectaccessreviews call when going through tsh kube login and k9s often does not connect properly (even when increasing timeout). However, when using tsh proxy kube I do not have any such issue at all.

I see logs like this in my teleport-agent and nothing appears awry but k9s reports context deadline exceeded:

2024-12-12T08:41:14Z INFO [KUBERNETE] Round trip: POST https://100.64.0.1:443/apis/authorization.k8s.io/v1/selfsubjectaccessreviews, code: 201, duration: 42.182415ms tls:version: 304, tls:resume:false, tls:csuite:1301, tls:server:kube-teleport-proxy-alpn.teleport.cluster.local pid:7.1 reverseproxy/reverse_proxy.go:223

I do not experience any issues with kubectl with tsh kube. Is this an expected difference? I would not usually expect going through proxy and not to have any difference with tooling like k9s.

[Ezzahhh]

I would just add that enabling debug logs on the cluster side did not reveal that much. When I observe hangs or disconnects in k9s there are no accompanying logs in the proxy service. If k9s does eventually connect after a bunch of retries, I do see the round trip messages as shown in the original post although sometimes with higher durations (couple hundred ms). This behaviour is observable in two clusters that are connected to Teleport (the only two joined so far during testing).

[Ezzahhh]

Is anybody able to replicate this issue? Specifically in our case, we're using a L4 load balancer (AWS NLB) with multiplex (as mentioned in #49622). tsh kube login with k9s is very unreliable but kubectl appears to work fine. I don't understand why the behaviour between the two tools would be different? It's preferable to not need to open a local proxy with tsh proxy kube both for convenience and performance. I seem to be able to replicate this across all my clusters connected with Teleport so perhaps there is a misconfig issue on my end.

[Ezzahhh]

As mentioned here, I believe the issue is related to CGNAT implemented by my ISP. I don't face this issue if I use a VPN or move to a completely different network.