Trouble setting it up

[yzislin]

Hello,
I am having really hard time getting teleport to setup with just simple SSH and App connectivity in the AWS.
I have deployed teleport (auth and proxy) service on the primary node. Added admin user. Then on client node, I am trying to deploy Node and App roles. From time to time Node role (aka ssh) works but I cannot get App one to work.
Primary node is behind AWS Application Load Balancer (so I can utilize AWS signed cert). Communication between client nodes and master is via Security groups (aka using private IPs).

The problem where I seem to get stuck at are the TLS certs. On primary node, I get these errors:
2022-06-13T06:59:07Z WARN [ALPN:PROX] Failed to handle client connection. error:[ ERROR REPORT: Original Error: *tls.permanentError remote error: tls: bad certificate Stack Trace: /go/src/github.com/gravitational/teleport/lib/srv/alpnproxy/proxy.go:376 github.com/gravitational/teleport/lib/srv/alpnproxy.(*Proxy).handleConn /go/src/github.com/gravitational/teleport/lib/srv/alpnproxy/proxy.go:314 github.com/gravitational/teleport/lib/srv/alpnproxy.(*Proxy).Serve.func1 /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit User Message: remote error: tls: bad certificate] alpnproxy/proxy.go:322

On client node, I get these:
2022-06-13T07:03:25Z DEBU [PROC:1] Failed to connect to Auth Server directly. auth-addrs:[teleport.mydomain.com:443] error:[ ERROR REPORT: Original Error: *trace.ConnectionProblemError Get "https://teleport.cluster.local/v2/domain": x509: certificate is valid for teleport.mydomain.com, not 6964656e746974795f736369656e6365.teleport.cluster.local Stack Trace: /go/src/github.com/gravitational/teleport/lib/httplib/httplib.go:146 github.com/gravitational/teleport/lib/httplib.ConvertResponse /go/src/github.com/gravitational/teleport/lib/auth/clt.go:293 github.com/gravitational/teleport/lib/auth.(*Client).Get /go/src/github.com/gravitational/teleport/lib/auth/clt.go:384 github.com/gravitational/teleport/lib/auth.(*Client).GetDomainName /go/src/github.com/gravitational/teleport/lib/auth/clt.go:1543 github.com/gravitational/teleport/lib/auth.(*Client).GetLocalClusterName /go/src/github.com/gravitational/teleport/lib/service/connect.go:974 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClientDirect /go/src/github.com/gravitational/teleport/lib/service/connect.go:869 github.com/gravitational/teleport/lib/service.(*TeleportProcess).newClient /go/src/github.com/gravitational/teleport/lib/service/connect.go:165 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connect /go/src/github.com/gravitational/teleport/lib/service/connect.go:125 github.com/gravitational/teleport/lib/service.(*TeleportProcess).connectToAuthService /go/src/github.com/gravitational/teleport/lib/service/connect.go:64 github.com/gravitational/teleport/lib/service.(*TeleportProcess).reconnectToAuthService /go/src/github.com/gravitational/teleport/lib/service/service.go:2156 github.com/gravitational/teleport/lib/service.(*TeleportProcess).registerWithAuthServer.func1 /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:521 github.com/gravitational/teleport/lib/service.(*LocalService).Serve /go/src/github.com/gravitational/teleport/lib/service/supervisor.go:269 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1 /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit User Message: Get "https://teleport.cluster.local/v2/domain": x509: certificate is valid for teleport.mydomain.com, not 6964656e746974795f736369656e6365.teleport.cluster.local] service/connect.go:892 2022-06-13T07:03:25Z DEBU [PROC:1] Failed to connect to Auth Server through tunnel. auth-addrs:[teleport.mydomain.com:443] error:[Get "https://teleport.cluster.local/v2/domain": ssh: handshake failed: ssh: overflow reading version string] service/connect.go:893 2022-06-13T07:03:25Z ERRO [PROC:1] "Node failed to establish connection to cluster: Failed to connect to Auth Server directly or over tunnel, no methods remaining.\n\tGet \"https://teleport.cluster.local/v2/domain\": x509: certificate is valid for teleport.mydomain.com, not 6964656e746974795f736369656e6365.teleport.cluster.local, Get \"https://teleport.cluster.local/v2/domain\": ssh: handshake failed: ssh: overflow reading version string." service/connect.go:86 2022-06-13T07:03:33Z DEBU [PROC:1] Retrying connection to auth server after waiting 47.193185623s. service/connect.go:98 2022-06-13T07:03:33Z DEBU [PROC:1] Connected state: never updated. service/connect.go:145 2022-06-13T07:03:33Z INFO [PROC:1] Connecting to the cluster mycluster with TLS client certificate. service/connect.go:164 2022-06-13T07:03:33Z DEBU [PROC:1] Attempting to connect to Auth Server directly. auth-addrs:[teleport.mydomain.com:443] service/connect.go:867 2022-06-13T07:03:33Z DEBU [PROC:1] Failed to connect to Auth Server directly. auth-addrs:[teleport.mydomain.com:443] service/connect.go:874 2022-06-13T07:03:33Z DEBU [PROC:1] Attempting to discover reverse tunnel address. auth-addrs:[teleport.mydomain.com:443] service/connect.go:883 2022-06-13T07:03:33Z DEBU [PROC:1] Attempting to connect to Auth Server through tunnel. auth-addrs:[teleport.mydomain.com:443] service/connect.go:885 2022-06-13T07:03:33Z DEBU Attempting GET teleport.mydomain.com:443/webapi/find webclient/webclient.go:118 2022-06-13T07:03:33Z DEBU Attempting GET teleport.mydomain.com:443/webapi/find webclient/webclient.go:118 2022-06-13T07:03:33Z DEBU Attempting GET teleport.mydomain.com:443/webapi/find webclient/webclient.go:118 2022-06-13T07:03:33Z DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:276

I have tried all ports for auth_servers on client. 3025 seems to work with SSH role.

I've tried using private IPs for auth_server communication. It works for SSH (maybe works for App).

After generating app token and trying to start services, these errors above pop-up.

I am using insecure on both servers. I am using insecure_verify true in the app config. I have been fighting with this for days. Curious if someone can help me. Thank you.

[yzislin]

Funny. I just got it working after 4 minutes of posting the question. The solution is a hack, IMO.
I've hardcoded private IP address of teleport.mydomain.com in /etc/hosts for client node so communication goes on local IPs.

[webvictim]

There are a few reasons this isn't working:

1. For apps to work, you should use the IP/port of the Teleport proxy service (port 443/3080) rather than the Teleport auth service (port 3025).

2. In general you'll have a much easier time if you can present a legitimate TLS cert on the proxy (i.e. one from ACM or LetsEncrypt).

3. ALBs only work at layer 7 (HTTP/HTTPS) but Teleport's default configuration expects a layer 4 LB (TCP) - so you should probably use an NLB instead. You can still present the ACM cert, but you'll also need to disable multiplexing in Teleport's config and add some extra listeners on the LB.

If you share your Teleport config file (/etc/teleport.yaml) and a screenshot of your AWS load balancer config I can help you set it up.

[yzislin]

Actually, it seems to be working now. I am accessing proxy service over 443 and using ALB seems to work just fine.
I do however have one issue with ssh sessions. They get disconnected within like 1 minute of inactivity. I did set on auth_service client_idle_timeout: never but that didnt help. Not sure how to solve this problem.

And also, is there a way to have users use their SSH keys to connect via WebUI ssh connection?

Thanks.

[webvictim]

When you connect to an SSH session via Teleport's web UI it uses a short-lived certificate for the user internally, so there will be no need to add an additional key/certificate anywhere.

Try these settings in your Teleport config file:

# Determines the interval at which Teleport will send keep-alive messages.
# The default is set to 5 minutes (300 seconds) to stay lower than the
# common load balancer timeout of 350 seconds.
# keep_alive_count_max is the number of missed keep-alive messages before
# the server tears down the connection to the client.
keep_alive_interval: 50s
keep_alive_count_max: 3

[yzislin]

Perfect. It worked like a charm. Thanks.

[yzislin]

Another problem.
I am trying to connect to nodes via tsh command. It fails with cert error.
I run :
tsh -d ssh --insecure --proxy=teleport.mydomain.com:443 --user=john

End of the error is:
DEBU [KEYSTORE] Returning Teleport TLS certificate "/root/.tsh/keys/teleport.mydomain.com/john-x509.pem" valid until "2022-06-15 07:23:08 +0000 UTC". client/keystore.go:307 DEBU [HTTP:PROX] No proxy set in environment, returning direct dialer. proxy/proxy.go:276 ERROR REPORT: Original Error: x509.UnknownAuthorityError x509: certificate signed by unknown authority Stack Trace: /go/src/github.com/gravitational/teleport/lib/utils/proxy/proxy.go:69 github.com/gravitational/teleport/lib/utils/proxy.directDial.dialALPNWithDeadline /go/src/github.com/gravitational/teleport/lib/utils/proxy/proxy.go:106 github.com/gravitational/teleport/lib/utils/proxy.directDial.Dial /go/src/github.com/gravitational/teleport/lib/client/api.go:2602 github.com/gravitational/teleport/lib/client.makeProxySSHClientWithTLSWrapper /go/src/github.com/gravitational/teleport/lib/client/api.go:2553 github.com/gravitational/teleport/lib/client.makeProxySSHClient /go/src/github.com/gravitational/teleport/lib/client/api.go:2525 github.com/gravitational/teleport/lib/client.(*TeleportClient).connectToProxy /go/src/github.com/gravitational/teleport/lib/client/api.go:2445 github.com/gravitational/teleport/lib/client.(*TeleportClient).ConnectToProxy.func1 /opt/go/src/runtime/asm_amd64.s:1581 runtime.goexit User Message: x509: certificate signed by unknown authority

Thanks for your help.

[yt-2215]

Hello, I am also doing this experiment at present, a cloud server has built a community version of "teleport", an Intranet machine, and I want to use "teleport" to enable them to make two-way communication. I am currently stuck for several days because of the TLS certificate issue by myself, can you give me the configuration steps for reference?