Teleport Proxy and Auth Service Migration

[Arrizki16]

Hi, I need help!

I am trying to move my proxy and auth service from an old server to a new server in the same region. I transferred the teleport data from my old server located at var/lib/teleport to the new server. This included roles, users, nodes, apps, database resources and session or events log. I also ensured that the configuration file /etc/teleport.yaml is set up exactly the same on the new server.

However, after the migration, all users are unable to log in, receiving the error message: "User Message: invalid username, password, or second factor." Additionally, the resources seem to be unregistered. What should I do? Thanks

I must create a new user and reset all users authentication to solve this issue temporary

[stevenGravy]

This sounds like the backend was not completely moved. Please share how you configured the auth service and any storage, was there any sep backend storage.

[Arrizki16]

Hi, heres my proxy and auth service yaml configuration, nothing special. the /var/lib/teleport was restored from old server.

version: v3
teleport:
  nodename: teleport-dev
  data_dir: /var/lib/teleport
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: ""
  diag_addr: ""
auth_service:
  enabled: "yes"
  listen_addr: 0.0.0.0:3025
  cluster_name: xxx.tubestream.dev
  proxy_listener_mode: multiplex
ssh_service:
  enabled: "yes"
proxy_service:
  enabled: "yes"
  web_listen_addr: 0.0.0.0:443
  public_addr: xxx.tubestream.dev:443
  https_keypairs: []
  https_keypairs_reload_interval: 0s
  acme:
    enabled: "yes"
    email: [email protected]

ssh and db node yaml configuration

version: v3
teleport:
  nodename: mititest
  data_dir: /var/lib/teleport
  join_params:
    token_name: c1e3xxx
    method: token
  proxy_server: xxx.tubestream.dev:443
  log:
    output: stderr
    severity: INFO
    format:
      output: text
  ca_pin: sha256:4980xxx
  diag_addr: ""
db_service:
  enabled: true
  databases:
  - name: mititest-db
    description: mongo db database for mititest server.
    protocol: "mongodb"
    uri: "mongodb://localhost:27017/"
    static_labels:
      env: development
auth_service:
  enabled: "no"
ssh_service:
  enabled: "yes"
  labels:
    teleport.internal/resource-id: xxx
    dynamic/env: development
  commands:
  - name: hostname
    command: [hostname]
    period: 1m0s
proxy_service:
  enabled: "no"
  https_keypairs: []
  https_keypairs_reload_interval: 0s
  acme: {}

[Arrizki16]

for now, i have to do additional steps below to get connected with ssh or db resources:

• create a new user using tsh to be able to go inside the cluster
• the token and ca value from the new server
• then remove the /var/lib/teleport directory in each node server
• renew mongo.cas and mongo.crt for database connection

but it really needs effort :)

[stevenGravy]

How did you move the /var/lib/teleport to the new server? I would compare the tctl get cert_authority. I expect it did not copy into the data directory cleanly. Please confirm there isn't a /var/lib/teleport/teleport directory on the new server or something that might have not copied correctly.

[Arrizki16]

using scp for /var/lib/teleport directory from old to new server. is there any additonal steps? i just know about tctl get cert_authority 🤔

[stevenGravy]

I expect that did not do a complete copy. I would see about using tar -cvf /var/lib/teleport or how you did a scp command. YOu can check other things such as roles, users. The cert_authority is the most critical as that allows existing agents and clients to reconnect.

[Arrizki16]

Yes, I mean use zip command first, then send it using scp command. I just checked in sqlite.db, there are some roles and users that are already added. Here's the structure of my /var/lib/teleport. But still invalid credentials. Perhaps the problem is around CA or something related to it but dunno what should i do.

teleport
├── acme
│   ├── acme_account+key
│   └── xxx.tubestream.dev
├── backend
│   └── sqlite.db
├── debug.sock
├── host_uuid
├── log
│   ├── 3862568b-8e50-4423-bd80-5cce922d5d4d
│   ├── events.log -> /var/lib/teleport/log/3862568b-8e50-4423-bd80-5cce922d5d4d/2025-01-27.00:00:00.log
│   ├── playbacks
│   ├── records
│   └── upload
└── proc
    └── sqlite.db

8 directories, 7 files

[stevenGravy]

yes if you retrieve the cert_authority on the new system it should be identical. I suspect files did not copy cleanly.