Migrating from SQLite to DynamoDB using TCTL

[pnrao1983]

This guide details a step-by-step workflow on how to migrate from SQLite to DynamoDB using the TCTL utility, focusing on a standalone Teleport instance using local directory (SQLite) migrating to a DynamoDB backend. While the example provided is specific, the principles and steps can be adapted to various backend migration scenarios.

Step 1: Preparing for Migration

The migration process starts with preparing your existing Teleport instance. The primary goal here is to create a comprehensive backup of your current cluster state, while ensuring that no data is lost during the transition.

1. Backup Data

• Execute the command tctl get all --with-secrets to obtain a complete backup if you have access to the Auth Server. Alternatively, if you have a cronjob set up to copy this output regularly, use that data.
• Redirect the output to a file named state.yaml .e.g: tctl get all --with-secrets > state.yaml

Note:
It is not possible to take tctl get all --with-secrets output remotely. We must take this from auth server host (VM or a pod) or use an Identity file.

• Additionally, copy the following files to your new Teleport instance: /etc/teleport.yaml, license.pem, and HTTPS key pairs (fullchain.pem and privkey.pem) to the new instance that will be created in step 2 below.

Note:

Do not store private files like the license.pem file and HTTPS key pairs in /var/lib/teleport as this folder is sometimes removed for troubleshooting purposes. Alternatively, you can optionally create a /etc/teleport directory for this purpose.

Step 2: Setting Up the New Teleport Instance

With your backup ready, it's time to set up the new Teleport instance.

1. Log into the New Instance:

• Install the required version of Teleport Binary.
• Attach the necessary IAM policies for DynamoDB, Route53, and S3 to a role, and attach the role to your EC2 host.

2. Configure Teleport

• Update /etc/teleport.yaml with new values such as region, table_name, audit_events_uri, and audit_sessions_uri. Here is a sample configuration:

Here is a sample teleport.yaml config file:

teleport:
  nodename: nara-cluster-rebuild
  advertise_ip: 10.0.9.63
  log:
    output: stderr
    severity: INFO
  data_dir: /var/lib/teleport
  storage:
    type: dynamodb
    region: us-west-1
    table_name: nara-cluster-rebuild
    audit_events_uri: ["dynamodb://nara-cluster-rebuild-events", "file:///var/lib/teleport/audit/events"]
    audit_sessions_uri: s3://nara-cluster-rebuild.pnrao.rocks/records

auth_service:
  enabled: yes
  keep_alive_interval: 1m
  keep_alive_count_max: 3
  listen_addr: 0.0.0.0:3025
  proxy_listener_mode: multiplex
  authentication:
    type: local
    local_auth: true
    second_factor: "off"
    webauthn:
      rp_id: example.teleport.com
  cluster_name: nara-cluster
  web_idle_timeout: 0
ssh_service:
  enabled: yes
  listen_addr: 0.0.0.0:3022
  labels:
     name: actinium
proxy_service:
  enabled: yes
  listen_addr: 0.0.0.0:3023
  tunnel_listen_addr: 0.0.0.0:443
  web_listen_addr: 0.0.0.0:443
  public_addr: example.teleport.com:443
  ssh_public_addr: example.teleport.com:3023
  tunnel_public_addr: example.teleport.com:443
  https_keypairs:
    - cert_file: /etc/teleport/fullchain.pem
      key_file: /etc/teleport/privkey.pem
  kubernetes:
    enabled: yes
    listen_addr: 0.0.0.0:3026
    public_addr: ['example.teleport.com:3026']
  mysql_listen_addr: "0.0.0.0:3036"

Step 3: Bootstrap and Launch

1. Bootstrap:

• Execute the bootstrap command: teleport start --config /etc/teleport.yaml bootstrap state.yaml.
• Once bootstrapping is complete, stop the process with CTRL+C

2. Start Teleport Services:

• Enable and start the Teleport service, while also verifying the status of service.

systemctl enable teleport
systemctl start teleport
systemctl status teleport
journalctl -fu teleport

An example output of the Teleport status:

[root@ip-10-0-9-63 ~]# systemctl status teleport
● teleport.service - Teleport Service
     Loaded: loaded (/usr/lib/systemd/system/teleport.service; disabled; preset: disabled)
     Active: active (running) since Fri 2024-06-14 17:43:26 UTC; 5s ago
   Main PID: 10754 (teleport)
      Tasks: 8 (limit: 1114)
     Memory: 322.7M
        CPU: 3.091s
     CGroup: /system.slice/teleport.service
             └─10754 /usr/local/bin/teleport start --config /etc/teleport.yaml --pid-file=/run/teleport.pid

Jun 14 17:43:32 ip-10-0-9-63.us-west-1.compute.internal teleport[10754]: 2024-06-14T17:43:32Z INFO [UPLOAD:1]  Creating directory. pid:10754.1 directory:/var/lib/teleport/log service/service.go:3083
Jun 14 17:43:32 ip-10-0-9-63.us-west-1.compute.internal teleport[10754]: 2024-06-14T17:43:32Z INFO [UPLOAD:1]  Creating directory. pid:10754.1 directory:/var/lib/teleport/log/upload service/service.g>
Jun 14 17:43:32 ip-10-0-9-63.us-west-1.compute.internal teleport[10754]: 2024-06-14T17:43:32Z INFO [UPLOAD]    uploader will scan /var/lib/teleport/log/upload/streaming/default every 5s filesessions/>
Jun 14 17:43:32 ip-10-0-9-63.us-west-1.compute.internal teleport[10754]: 2024-06-14T17:43:32Z INFO [UPLOAD:1:] upload completer will run every 5m0s events/complete.go:161

Step 4: Update DNS Records
Update your DNS records to reflect the new IP address. This step ensures that all resources can join the new cluster seamlessly.

Step 5: Verify that resources are visible

Log in with your existing credentials and verify that the resources are visible:

[root@ip-10-0-9-63 ~]# tsh ls
Enter password for Teleport user admin:
Enter an OTP code from a device:
Node Name        Address          Labels
---------------- ---------------- ---------------------------------------------------------------------------------
nara-auto-enroll ⟵ Tunnel         hostname=nara-auto-enroll,aws/IEName=Nara,aws/Name=nara-auto-enroll1,aws/env=prod
nara-cluster     10.0.131.85:3022 name=actinium,aws/IEName=Nara,aws/Name=nara-cluster
nara-services    ⟵ Tunnel         hostname=nara-services,aws/IEName=Nara,aws/Name=nara-services
nara-ubuntu      ⟵ Tunnel         hostname=ip-10-0-132-11,aws/IEName=Nara,aws/Name=nara-ubuntu,aws/env=prod

Troubleshooting Common Issues
During the migration, you might encounter several issues. Here’s how to resolve them:

1. Dynamo DB Access Denied Errors:

ERROR: initialization failed
	AccessDeniedException: User: arn:aws:sts::278576220453:assumed-role/nara-cluster/i-001b0af2c5335e4e3 is not authorized to perform: dynamodb:DescribeTable on resource: arn:aws:dynamodb:us-west-1:278576220453:table/nara-cluster-rebuild because no identity-based policy allows the dynamodb:DescribeTable action
	status code: 400, request id: B6APDI5CNSGN5J72V8UFCCQK6BVV4KQNSO5AEMVJF66Q9ASUAAJG

• If you see IAM policy errors related to DynamoDB, update the IAM policies accordingly to grant the necessary permissions.

2. S3 Bucket Access Denied Errors:

2024-06-12T15:30:57Z �INFO [S3]        Setting up bucket "nara-cluster-rebuild.pnrao.rocks", sessions path "/records" in region "us-west-1". s3sessions/s3handler.go:220
ERROR: initialization failed
	Access Denied
ERROR REPORT:
Original Error: *errors.errorString context canceled
Stack Trace:
	github.com/gravitational/teleport/e/lib/plugins/manager.go:209 github.com/gravitational/teleport/e/lib/plugins.(*Manager).runInner
	github.com/gravitational/teleport/e/lib/plugins/manager.go:154 github.com/gravitational/teleport/e/lib/plugins.(*Manager).Run
	github.com/gravitational/teleport/e/lib/plugins/register.go:40 github.com/gravitational/teleport/e/lib/plugins.RegisterPluginManager.func1
	github.com/gravitational/teleport/lib/service/supervisor.go:587 github.com/gravitational/teleport/lib/service.(*LocalService).Serve
	github.com/gravitational/teleport/lib/service/supervisor.go:312 github.com/gravitational/teleport/lib/service.(*LocalSupervisor).serve.func1
	runtime/asm_amd64.s:1650 runtime.goexit

• Fix the S3 bucket issue by creating a new one or updating the existing IAM to make the bucket for you.

3. PEM-Encoded Block Errors:

ERROR: initialization failed
	refusing to bootstrap backend
		expected PEM-encoded block

Suppose you run into a PEM-encoded block error shown below during the initialization or bootstrap process. Here is how you can resolve the issue:

• Ensure that cryptographic certificates in the state.yaml are correctly formatted in PEM format and have not been tampered with.

• If the above has been confirmed, manually remove the kind: cert_authority block entries for all trusted clusters from the state.yaml file. More information on this can be found [(here)]

• There is a bug filed to fix the leaf cluster issue: https://github.com/gravitational/teleport/issues/9547

Conclusion:

While migrating backends is a critical task, It can be streamlined with careful planning and execution. By following these steps and troubleshooting tips, you can ensure a smooth transition and minimal downtime.

[dboslee]

@pnrao1983 #41866 adds the command teleport backend clone which allows for cloning backend data from a source to a destination backend granted the process is able to connect and authenticate with both backends.

The recommended steps are as follows:

1. Stop the Teleport Auth service
2. Run teleport backend clone -c clone.yaml. The following is an example of the clone.yaml configuration file:

  # src is the configuration for the backend where data is cloned from.
  src: 
    type: sqlite
    path: /var/lib/teleport_data
  # dst is the configuration for the backend where data is cloned to.
  dst:
    type: dynamodb
    region: us-east-1
    table: teleport_backend
  # parallel is the amount of backend data cloned in parallel.
  # If a clone operation is taking too long consider increasing this value.
  parallel: 100
  # force, if set to true, will continue cloning data to a destination
  # regardless of whether data is already present. By default this is false
  # to protect against overwriting the data of an existing Teleport cluster.
  force: false

3. Update the Teleport Auth service's configuration to point to the new backend.
4. Start the Teleport Auth service.