fix: restore dump from s3 (#25)

Author: agneum

README.md

@@ -1,192 +1,17 @@
 [[_TOC_]]
 
-# How to setup Database Lab using Terraform in AWS
+# Database Lab Terraform Module
 
 This [Terraform Module](https://www.terraform.io/docs/language/modules/index.html) is responsible for deploying the [Database Lab Engine](https://gitlab.com/postgres-ai/database-lab) to cloud hosting providers.
 
 Your source PostgreSQL database can be located anywhere, but DLE with other components will be created on an EC2 instance under your AWS account. Currently, only "logical" mode of data retrieval (dump/restore) is supported – the only available method for managed PostgreSQL cloud services such as RDS Postgres, RDS Aurora Postgres, Azure Postgres, or Heroku. "Physical" mode is not yet supported, but it will be in the future. More about various data retrieval options for DLE: https://postgres.ai/docs/how-to-guides/administration/data.
 
-## Supported Cloud Platforms:
+## Supported Cloud Platforms
 - AWS
 
-## Prerequisites
-- [AWS Account](https://aws.amazon.com)
-- [Terraform Installed](https://learn.hashicorp.com/tutorials/terraform/install-cli) (minimal version: 1.0.0)
-- AWS [Route 53](https://aws.amazon.com/route53/) Hosted Zone (For setting up TLS) for a domain or sub-domain you control
-- You must have AWS Access Keys and a default region in your Terraform environment (See section on required IAM Permissions)
-- The DLE runs on an EC2 instance which can be accessed using a selected set of SSH keys uploaded to EC2. Use the Terraform parameter `aws_keypair` to specify which EC2 Keypair to use
-- Required IAM Permissions: to successfully run this Terraform module, the IAM User/Role must have the following permissions:
-    * Read/Write permissions on EC2
-    * Read/Write permissions on Route53
-    * Read/Write permissions on Cloudwatch
+## Installation
 
-## Configuration overview
-- :construction: Currently, it is supposed that you run `terraform` commands on a Linux machine. MacOS and Windows support is not yet implemented (but planned).
-- It is recommended to clone this Git repository and adjust for your needs. Below we provide the detailed step-by-step instructions for quick start (see "Quick start") for a PoC setup
-- To configure parameters used by Terraform (and the Database Lab Engine itself), you will need to modify `terraform.tfvars` and create a file with secrets (`secret.tfvars`)
-- This Terraform module can be run independently or combined with any other standard Terraform module. You can learn more about using Terraform and the Terraform CLI [here](https://www.terraform.io/docs/cli/commands/index.html)
-- The variables can be set in multiple ways with the following precedence order (lowest to highest):
-    - default values in `variables.tf`
-    - values defined in `terraform.tfvars`
-    - values passed on the command line
-- All variables starting with `postgres_` represent the source database connection information for the data (from that database) to be fetched by the DLE. That database must be accessible from the instance hosting the DLE (that one created by Terraform)
-
-## How-to guide: using this Terraform module to set up DLE and its components
-The following steps were tested on Ubuntu 20.04 but supposed to be valid for other Linux distributions without significant modification.
-
-1. SSH to any machine with internet access, it will be used as deployment machine
-1. Install Terraform  https://learn.hashicorp.com/tutorials/terraform/install-cli. Example for Ubuntu:
-    ```shell
-    sudo apt-get update && sudo apt-get install -y gnupg software-properties-common curl 
-    curl -fsSL https://apt.releases.hashicorp.com/gpg | sudo apt-key add -
-    sudo apt-add-repository "deb [arch=amd64] https://apt.releases.hashicorp.com $(lsb_release -cs) main"  # Adjust if you have ARM platform.
-    sudo apt-get update && sudo apt-get install terraform
-    # Verify installation.
-    terraform -help
-    ```
-1. Get TF code for Database Lab:
-    ```shell
-    git clone https://gitlab.com/postgres-ai/database-lab-infrastructure.git
-    cd database-lab-infrastructure/
-    ```
-1. Edit `terraform.tfvars` file. In our example, we will use Heroku demo database as a source:
-    ```config
-    dle_version_full = "2.4.1"
-
-    aws_ami_name = "DBLABserver*"
-    aws_keypair = "YOUR_AWS_KEYPAIR"
-
-    aws_deploy_region = "us-east-1"
-    aws_deploy_ebs_availability_zone = "us-east-1a"
-    aws_deploy_ec2_instance_type = "t2.large"
-    aws_deploy_ec2_instance_tag_name = "DBLABserver-ec2instance"
-    aws_deploy_ebs_size = "40"
-    aws_deploy_ebs_type = "gp2"
-    aws_deploy_allow_ssh_from_cidrs = ["0.0.0.0/0"]
-    aws_deploy_dns_api_subdomain = "tf-test" # subdomain in aws.postgres.ai, fqdn will be ${dns_api_subdomain}-engine.aws.postgres
-
-    # Source – two options. Choose one of two:
-    #    - direct connection to source DB
-    #    - dump stored on AWS S3
-
-    # option 1 – direct PG connection
-    source_type = "postgres" # source is working dome postgres database
-    source_postgres_version = "13"
-    source_postgres_host = "ec2-3-215-57-87.compute-1.amazonaws.com" # an example DB at Heroku
-    source_postgres_port = "5432"
-    source_postgres_dbname = "d3dljqkrnopdvg" # an example DB at Heroku
-    source_postgres_username = "bfxuriuhcfpftt" # an example DB at Heroku
-    
-    # option 2 – dump on S3. Important: your AWS user has to be able to create IAM roles to work with S3 buckets in your AWS account
-    # source_type = 's3' # source is dump stored on demo s3 bucket
-    # source_pgdump_s3_bucket = "tf-demo-dump" # This is an example public bucket
-    # source_pgdump_path_on_s3_bucket = "heroku.dmp" # This is an example dump from demo database
-    
-    dle_debug_mode = "true"
-    dle_retrieval_refresh_timetable = "0 0 * * 0"
-    postgres_config_shared_preload_libraries = "pg_stat_statements,logerrors" # DB Migration Checker requires logerrors extension
-
-    platform_project_name = "aws_test_tf"
-    ```
-1. Create `secret.tfvars` containing `source_postgres_password`, `platform_access_token`, and `vcs_github_secret_token`. An example:
-    ```config
-    source_postgres_password = "dfe01cbd809a71efbaecafec5311a36b439460ace161627e5973e278dfe960b7" # an example DB at Heroku
-    platform_access_token = "YOUR_ACCESS_TOKEN"   # to generate, open https://console.postgres.ai/, choose your organization,
-                                                # then "Access tokens" in the left menu
-    vcs_github_secret_token = "vcs_secret_token"  # generate a personal access token with scope of "repo"
-    ```
-    To generate a personal GitHub access token with the scope of "repo", open the [guide on GitHub Docs](https://docs.github.com/en/github/authenticating-to-github/keeping-your-account-and-data-secure/creating-a-personal-access-token) and follow the instructions.
-    
-    Note that the "repo" scope essentially gives full access to all user-controlled repositories. Should you have any concerns about which repositories the DLE can have access to, consider using a separate GitHub account that has access to the reduced number of repositories.
-1. Initialize
-    ```shell
-    terraform init
-    ```
-1. Set environment variables with AWS credentials:
-    ```shell
-    export AWS_ACCESS_KEY_ID = "keyid"  # todo: how to get it?
-    export AWS_SECRET_ACCESS_KEY = "accesskey"
-    ```
-1. Deploy:
-    ```shell
-    terraform  apply -var-file="secret.tfvars" -auto-approve
-    ```
-1. If everything goes well, you should get an output like this:
-    ```config
-    vcs_db_migration_checker_verification_token = "gsio7KmgaxECfJ80kUx2tUeIf4kEXZex"
-    dle_verification_token = "zXPodd13LyQaKgVXGmSCeB8TUtnGNnIa"
-    ec2_public_dns = "ec2-11-111-111-11.us-east-2.compute.amazonaws.com"
-    ec2instance = "i-0000000000000"
-    ip = "11.111.111.11"
-    platform_joe_signing_secret = "lG23qZbUh2kq0ULIBfW6TRwKzqGZu1aP"
-    public_dns_name = "demo-api-engine.aws.postgres.ai"  # todo: this should be URL, not hostname – further we'll need URL, with protocol – `https://`
-    ```
-
-1. To verify result and check the progress, you might want to connect to the just-created EC2 machine using IP address or hostname from the Terraform output. In our example, it can be done using this one-liner (you can find more about DLE logs and configuration on this page: https://postgres.ai/docs/how-to-guides/administration/engine-manage):
-    ```shell
-    echo "sudo docker logs dblab_server -f" | ssh ubuntu@18.118.126.25 -i postgres_ext_test.pem
-    ```
-
-    Once you see the message like:
-    ```
-    2021/07/02 10:28:51 [INFO]   Server started listening on :2345.
-    ```
-    – it means that the DLE server started successfully and is waiting for you commands
-
- 1. Sign in to the [Postgres.ai Platform](https://console.postgres.ai/) and register your new DLE server:
-     1. Go to `Database Lab > Instances` in the left menu
-     1. Press the "Add instance" button
-     1. `Project` – specify any name (this is how your DLE server will be named in the platform)
-     1. `Verification token` – use the token generated above (`verification_token` value); do NOT press the "Generate" button here
-     1. `URL` – use the value generated above // todo: not convenient, we need URL but reported was only hostname
-     1. Press the "Verify URL" button to check the connectivity. Then press "Add". If everything is right, you should see the DLE page with green "OK" status:
-    <img src="/uploads/8371e7f79de199aa017ff2df82b8f704/image.png" width="400" />
- 1. Add Joe chatbot for efficient SQL optimization workflow:
-    1. Go to the "SQL Optimization > Ask Joe" page using the left menu, click the "Add instance" button, specify the same project as you defined in the previous step
-    1. `Signing secret` – use `platform_joe_signing_secret` from the Terraform output
-    1. `URL` – use `public_dns_name` values from the Terraform output with port `444`; in our example, it's `https://demo-api-engine.aws.postgres.ai:444`
-    1. Press "Verify URL" to check connectivity and then press "Add". You should see:
-    <img src="/uploads/252e5f74cd324fc4df301bbf7c2bdd25/image.png" width="400" />
-
-    Now you can start using Joe chatbot for SQL execution plans troubleshooting and verification of optimization ideas. As a quick test, go to `SQL Optimization > Ask Joe` in the left menu, and enter `\dt+` command (a psql command to show the list of tables with sizes). You should see how Joe created a thin clone behind the scenes and immediately ran this psql command, presenting the result to you:
-    <img src="/uploads/d9e9e1fdafb0ded3504691cec9018868/image.png" width="400" />
-
-1. Set up [DB migration checker](https://postgres.ai/docs/db-migration-checker). Prepare a repository with your DB migrations(Flyway, Sqitch, Liquibase, etc.):
-   1. Add secrets:
-      - `DLMC_CI_ENDPOINT` - an endpoint of your Database Lab Migration Checker service – use `vcs_db_migration_checker_registration_url` from the Terraform output
-      - `DLMC_VERIFICATION_TOKEN` - verification token for the Database Lab Migration Checker API – use `vcs_db_migration_checker_verification_token` from the Terraform output
-   1. Configure a new workflow in the created repository (see an example of configuration: https://github.com/postgres-ai/green-zone/blob/master/.github/workflows/main.yml)
-      - add a custom action: https://github.com/marketplace/actions/database-lab-realistic-db-testing-in-ci
-      - provide input params for the action (the full list of available input params)
-      - provide environment variables:
-        - `DLMC_CI_ENDPOINT` - use a CI Checker endpoint from the repository secrets
-        - `DLMC_VERIFICATION_TOKEN` - use a verification token from the repository secrets
-
-1. Install and try the client CLI (`dblab`)
-    1. Follow the [guide](https://postgres.ai/docs/how-to-guides/cli/cli-install-init) to install Database Lab CLI
-    1. Initialize CLI:
-    ```shell
-    dblab init --environment-id=<ANY NAME FOR ENVIRONMENT> --url=https://<public_dns_name> --token=<your_personal_token_from_postgres_ai_platform>
-    ```
-    1. Try it:
-    ```shell
-    dblab instance status
-    ```
-    It should return the OK status:
-    ```json
-    {
-        "status": {
-            "code": "OK",
-            "message": "Instance is ready"
-        },
-        ...
-    }
-    ```
-
-## Important Note
-When the DLE creates new database clones, it makes them available on incremental ports in the 6000 range (e.g. 6000, 6001, ...). The DLE CLI will also report that the clone is available on a port in the 6000 range.  However, please note that these are the ports when accessing the DLE from `localhost`.  This Terraform module deploys [Envoy](https://www.envoyproxy.io/) to handle SSL termination and port forwarding to connect to DLE generated clones.
-
-Bottom Line: When connecting to clones, add `3000` to the port number reported by the DLE CLI to connect to the clone. for example, if the CLI reports that a new clone is available at port `6001` connect that clone at port `9001`.
+Follow the [how-to guide](https://postgres.ai/docs/how-to-guides/administration/install-database-lab-with-terraform) to install Database Lab with Terraform on AWS
 
 ## Known Issues
 ### Certificate Authority Authorization (CAA) for your Hosted Zone

dle-logical-init.sh.tpl

@@ -4,10 +4,10 @@ set -x
 
 sleep 20
 #run certbot and copy files to envoy
-# to avoid restrinctions from letsencrypt like "There were too many requests of a given type ::
+# to avoid restrictions from letsencrypt like "There were too many requests of a given type ::
 # Error creating new order :: too many certificates (5) already issued for this exact set of domains
 # in the last 168 hours: demo-api-engine.aws.postgres.ai: see https://letsencrypt.org/docs/rate-limits/"
-# follwing three lines were commented out and mocked up. In real implementation inline certs have to be
+# following three lines were commented out and mocked up. In real implementation inline certs have to be
 # removed and letsencrypt generated certs should be used
 
 
@@ -123,11 +123,14 @@ sed -ri "s/:13/:${source_postgres_version}/g"  ~/.dblab/server.yml
 case "${source_type}" in 
 
   postgres)
+  # Mount directory to store dump files.
+  extra_mount="--volume /var/lib/dblab/dblab_pool_00/dump:/var/lib/dblab/dblab_pool/dump"
+
   sed -ri "s/^(\s*)(host: 34.56.78.90$)/\1host: ${source_postgres_host}/" ~/.dblab/server.yml
   sed -ri "s/^(\s*)(port: 5432$)/\1port: ${source_postgres_port}/" ~/.dblab/server.yml
   sed -ri "s/^(\s*)(            username: postgres$)/\1            username: ${source_postgres_username}/" ~/.dblab/server.yml
   sed -ri "s/^(\s*)(password:.*$)/\1password: ${source_postgres_password}/" ~/.dblab/server.yml
-  #restore pg_dump via pipe -  without saving it on the disk
+  # restore pg_dump via pipe -  without saving it on the disk
   sed -ri "s/^(\s*)(parallelJobs:.*$)/\1parallelJobs: 1/" ~/.dblab/server.yml
   sed -ri "s/^(\s*)(# immediateRestore:.*$)/\1immediateRestore: /" ~/.dblab/server.yml
   sed -ri "s/^(\s*)(#   forceInit: false.*$)/\1  forceInit: true /" ~/.dblab/server.yml
@@ -142,7 +145,7 @@ case "${source_type}" in
   mkdir -p "${source_pgdump_s3_mount_point}"
   s3fs ${source_pgdump_s3_bucket} ${source_pgdump_s3_mount_point} -o iam_role -o allow_other
 
-  s3_mount="--volume ${source_pgdump_s3_mount_point}:${source_pgdump_s3_mount_point}"
+  extra_mount="--volume ${source_pgdump_s3_mount_point}:${source_pgdump_s3_mount_point}"
 
   sed -ri "s/^(\s*)(- logicalDump.*$)/\1#- logicalDump /" ~/.dblab/server.yml
   sed -ri "s|^(\s*)(        dumpLocation:.*$)|\1        dumpLocation: ${source_pgdump_s3_mount_point}/${source_pgdump_path_on_s3_bucket}|" ~/.dblab/server.yml
@@ -156,11 +159,10 @@ sudo docker run \
  --privileged \
  --publish 2345:2345 \
  --volume /var/run/docker.sock:/var/run/docker.sock \
- --volume /var/lib/dblab/dblab_pool_00/dump:/var/lib/dblab/dblab_pool/dump \
  --volume /var/lib/dblab:/var/lib/dblab/:rshared \
  --volume ~/.dblab/server.yml:/home/dblab/configs/config.yml \
- --mount type=bind,source=/root/.dblab/postgres_conf,target=/home/dblab/configs/postgres \
- $s3_mount \
+ --volume /root/.dblab/postgres_conf:/home/dblab/configs/postgres \
+ $extra_mount \
  --env DOCKER_API_VERSION=1.39 \
  --detach \
  --restart on-failure \
@@ -180,7 +182,7 @@ dblab init \
  --token=${dle_verification_token} \
  --insecure
 
-#configure and run Joe Bot container
+# Configure and run Joe Bot container.
 cp /home/ubuntu/joe.yml ~/.dblab/joe.yml
 sed -ri "s/^(\s*)(debug:.*$)/\1debug: ${dle_debug_mode}/" ~/.dblab/joe.yml
 sed -ri "s/^(\s*)(  token:.*$)/\1  token: ${platform_access_token}/" ~/.dblab/joe.yml
@@ -198,8 +200,8 @@ sudo docker run \
     --detach \
 postgresai/joe:latest
 
-#configure and run DB Migration Checker
-curl https://gitlab.com/postgres-ai/database-lab/-/raw/master/configs/config.example.run_ci.yaml --output ~/.dblab/run_ci.yaml
+# Configure and run DB Migration Checker.
+curl https://gitlab.com/postgres-ai/database-lab/-/raw/${dle_version_full}/configs/config.example.run_ci.yaml --output ~/.dblab/run_ci.yaml
 sed -ri "s/^(\s*)(debug:.*$)/\1debug: ${dle_debug_mode}/" ~/.dblab/run_ci.yaml
 sed -ri "s/^(\s*)(  verificationToken: \"secret_token\".*$)/\1  verificationToken: ${vcs_db_migration_checker_verification_token}/" ~/.dblab/run_ci.yaml
 sed -ri "s/^(\s*)(  url: \"https\\:\\/\\/dblab.domain.com\"$)/\1  url: \"http\\:\\/\\/dblab_server\\:2345\"/" ~/.dblab/run_ci.yaml
@@ -212,4 +214,4 @@ sudo docker run --name dblab_ci_checker -it --detach \
 --volume /var/run/docker.sock:/var/run/docker.sock \
 --volume /tmp/ci_checker:/tmp/ci_checker \
 --volume ~/.dblab/run_ci.yaml:/home/dblab/configs/run_ci.yaml \
-postgresai/dblab-ci-checker:2.4.1
+postgresai/dblab-ci-checker:${dle_version_full}

role.tf

@@ -1,6 +1,6 @@
 resource "aws_iam_role" "db_lab_engine_role" {
   count = "${var.source_type == "s3" ? 1 : 0}"
-  name               = "database_lab_engine"
+  name_prefix = "database_lab_engine_"
   assume_role_policy = jsonencode({
     Version = "2012-10-17"
     Statement = [
@@ -35,8 +35,9 @@ resource "aws_iam_role" "db_lab_engine_role" {
     })
   }
 }
+
 resource "aws_iam_instance_profile" "instance_profile" {
-    count = "${var.source_type == "s3" ? 1 : 0}"
-    name = "dle-instance-profile"
-    role = "${aws_iam_role.db_lab_engine_role[0].name}"
+  count = "${var.source_type == "s3" ? 1 : 0}"
+  name_prefix = "dle_instance_profile_"
+  role = "${aws_iam_role.db_lab_engine_role[0].name}"
 }