Skip to content
This repository was archived by the owner on May 25, 2026. It is now read-only.

install.md

Latest commit

 

History

History
329 lines (264 loc) · 11.9 KB

install.md

File metadata and controls

329 lines (264 loc) · 11.9 KB
Raw

How to install Grizzly

Creating Google Cloud Projects

For this platform, you need four Google Cloud projects with the Project IDs as [stem_name]-metadata, [stem_name]-dev, [stem_name]-uat, and [stem_name]-prod. For example, grizzly-metadata, grizzly-dev, grizzly-uat, and grizzly-prod. Please note that while it is possible to set a Project's name and ID as different values, it is important that the name and ID should match exactly.

Create new projects for this tutorial.

  1. Create the four Google Cloud projects and ensure that you have IAM roles/owner access for each project.
  2. Make sure that billing is enabled for your Google Cloud project.

Costs

This tutorial uses billable components of Google Cloud, including the following:

Use the pricing calculator to generate a cost estimate based on your projected usage.

Enabling Cloud Shell

Open Cloud Shell.

At the bottom of the Cloud Console, a Cloud Shell session opens and displays a command-line prompt. Cloud Shell is a shell environment with the Cloud SDK already installed, including the gcloud command-line tool, and with values already set for your current project. It can take a few seconds for the session to initialize.

Setting up your environment

  1. In Cloud Shell, set your Git credentials, clone the source repository, and go to the directory for this tutorial.

    git config --global user.email "abc@example.com"

    git config --global user.name "Your Name"

    sudo rm -rf ~/grizzly_repo && mkdir ~/grizzly_repo

    cd ~/grizzly_repo

    gh auth login

    gh repo clone google/grizzly ./grizzly

  2. In Cloud Shell, set the required environment variables.

    • The [stem_name] value should match the Project IDs you created earlier.

    • The [composer_location] value should be from a Region Name from the list of available Cloud Storage locations. Please note that the Grizzly platform does not currently support Dual-regions or Multi-regions.

    • The [gcp_resource_location] value should be "us" or "eu". Only if the location is outside of the United States or the European Union, the value should match the [composer_location] value.

    • The [app_engine_location] value should be from a Region Name from the list of available App Engine locations.

    • The [security_user] value is a user or Google group that will be configured for Grizzly's demo examples. Supported formats of the parameter are user:abc@example.com or group:abc_group@example.com.

      cd ~/grizzly_repo/grizzly
./tools/init_grizzly_environment_from_scratch.sh \
--GCP_PROJECT_METADATA "[stem_name]-metadata" \
--GCP_PROJECT_DEV "[stem_name]-dev" \
--GCP_PROJECT_UAT "[stem_name]-uat" \
--GCP_PROJECT_PROD "[stem_name]-prod" \
--GCP_RESOURCE_LOCATION "[gcp_resource_location]" \
--AIRFLOW_LOCATION "[composer_location]" \
--APP_ENGINE_LOCATION "[app_engine_location]" \
--COMPOSER_IMAGE "composer-2.1.1-airflow-2.3.4" \
--SECURITY_USER "[security_user]"

      For example,

      cd ~/grizzly_repo/grizzly
./tools/init_grizzly_environment_from_scratch.sh \
--GCP_PROJECT_METADATA "grizzly-metadata" \
--GCP_PROJECT_DEV "grizzly-dev" \
--GCP_PROJECT_UAT "grizzly-uat" \
--GCP_PROJECT_PROD "grizzly-prod" \
--GCP_RESOURCE_LOCATION "us" \
--AIRFLOW_LOCATION "us-central1" \
--APP_ENGINE_LOCATION "us-central" \
--COMPOSER_IMAGE "composer-2.1.1-airflow-2.3.4" \
--SECURITY_USER "user:abc@example.com"

      Once the script completes, it will create all required terraform variable files and initialize the Git repository on the [stem_name]-metadata project.

  3. In Cloud Shell, build the [stem_name]-metadata GCP project.

    cd ~/grizzly/metadata/grizzly_framework/terraform/metadata

    terraform init && terraform apply

    The terraform script takes about 30 minutes to an hour to complete. If you see an error, rerun the terraform init && terraform apply command.

  4. In Cloud Shell, build the [stem_name]-dev GCP project.

    cd ~/grizzly_repo/grizzly/tools
./apply_grizzy_terraform.sh \
--GCP_PROJECT_METADATA "[stem_name]-metadata" \
--GCP_RESOURCE_LOCATION "[gcp_resource_location]" \
--ENVIRONMENT dev

    For example,

    cd ~/grizzly_repo/grizzly/tools
./apply_grizzy_terraform.sh \
--GCP_PROJECT_METADATA "grizzly-metadata" \
--GCP_RESOURCE_LOCATION "us" \
--ENVIRONMENT dev

  5. In Cloud Shell, build the [stem_name]-uat GCP project.

    cd ~/grizzly_repo/grizzly/tools
./apply_grizzy_terraform.sh \
--GCP_PROJECT_METADATA "[stem_name]-metadata" \
--GCP_RESOURCE_LOCATION "[gcp_resource_location]" \
--ENVIRONMENT uat

    For example,

    cd ~/grizzly_repo/grizzly/tools
./apply_grizzy_terraform.sh \
--GCP_PROJECT_METADATA "grizzly-metadata" \
--GCP_RESOURCE_LOCATION "us" \
--ENVIRONMENT uat

  6. In Cloud Shell, build the [stem_name]-prod GCP project.

    cd ~/grizzly_repo/grizzly/tools
./apply_grizzy_terraform.sh \
--GCP_PROJECT_METADATA "[stem_name]-metadata" \
--GCP_RESOURCE_LOCATION "[gcp_resource_location]" \
--ENVIRONMENT prod

    For example,

    cd ~/grizzly_repo/grizzly/tools
./apply_grizzy_terraform.sh \
--GCP_PROJECT_METADATA "grizzly-metadata" \
--GCP_RESOURCE_LOCATION "us" \
--ENVIRONMENT prod

  7. In the [stem_name]-dev project, monitor the first run of each DAG by navigating to Composer and then opening the Airflow instance.

  8. Once the Airflow DAGs finish, navigate to Cloud Build in the [stem_name]-metadata project.

    • Deploy the demo machine learning model by running the deploy-bigquery trigger after setting its SCOPE_FILE value to SCOPE_ML.yml.

    • Import the initial Git metadata into Grizzly's log tables by running the import-git-rep trigger after setting its Branch value to dev.

  9. In the [stem_name]-metadata project, enable the column-level data flow visualization.

  10. Install and configure the Superset application to see the demo dashboards. This step is optional and can be performed later.

    • On your Mac, Windows, or Linux desktop, install and then open the Docker application.

    • From the command line of your Mac, Windows, or Linux desktop, copy and configure the Superset application.

      # pull stable version 1.5.2
docker pull apache/superset:1.5.2

      # tag stable version 1.5.2 to grizzly_superset_152
docker tag apache/superset:1.5.2 grizzly_superset_152

      # execute grizzly_superset_152 as grizzly_superset_152 container
docker run -d -p 8080:8088 --name grizzly_superset_152 grizzly_superset_152

      # init admin user
docker exec -it grizzly_superset_152 superset fab create-admin \
--username admin \
--firstname superset \
--lastname admin \
--email superset@example.com \
--password admin

      # init db upgrade
docker exec -it grizzly_superset_152 superset db upgrade

      # load examples
docker exec -it grizzly_superset_152 superset load_examples

      # load pybigquery lib
docker exec -it grizzly_superset_152 pip3 install pybigquery

      # init superset
docker exec -it grizzly_superset_152 superset init

      # restart grizzly_superset_152 to pickup bigquery
docker restart grizzly_superset_152

    • On your desktop, open Superset in a browser window by entering http://localhost:8080. Username and password are admin.

    • In the [stem_name]-dev project, create a Service Account Key with the BigQuery Data Viewer, BigQuery Job User, and BigQuery Read Session User permissions. Download the key as a json file to your desktop.

    • In Superset, create the BigQuery default database connection using the json Service Account Key you downloaded in the previous step. The Display Name value should be bq_connection instead of the default "Google BigQuery".

    • Using your browser, download the eight example dashboard json files to your desktop. Using Superset, import each dashboard (Settings -> Import Dashboards).