Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update retl bq docs #7134

Merged
merged 38 commits into from
Oct 15, 2024
Merged
Changes from 8 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
e9ead23
update Reverse-ETL bigQuery docs
gilsegment Oct 9, 2024
adb2ed4
change the order
gilsegment Oct 9, 2024
7533105
update role creation section
gilsegment Oct 9, 2024
baea5ce
updating bigquery source setup
gilsegment Oct 10, 2024
c923a0d
typo and make the custom role more strict
gilsegment Oct 10, 2024
8ba9f5d
more info regarding location
gilsegment Oct 10, 2024
fc28a3d
quick fix [netlify-build]
pwseg Oct 10, 2024
272a6e2
add highlight
gilsegment Oct 10, 2024
c3bc887
add additional required permission
gilsegment Oct 10, 2024
d52f819
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
8490ddd
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
060be82
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
f2c2bce
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
62beb23
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
018f111
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
3be45e5
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
225da70
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
7c22cc1
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
1379da9
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
3f6d479
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
96e98bf
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
5238b2b
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
7d533f6
Apply suggestions from code review
gilsegment Oct 10, 2024
720c275
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
fc7edc1
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
6fe618d
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
fa8ba24
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 10, 2024
70d7adf
use lower case when needed
gilsegment Oct 10, 2024
d3431e2
[netlify-build]
gilsegment Oct 11, 2024
52069f9
remove extra dot
gilsegment Oct 11, 2024
171fa39
[netlify-build] some additional changes
gilsegment Oct 11, 2024
459c238
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
e8b44cb
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
6faadd5
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
20e96e9
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
3bb85f0
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
d0541db
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
4baab70
Update src/connections/reverse-etl/reverse-etl-source-setup-guides/bi…
gilsegment Oct 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -3,53 +3,73 @@ title: BigQuery Reverse ETL Setup
redirect_from:
- '/reverse-etl/bigquery-setup/'
---

gilsegment marked this conversation as resolved.
Show resolved Hide resolved
Set up BigQuery as your Reverse ETL source.
## Constructing your own role or policy

> warning ""
> You need to be an account admin to set up the Segment BigQuery connector as well as write permissions for the `__segment_reverse_etl` dataset.

To set up the Segment BigQuery connector:
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
2. Click **+ Create Service Account** to create a new service account
3. Enter your **Service account name** and a description of what the account will do
4. Click **Create and Continue**.
5. In the **Grant this service account access to project** section, select the *BigQuery User* role to add.
6. Click **+ Add another role** and add the *BigQuery Job User* role.
7. Click **Continue**.
8. Click **Done**.
9. Search for the service account you just created.
10. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
11. Click **Add Key > Create new key**.
12. In the pop-up window, select **JSON** for the key type and click **Create**. The file will download.
13. Copy all the content within the file you just created and downloaded.
14. Navigate to the Segment UI and paste all the credentials you copied from step 13 into the **Enter your credentials** section.
19. Enter your **Data Location**.
20. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
6. Click **Add source** if the test connection is successful.

After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model).

> info "BigQuery Reverse ETL sources support Segment's dbt extension"
> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.
When creating a role and service-account, we offer two approaches:
1. **Grant Full Access**: This option provides Segment with all the necessary permissions and allows us to complete the setup for you automatically. It's the quicker option, requiring minimal effort on your part.
2. **Grant Limited Access**: This option is more secure, as it restricts permissions. However, due to the limited access, a few additional setup steps will need to be completed manually by you. These are one-time steps, and we will guide you through the process.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved

## Constructing your own role or policy
When you construct your own role or policy, Segment needs the following permissions:
You are free to choose the approach that best suits your needs, and can skip the other.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved

Permission | Details
---------- | --------
`bigquery.datasets.create` | This allows Segment to create/manage a `__segment_reverse_etl` dataset for tracking state between syncs.
`bigquery.datasets.get` | This allows Segment to determine if the aforementioned dataset exists.
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.
### Grant Full Access
With this approach we will use BigQuery predefined roles:
1. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
2. Click **+ Create Service Account** to create a new service account.
3. Enter your **Service account name** and a description of what the service-account will do.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
4. Click **Create and Continue**.
5. Click **+ Add another role** and add the **BigQuery User** role.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
6. Click **+ Add another role** and add the **BigQuery Data Editor** role.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
7. Click **Continue**.
8. Click **Done**.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved

If you don't want to grant Segment `bigquery.datasets.create` access, you can create your own `__segment_reverse_etl` dataset and give Segment the following permissions:
### Grant Limited Access
With this approach we will use custom role with the below permissions:
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
With this approach we will use custom role with the below permissions:
With this approach, create a custom role for Segment with the following permissions:


Permission | Details
---------- | --------
`bigquery.datasets.get` | This allows Segment to determine if the `__segment_reverse_etl` dataset exists.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
`bigquery.tables.create` | This allows Segment to determine if the tables Segment uses to track state in the `__segment_reverse_etl` dataset exists.
`bigquery.jobs.create` | This allows Segment to execute queries on any datasets or tables your model query references, and also allows Segment to manage tables used for tracking.

The `bigquery.datasets.*` permissions can be scoped only to the `__segment_reverse_etl` dataset.

After you've successfully added your BigQuery source, [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.
1. Navigate to **IAM & Admin > Roles** in BigQuery.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
2. Click **+ CREATE ROLE** to create a new role.
3. Add **title** and **Description** as you like.
4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
3. Add **title** and **Description** as you like.
4. Click **ADD PERMISSIONS** and add the permission listed in the above tables. You will to repeat that until all required permissions were added.
3. Add a **Title** for your role and a **Description** of what the role will have access to.
4. Click **ADD PERMISSIONS** and add all of the permissions listed in the table above.

5. Click **CREATE**.
6. Navigate to **IAM & Admin > Service Accounts** in BigQuery.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
7. Click **+ Create Service Account** to create a new service account.
8. Enter your **Service account name** and a description of what the account will do.
9. Click **Create and Continue**.
10. In the **Grant this service account access to project** section, select the role you just created.
11. Click **Continue**.
12. Click **Done**.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The docs site automatically has the correct numbering, so no need to worry about the rest of the items in this list!

Suggested change
11. Click **Continue**.
12. Click **Done**.
11. Click **Continue**, then click **Done**.

13. Navigate to the BigQuery SQL editor and create a dataset that will be used by Segment:
```
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
CREATE SCHEMA IF NOT EXISTS `__segment_reverse_etl`;
```
14. Grant additional permissions just on the newly created dataset:
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
```
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
GRANT `roles/bigquery.dataEditor` ON SCHEMA `__segment_reverse_etl` TO "serviceAccount:<YOUR SERVICE ACCOUNT EMAIL>";
```

gilsegment marked this conversation as resolved.
Show resolved Hide resolved
## Set up BigQuery as your Reverse ETL source
1. In the BigQuery console, search for the service account you just created.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved
2. When your service account pulls up, click the 3 dots under **Actions** and select **Manage keys**.
3. Click **Add Key > Create new key**.
4. In the pop-up window, select **JSON** for the key type and click **Create**. The file will be downloaded.
5. Copy all the content within the **JSON** file created and downloaded in previous section.
6. Navigate to the Segment UI and paste all the credentials you copied from previous step into the **Enter your credentials** section.
7. Enter your **Data Location**. The data location can be found by:
- In the BigQuery console navigate to your dataset: In the explorer panel on the left, expand the project and dataset to view the tables.
- Click on the name of the dataset, and it will open a page showing its details.
- The Location of the dataset (like US or EU) is displayed in the Dataset Info.
8. Click **Test Connection** to test to see if the connection works. If the connection fails, make sure you have the right permissions and credentials and try again.
9. Click **Add source** if the test connection is successful.
gilsegment marked this conversation as resolved.
Show resolved Hide resolved

After you've added BigQuery as a source, you can [add a model](/docs/connections/reverse-etl/setup/#step-2-add-a-model) and follow the rest of the steps in the Reverse ETL setup guide.

> info "BigQuery Reverse ETL sources support Segment's dbt extension"
> If you have an existing dbt account with a Git repository, you can use [Segment's dbt extension](/docs/segment-app/extensions/dbt/) to centralize model management and versioning, reduce redundancies, and run CI checks to prevent breaking changes.