You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: tidb-cloud/changefeed-sink-to-apache-kafka.md
+33-55Lines changed: 33 additions & 55 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -52,23 +52,9 @@ Private Connect leverages **Private Link** or **Private Service Connect** techno
52
52
53
53
TiDB Cloud currently supports Private Connect only for self-hosted Kafka. It does not support direct integration with MSK, Confluent Kafka, or other Kafka SaaS services. To connect to these Kafka SaaS services via Private Connect, you can deploy a [kafka-proxy](https://github.com/grepplabs/kafka-proxy) as an intermediary, effectively exposing the Kafka service as self-hosted Kafka. For a detailed example, see [Set Up Self-Hosted Kafka Private Service Connect by Kafka-proxy in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md#set-up-self-hosted-kafka-private-service-connect-by-kafka-proxy). This setup is similar across all Kafka SaaS services.
54
54
55
-
- If your Apache Kafka service is hosted in AWS, follow [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:
56
-
57
-
- The ID in Kafka Advertised Listener Pattern
58
-
- The Endpoint Service Name
59
-
- The Bootstrap Ports
60
-
61
-
- If your Apache Kafka service is hosted in Google Cloud, follow [Set Up Self-Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:
62
-
63
-
- The ID in Kafka Advertised Listener Pattern
64
-
- The Service Attachment
65
-
- The Bootstrap Ports
66
-
67
-
- If your Apache Kafka service is hosted in Azure, follow [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md) to ensure that the network connection is properly configured. After setup, provide the following information in the TiDB Cloud console to create the changefeed:
68
-
69
-
- The ID in Kafka Advertised Listener Pattern
70
-
- The Alias of Private Link Service
71
-
- The Bootstrap Ports
55
+
- If your Apache Kafka service is hosted on AWS, follow [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) to configure the network connection and obtain the **Bootstrap Ports** information, and then follow [Set Up Private Endpoint for Changefeeds](/tidb-cloud/set-up-sink-private-endpoint.md) to create a private endpoint.
56
+
- If your Apache Kafka service is hosted on Google Cloud, follow [Set Up Self-Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md) to configure the network connection and obtain the **Bootstrap Ports** information, and then follow [Set Up Private Endpoint for Changefeeds](/tidb-cloud/set-up-sink-private-endpoint.md) to create a private endpoint.
57
+
- If your Apache Kafka service is hosted on Azure, follow [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md) to configure the network connection and obtain the **Bootstrap Ports** information, and then follow [Set Up Private Endpoint for Changefeeds](/tidb-cloud/set-up-sink-private-endpoint.md) to create a private endpoint.
72
58
73
59
</div>
74
60
<divlabel="VPC Peering">
@@ -139,63 +125,55 @@ The steps vary depending on the connectivity method you select.
139
125
<divlabel="Private Link (AWS)">
140
126
141
127
1. In **Connectivity Method**, select **Private Link**.
142
-
2. Authorize the [AWS Principal](https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_principal.html#principal-accounts) of TiDB Cloud to create an endpoint for your endpoint service. The AWS Principal is provided in the tip on the web page.
143
-
3. Make sure you select the same **Number of AZs** and **AZ IDs of Kafka Deployment**, and fill the same unique ID in **Kafka Advertised Listener Pattern** when you [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md) in the **Network** section.
144
-
4. Fill in the **Endpoint Service Name** which is configured in [Set Up Self-Hosted Kafka Private Link Service in AWS](/tidb-cloud/setup-aws-self-hosted-kafka-private-link-service.md).
145
-
5. Fill in the **Bootstrap Ports**. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
146
-
6. Select an **Authentication** option according to your Kafka authentication configuration.
128
+
2. In **Private Endpoint**, select the private endpoint that you created in the [Network](#network) section. Make sure the AZs of the private endpoint match the AZs of the Kafka deployment.
129
+
3. Fill in the **Bootstrap Ports** that you obtained from the [Network](#network) section. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
130
+
4. Select an **Authentication** option according to your Kafka authentication configuration.
147
131
148
132
- If your Kafka does not require authentication, keep the default option **Disable**.
149
133
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.
150
-
151
-
7. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
152
-
8. Select a **Compression** type for the data in this changefeed.
153
-
9. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
154
-
10. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
155
-
11. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
156
-
12. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
157
-
13. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
134
+
5. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
135
+
6. Select a **Compression** type for the data in this changefeed.
136
+
7. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
137
+
8. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
138
+
9. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
139
+
10. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
140
+
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
158
141
159
142
</div>
160
143
<divlabel="Private Service Connect (Google Cloud)">
161
144
162
145
1. In **Connectivity Method**, select **Private Service Connect**.
163
-
2. Ensure that you fill in the same unique ID in **Kafka Advertised Listener Pattern** when you [Set Up Self-Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md) in the **Network** section.
164
-
3. Fill in the **Service Attachment** that you have configured in [Setup Self Hosted Kafka Private Service Connect in Google Cloud](/tidb-cloud/setup-self-hosted-kafka-private-service-connect.md)
165
-
4. Fill in the **Bootstrap Ports**. It is recommended that you provide more than one port. You can use commas `,` to separate multiple ports.
166
-
5. Select an **Authentication** option according to your Kafka authentication configuration.
146
+
2. In **Private Endpoint**, select the private endpoint that you created in the [Network](#network) section.
147
+
3. Fill in the **Bootstrap Ports** that you obtained from the [Network](#network) section. It is recommended that you provide more than one port. You can use commas `,` to separate multiple ports.
148
+
4. Select an **Authentication** option according to your Kafka authentication configuration.
167
149
168
150
- If your Kafka does not require authentication, keep the default option **Disable**.
169
151
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.
170
-
171
-
6. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
172
-
7. Select a **Compression** type for the data in this changefeed.
173
-
8. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
174
-
9. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
175
-
10. TiDB Cloud creates the endpoint for **Private Service Connect**, which might take several minutes.
176
-
11. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
177
-
12. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
152
+
5. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
153
+
6. Select a **Compression** type for the data in this changefeed.
154
+
7. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
155
+
8. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
156
+
9. TiDB Cloud creates the endpoint for **Private Service Connect**, which might take several minutes.
157
+
10. Once the endpoint is created, log in to your cloud provider console and accept the connection request.
158
+
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
178
159
179
160
</div>
180
161
<divlabel="Private Link (Azure)">
181
162
182
163
1. In **Connectivity Method**, select **Private Link**.
183
-
2. Authorize the Azure subscription of TiDB Cloud or allow anyone with your alias to access your Private Link service before creating the changefeed. The Azure subscription is provided in the **Reminders before proceeding** tip on the web page. For more information about the visibility of Private Link service, see [Control service exposure](https://learn.microsoft.com/en-us/azure/private-link/private-link-service-overview#control-service-exposure) in Azure documentation.
184
-
3. Make sure you fill in the same unique ID in **Kafka Advertised Listener Pattern** when you [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md) in the **Network** section.
185
-
4. Fill in the **Alias of Private Link Service** which is configured in [Set Up Self-Hosted Kafka Private Link Service in Azure](/tidb-cloud/setup-azure-self-hosted-kafka-private-link-service.md).
186
-
5. Fill in the **Bootstrap Ports**. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
187
-
6. Select an **Authentication** option according to your Kafka authentication configuration.
164
+
2. In **Private Endpoint**, select the private endpoint that you created in the [Network](#network) section.
165
+
3. Fill in the **Bootstrap Ports** that you obtained in the [Network](#network) section. It is recommended that you set at least one port for one AZ. You can use commas `,` to separate multiple ports.
166
+
4. Select an **Authentication** option according to your Kafka authentication configuration.
188
167
189
168
- If your Kafka does not require authentication, keep the default option **Disable**.
190
169
- If your Kafka requires authentication, select the corresponding authentication type, and then fill in the **user name** and **password** of your Kafka account for authentication.
191
-
192
-
7. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
193
-
8. Select a **Compression** type for the data in this changefeed.
194
-
9. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
195
-
10. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
196
-
11. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
197
-
12. Once the endpoint is created, log in to the [Azure portal](https://portal.azure.com/) and accept the connection request.
198
-
13. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
170
+
5. Select your **Kafka Version**. If you do not know which one to use, use **Kafka v2**.
171
+
6. Select a **Compression** type for the data in this changefeed.
172
+
7. Enable the **TLS Encryption** option if your Kafka has enabled TLS encryption and you want to use TLS encryption for the Kafka connection.
173
+
8. Click **Next** to test the network connection. If the test succeeds, you will be directed to the next page.
174
+
9. TiDB Cloud creates the endpoint for **Private Link**, which might take several minutes.
175
+
10. Once the endpoint is created, log in to the [Azure portal](https://portal.azure.com/) and accept the connection request.
176
+
11. Return to the [TiDB Cloud console](https://tidbcloud.com) to confirm that you have accepted the connection request. TiDB Cloud will test the connection and proceed to the next page if the test succeeds.
0 commit comments