Related Template(s)
Primary: Datastream to Cloud Spanner template
Secondary: Spanner to SourceDB template, SourceDB to Spanner template
What feature(s) are you requesting?
Currently, you can build a Datastream to stream Spanner CDC into GCS bucket: https://docs.cloud.google.com/datastream/docs/sources-spanner
There also exist a complementary Dataflow template to consume Datastream from GCS and stream to Spanner: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/datastream-to-spanner/README_Cloud_Datastream_to_Spanner.md
Unfortunately, the two are not compatible with each other since the Dataflow template does not allow Spanner as a Datastream source type, and other types are not compatible with Spanner: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/datastream-to-spanner/src/main/java/com/google/cloud/teleport/v2/templates/source/DatastreamToSpannerSourceConnectorRegistry.java#L35-L50
The request is to add support for Spanner within the template by implementing a complementary connector, so that it is possible to build Spanner-Spanner live migrations as per: https://docs.cloud.google.com/spanner/docs/data-migration#live-data-migration
Alternatively, if this route is not planned to be supported, or if there is another, supported way to perform live migrations between Spanner databases, the request is to cover that route in documentation and tooling. So far, I have reviewed multiple possible routes and some seem possible but not covered, so it is not clear whether they are supported at all.
Workarounds
Performing "Spanner instance move" is not the same use case. It moves entire Spanner instances as-is, which is not suitable for cases like re-architecting the Spanner instances, which is what we currently do. As such, we need a way to perform a live migration of one Spanner DB at a time, between 2 arbitrary Spanner instances.
The workaround I found and tried is that you can stream your CDC directly between Spanner instances using Spanner to SourceDB streaming template, however this seems to lack official support and is not documented, so I have low confidence in this solution: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/spanner-to-sourcedb/README.md / https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/spanner-to-sourcedb/src/main/java/com/google/cloud/teleport/v2/templates/SpannerToSourceDb.java#L410
Yet another workaround, which I haven't tried, is to try the other template (Source DB to Spanner), which seems to accept just about any JDBC driver and connection string, so in theory you can provide Spanner JDBC driver. However, it lacks documentation on which drivers are available OOTB and which needs to be supplied by the user: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/sourcedb-to-spanner/README.md#optional-parameters
Thank you!
Related Template(s)
Primary: Datastream to Cloud Spanner template
Secondary: Spanner to SourceDB template, SourceDB to Spanner template
What feature(s) are you requesting?
Currently, you can build a Datastream to stream Spanner CDC into GCS bucket: https://docs.cloud.google.com/datastream/docs/sources-spanner
There also exist a complementary Dataflow template to consume Datastream from GCS and stream to Spanner: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/datastream-to-spanner/README_Cloud_Datastream_to_Spanner.md
Unfortunately, the two are not compatible with each other since the Dataflow template does not allow Spanner as a Datastream source type, and other types are not compatible with Spanner: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/datastream-to-spanner/src/main/java/com/google/cloud/teleport/v2/templates/source/DatastreamToSpannerSourceConnectorRegistry.java#L35-L50
The request is to add support for Spanner within the template by implementing a complementary connector, so that it is possible to build Spanner-Spanner live migrations as per: https://docs.cloud.google.com/spanner/docs/data-migration#live-data-migration
Alternatively, if this route is not planned to be supported, or if there is another, supported way to perform live migrations between Spanner databases, the request is to cover that route in documentation and tooling. So far, I have reviewed multiple possible routes and some seem possible but not covered, so it is not clear whether they are supported at all.
Workarounds
Performing "Spanner instance move" is not the same use case. It moves entire Spanner instances as-is, which is not suitable for cases like re-architecting the Spanner instances, which is what we currently do. As such, we need a way to perform a live migration of one Spanner DB at a time, between 2 arbitrary Spanner instances.
The workaround I found and tried is that you can stream your CDC directly between Spanner instances using Spanner to SourceDB streaming template, however this seems to lack official support and is not documented, so I have low confidence in this solution: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/spanner-to-sourcedb/README.md / https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/spanner-to-sourcedb/src/main/java/com/google/cloud/teleport/v2/templates/SpannerToSourceDb.java#L410
Yet another workaround, which I haven't tried, is to try the other template (Source DB to Spanner), which seems to accept just about any JDBC driver and connection string, so in theory you can provide Spanner JDBC driver. However, it lacks documentation on which drivers are available OOTB and which needs to be supplied by the user: https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/sourcedb-to-spanner/README.md#optional-parameters
Thank you!