Skip to content

rfc: spark structured streaming sink and source platform#11

Open
cgpoh wants to merge 10 commits intodatahub-project:mainfrom
cgpoh:spark-streaming-source-sink
Open

rfc: spark structured streaming sink and source platform#11
cgpoh wants to merge 10 commits intodatahub-project:mainfrom
cgpoh:spark-streaming-source-sink

Conversation

@cgpoh
Copy link

@cgpoh cgpoh commented Apr 21, 2025

Summary by CodeRabbit

  • New Features

    • Added support for specifying the data platform for Spark Structured Streaming sources and sinks via new configuration options, improving lineage capture for platforms without recognizable prefixes.
    • Introduced configuration to handle multiple streaming environments and platform instances within a single Spark application for better dataset URN distinction.
  • Documentation

    • Updated documentation to describe the new configuration properties for platform specification in streaming sources and sinks.

Loading
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant