Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[connector] Support spark catalog and introduce some basic classes #544

Open
wants to merge 11 commits into
base: main
Choose a base branch
from
Open

[connector] Support spark catalog and introduce some basic classes #544

wants to merge 11 commits into from

Conversation

Alibaba-HZY
Copy link
Contributor

Purpose

Linked issue: open #228
Linked issue: open #155

Support spark catalog and introduce some basic classes to support spark read and write

Brief change log

1.Based on #258 and discussion with @XuQianJin-Stars , I resubmitted [support spakr catalog ] and introduce some basic classes(SparkInternalRow,SparkRow,FlussInputPartition) to support spark read and write.
2.support SupportsPartitionManagement in spark table

Tests

com.alibaba.fluss.connector.spark.catalog.SparkCatalogITCase
com.alibaba.fluss.connector.spark.utils.SparkConversionsTest
com.alibaba.fluss.connector.spark.utils.SparkTypeUtilsTest
com.alibaba.fluss.connector.spark.SparkInternalRowTest

API and Format

no

Documentation

no

@Alibaba-HZY
Copy link
Contributor Author

@wuchong cc

@Alibaba-HZY Alibaba-HZY force-pushed the spark-catalog branch 4 times, most recently from b90a1f8 to abe3482 Compare March 10, 2025 03:46
@Alibaba-HZY
Merge branch 'refs/heads/main' into spark-3.3-catalog
12f9d26 
# Conflicts:
#	fluss-connectors/fluss-connector-flink/src/main/java/com/alibaba/fluss/connector/flink/catalog/FlinkCatalog.java
#	fluss-connectors/fluss-connector-spark/fluss-connector-spark-common/src/main/java/com/alibaba/fluss/connector/spark/exception/CatalogException.java
@Alibaba-HZY
Merge branch 'refs/heads/main' into spark-3.3-catalog
089c172
@XuQianJin-Stars
Copy link
Contributor

hi @Alibaba-HZY This PR looks good overall, I will review it later. Thanks.

@wuchong
Copy link
Member

wuchong commented Mar 14, 2025

Sorry, I just merged #597 and causes some conflicts in this PR. The main change of #594 is moving fluss-connectors/fluss-connector-flink to a top-level module fluss-flink with sub-modules for different Flink versions. This can make the module structure much simpler.

Therefore, we can also move fluss-connectors/fluss-connector-flink to a top-level module fluss-spark with sub-modules for different Spark versions.

@Alibaba-HZY
Copy link
Contributor Author

Sorry, I just merged #597 and causes some conflicts in this PR. The main change of #594 is moving fluss-connectors/fluss-connector-flink to a top-level module fluss-flink with sub-modules for different Flink versions. This can make the module structure much simpler.

Therefore, we can also move fluss-connectors/fluss-connector-flink to a top-level module fluss-spark with sub-modules for different Spark versions.

ok i will handle this

@Alibaba-HZY
Copy link
Contributor Author

Sorry, I just merged #597 and causes some conflicts in this PR. The main change of #594 is moving fluss-connectors/fluss-connector-flink to a top-level module fluss-flink with sub-modules for different Flink versions. This can make the module structure much simpler.

Therefore, we can also move fluss-connectors/fluss-connector-flink to a top-level module fluss-spark with sub-modules for different Spark versions.

down

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@XuQianJin-Stars XuQianJin-Stars XuQianJin-Stars left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@Alibaba-HZY @XuQianJin-Stars @wuchong @chunxiaozheng