[Umbrella][Feature] Datastream API Support

[polyzos]

Search before asking

• I searched in the issues and found nothing similar.

Motivation

Although the implementation is there, we need to provide a cleaner source/sink API as it is currently somewhat messy and has a lot of arguments.

Solution

• Introduce Deserialization Schemas #625
• Introduce FlussSource for Flink DataStream API #626
• Introduce FlussSink for Flink DataStream API #627
• Add Flink DataStream Source/Sink Documentation #628

Source Features

• Add deserialization schema
• Make FlinkSource and related classes Generic
• Introduce FlussSource
• Introduce RowConverter Utils
• Support Projection Pushdown

Fluss Source Usage:

   FlussSource<Order> flussSource =
                FlussSource.<Order>builder()
                        .setBootstrapServers(bootstrapServers)
                        .setDatabase(DEFAULT_DB)
                        .setTable(pkTableName)
                        .setStartingOffsets(OffsetsInitializer.earliest())
                        .setScanPartitionDiscoveryIntervalMs(1000L)
                        .setDeserializationSchema(new OrderDeserializationSchema())
                        .build();

        DataStreamSource<Order> stream =
                env.fromSource(flussSource, WatermarkStrategy.noWatermarks(), "Fluss Source");

Currently, it supports setting some key properties via the builder pattern, but to avoid overwhelming users they can set the rest with Configuration via the setFlussConfig(). We can always add more if required and after feedback, I just thought it's better like this for now.

Fluss Sink Usage:

Anything else?

No response

Willingness to contribute

• I'm willing to submit a PR!

[wuchong]

Thanks @polyzos for opening it and taking care of this. Let's discuss the proposed API first here.

I think Kafka and MongoDB connectors are good example to learn from.

• https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/kafka/
• https://nightlies.apache.org/flink/flink-docs-release-2.0/docs/connectors/datastream/mongodb/

[wuchong]

@polyzos Do you want to contribute this issue?

[polyzos]

@wuchong Hopefully yes.. I'm gonna spend some time investigating and think of the design and will circle back.

[polyzos]

Design Proposal
https://docs.google.com/document/d/1EZxKgTJV5tUlzE7DlWbeQIvBTNvOElCAqNqtVj3URFs/edit?tab=t.0