Use a better and consistent naming conventions for source properties

[TennyZhuang]

I’d like to start a discussion about the property style, although it may introduce some breaking change, it’s very important and it will be much harder if we do it in the future.

Why it’s important?

The properties (aka storage parameter) is the WITH clause defined by ourselves in some DDL.

As soon as our users try to get started with RisingWave, they have to create a source and use some properties. If the name conventions are very weird or inconsistent, it will give users a poor impression even before they use RisingWave.

What's the current state?

I'm not sure where our current style refer from, but I guess it's from FlinkSQL. Unfortunately, it seems that currently we mixed many styles and even some mistakes by ourselves.

Why all properties are prefixed by the connector name?

Our naming of kafka source is almost same as FlinkSQL, but always prefixed with kafka:

FlinkSQL example:

CREATE TABLE KafkaTable (
  `user_id` BIGINT,
  `item_id` BIGINT,
  `behavior` STRING,
  `ts` TIMESTAMP(3) METADATA FROM 'timestamp'
) WITH (
  'connector' = 'kafka',
  'topic' = 'user_behavior',
  'properties.bootstrap.servers' = 'localhost:9092',
  'properties.group.id' = 'testGroup',
  'scan.startup.mode' = 'earliest-offset',
  'format' = 'csv'
)

Our example:

CREATE MATERIALIZED SOURCE IF NOT EXISTS source_abc (
   column1 varchar,
   column2 integer,
)
WITH (
   connector='kafka',
   kafka.topic='demo_topic',
   kafka.brokers='172.10.1.1:9090,172.10.1.2:9090',
   kafka.scan.startup.mode='latest',
   kafka.time.offset='140000000',
   kafka.consumer.group='demo_consumer_name'
)
ROW FORMAT JSON;

So why?

Some connectors use dot as separator

In DataGen and Nexmark source, they may refer to kafka example but misunderstood that the dot is used as a separator. The dot is represented as a sub-field operator but not a separator. For example, in FlinkSQL, they have a property scan.topic-partition-discovery.interval, so the below property names are definitely wrong:

• datagen.split.num
• datagen.rows.per.second
• nexmark.in.flight.auctions
• nexmark.out.of.order.group.size
• etc.

I guess it may be very weired for native speakers.

Why mixing dot and underscore?

We use . as sub-field operator and _ as separator, e.g. kinesis.credentials.session_token, which I can't find any reference.

There are three projects we can find and refer to:

• FlinkSQL uses . as sub-field operator and - as separator, e.g. scan.topic-partition-discovery.interval
• Materialize fully uses _, e.g. topic_metadata_refresh_interval_ms
• PostgreSQL storage parameter almost uses _, e.g. autovacuum_vacuum_cost_delay, but use . for toast table.

What should we do?

We have few users in production now and IMHO the breaking changes are acceptable. We should choose one option and write some guides for it. Every reviewer of the connector module and the doc team should be fully aware of the guide and introduce new properties carefully.

[TennyZhuang]

Personally, I am inclined toward pg's style.

[fuyufjh]

Either LGTM.

I guess Flink's style is inspired by Java properties file. Anyway, Flink SQL is built on Flink so they must share lots of components.

One thing that I totally agree with is that in our style the dot should be replaced with underscore, e.g. kafka.brokers should be replaced with kafka_brokers. Otherwise, it looks confusing with the identifiers (database.schema.table).

[fuyufjh]

I just found some strange things 🤣

nexmark.min.event.gap.in.ns

The dots should be used as a separator for module/components (like packages) instead of "space", so the correct name is

nexmark.min_event_gap_in_ns
// or
nexmark.min-event-gap-in-ns