Skip to content

[FLINK-40264][format/avro] Propagate legacyTimestampMapping into nested row converters - #28847

Open
kumarpritam863 wants to merge 1 commit into
apache:masterfrom
kumarpritam863:FLINK-40264
Open

[FLINK-40264][format/avro] Propagate legacyTimestampMapping into nested row converters#28847
kumarpritam863 wants to merge 1 commit into
apache:masterfrom
kumarpritam863:FLINK-40264

Conversation

@kumarpritam863

Copy link
Copy Markdown

What is the purpose of the change

AvroToRowDataConverters#createConverter dispatches a nested ROW to the single-argument overload:

case ROW:
    return createRowConverter((RowType) type);

createRowConverter(RowType) defaults legacyTimestampMapping to true, so the flag is dropped for every nesting level below the top one. Since TIMESTAMP_LTZ is only supported by the non-legacy mapping, a nested TIMESTAMP_LTZ column
fails even when the non-legacy mapping was explicitly requested:

java.lang.UnsupportedOperationException: Unsupported type: TIMESTAMP_LTZ(3) NOT NULL

The failure happens while the deserialization schema is being constructed, so the job never starts. AvroSchemaConverter#convertToSchema handles the same type correctly with legacyTimestampMapping = false, and RowDataToAvroConverters
propagates the flag properly, so only the deserialization side is affected.

Introduced by FLINK-33198, first released in 1.19.0.

Brief change log

- `AvroToRowDataConverters` now passes `legacyTimestampMapping` down when building the converter for a nested `ROW`.

This is strictly a widening: the only code paths whose behaviour changes are ones that throw today.

Verifying this change

This change added tests and can be verified as follows:

- Added `AvroRowDataDeSerializationSchemaTest#testTimestampTypeNewMappingInNestedRow`, which round-trips a row holding a nested row with a `TIMESTAMP_LTZ(3)` column under the non-legacy mapping. Without the one-line fix it fails at

construction with Unsupported type: TIMESTAMP_LTZ(3) NOT NULL.
- Ran mvn clean verify for flink-formats/flink-avro and flink-formats/flink-avro-confluent-registry: 365 and 27 tests respectively, no failures, 0 checkstyle violations, spotless and ArchUnit clean, japicmp reports no incompatibility.

Does this pull request potentially affect one of the following parts:

- Dependencies (does it add or upgrade a dependency): **no**
- The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no**
- The serializers: **yes** — `AvroToRowDataConverters`, but only for nested rows under the non-legacy timestamp mapping, which cannot be constructed at all today.
- The runtime per-record code paths (performance sensitive): **no**
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: **no**
- The S3 file system connector: **no**

Documentation

- Does this pull request introduce a new feature? **no**
- If yes, how is the feature documented? **not applicable**

Note for reviewers

This is a backport candidate: the bug is present in every release from 1.19.0 onward.

FLINK-40262, submitted separately, restructures createRowConverter into createRowConverterInternal and in doing so necessarily threads the same flag, so it fixes this symptom as a side effect but carries no regression test for it. Merging
this one first keeps the test; whichever lands second needs a trivial rebase.


Was generative AI tooling used to co-author this PR?
  • Yes (please specify the tool below)

Generated-by: Claude Code (Anthropic Claude Opus 5)


…ed row converters

AvroToRowDataConverters#createConverter dispatched nested ROW types to the
single-argument createRowConverter overload, which always assumes the legacy
timestamp mapping. As a result a nested TIMESTAMP_LTZ column failed with
'Unsupported type: TIMESTAMP_LTZ' even when the non-legacy mapping was
requested, because only the non-legacy mapping supports that type.
@flinkbot

flinkbot commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

CI report:

Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants