[FLINK-40264][format/avro] Propagate legacyTimestampMapping into nested row converters - #28847
Open
kumarpritam863 wants to merge 1 commit into
Open
[FLINK-40264][format/avro] Propagate legacyTimestampMapping into nested row converters#28847kumarpritam863 wants to merge 1 commit into
kumarpritam863 wants to merge 1 commit into
Conversation
…ed row converters AvroToRowDataConverters#createConverter dispatched nested ROW types to the single-argument createRowConverter overload, which always assumes the legacy timestamp mapping. As a result a nested TIMESTAMP_LTZ column failed with 'Unsupported type: TIMESTAMP_LTZ' even when the non-legacy mapping was requested, because only the non-legacy mapping supports that type.
Collaborator
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
AvroToRowDataConverters#createConverterdispatches a nestedROWto the single-argument overload:createRowConverter(RowType)defaultslegacyTimestampMappingtotrue, so the flag is dropped for every nesting level below the top one. SinceTIMESTAMP_LTZis only supported by the non-legacy mapping, a nestedTIMESTAMP_LTZcolumnfails even when the non-legacy mapping was explicitly requested:
The failure happens while the deserialization schema is being constructed, so the job never starts.
AvroSchemaConverter#convertToSchemahandles the same type correctly withlegacyTimestampMapping = false, andRowDataToAvroConverterspropagates the flag properly, so only the deserialization side is affected.
Introduced by FLINK-33198, first released in 1.19.0.
Brief change log
This is strictly a widening: the only code paths whose behaviour changes are ones that throw today.
Verifying this change
This change added tests and can be verified as follows:
construction with
Unsupported type: TIMESTAMP_LTZ(3) NOT NULL.- Ran
mvn clean verifyforflink-formats/flink-avroandflink-formats/flink-avro-confluent-registry: 365 and 27 tests respectively, no failures, 0 checkstyle violations, spotless and ArchUnit clean, japicmp reports no incompatibility.Does this pull request potentially affect one of the following parts:
Documentation
Note for reviewers
This is a backport candidate: the bug is present in every release from 1.19.0 onward.
FLINK-40262, submitted separately, restructures
createRowConverterintocreateRowConverterInternaland in doing so necessarily threads the same flag, so it fixes this symptom as a side effect but carries no regression test for it. Mergingthis one first keeps the test; whichever lands second needs a trivial rebase.
Was generative AI tooling used to co-author this PR?
Generated-by: Claude Code (Anthropic Claude Opus 5)