[Feature] Add flat_json.column_paths to control which JSON paths are … - #1
Open
moweonlee wants to merge 9 commits into
Open
[Feature] Add flat_json.column_paths to control which JSON paths are …#1moweonlee wants to merge 9 commits into
moweonlee wants to merge 9 commits into
Conversation
…always columnized Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
github-actions
Bot
added
the
documentation
🌎 Translation Required?Thanks for your doc contribution! The following languages are missing or outdated for your changes. 🤖 Automated TranslationsIf you are fluent in any of the missing languages you can add them. If not, a maintainer can generate the translations below. Maintainer: Check the ones you wish to generate:
|
- Rename toThrift() back to toTFlatJsonConfig() in FlatJsonConfig.java; CreateReplicaTask.java still calls the original name and the rename would cause a compilation error. - Restore the 4-parameter update(bool, double, double, int) helper in flat_json_config.h that was removed in the refactor commit; other parts of the codebase may rely on it and the original author had intent behind exposing it. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…lay merge modifyTableProperties() uses putAll (merge, not replace). When a user removes all forced paths for a JSON column, the corresponding flat_json.column_paths.<col> key is absent from FlatJsonConfig.toProperties(). On follower FEs the stale key survives the putAll and causes buildFromProperties() to reconstruct a FlatJsonConfig with ghost paths, silently forcing paths that should no longer be columnized. Fix: purge all flat_json.column_paths.* keys from the existing properties map before the putAll when replaying OP_MODIFY_FLAT_JSON_CONFIG, mirroring the SchemaChangeHandler.removeIf() that already does this on the leader. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… CREATE TABLE CREATE TABLE did not verify that column names referenced in flat_json.column_paths.* properties are actual JSON-typed columns in the table schema. Unlike ALTER TABLE (which already validates via AlterTableClauseAnalyzer), CREATE TABLE silently accepted invalid column names and stored them, causing BE to silently ignore the configured paths. Added the same JSON-column existence check used in AlterTableClauseAnalyzer to processFlatJsonConfig() in OlapTableFactory so that CREATE TABLE fails fast with a clear error when a non-existent or non-JSON column is referenced.
…hs ALTER Three issues fixed: 1. AlterTableClauseAnalyzer: when flat_json.enable and flat_json.column_paths.* were present in the same ALTER TABLE, the else-if chain caused column_paths validation (JSON column existence check, path format) to be silently skipped. Extracted the validation into validateFlatJsonColumnPathsProperties() and call it from both the enable branch and the dedicated column_paths branch. 2. AlterTableClauseAnalyzer: the properties.size() != 1 guard did not include FLAT_JSON_PROPERTY_PREFIX, so users could not bundle flat_json.enable and flat_json.column_paths.* in a single ALTER TABLE statement. Added the prefix to the existing multi-property allowlist alongside dynamic_partition and binlog. 3. SchemaChangeHandler: stale flat_json.column_paths.* key removal ran without holding the write lock, creating a narrow window where SHOW CREATE TABLE could observe the old keys already erased but the new values not yet written by the WAL lambda. Moved the removal inside the WRITE lock block immediately before modifyFlatJsonMeta so the removal and WAL update are atomic. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…dled ALTER When flat_json.enable=false and flat_json.column_paths.* are combined in a single ALTER TABLE, the Analyzer previously called validateFlatJsonColumnPathsProperties without first verifying that enable=true, allowing an invalid combination to pass FE validation. The execution layer (updateFlatJsonConfigMeta) would then throw a RuntimeException instead of a proper SemanticException. Add an explicit guard in the enable branch: if flat_json.enable is not "true" and column_paths properties are present, report a SemanticException immediately so the error is clear and caught at the right layer. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
moweonlee
added a commit
that referenced
this pull request
May 26, 2026
…le orphans Address two follow-up review comments on flat_json ALTER propagation: - Round-trip flat_json.version through FlatJsonConfig.toProperties() and TableProperty.buildFlatJsonConfig() so FE failover preserves the version monotonicity the BE version-gate depends on. Without this, OP_MODIFY_FLAT_JSON_CONFIG replay reset version to 0 while BEs already held higher versions, causing subsequent ALTERs to be silently dropped by BE (new_version <= old_version). - Treat an unset flat_json_config_version on a tablet report as 0 in ReportHandler so tablets that never received an initial ALTER push (orphans from missed pushes) are picked up by heartbeat reconciliation instead of being skipped indefinitely. Verified both bugs end-to-end on a local FE+BE cluster by swapping between pre-fix and post-fix FE jars against the same BE: - EditLog persistence: with pre-fix FE, ALTER #1, ALTER #2, FE restart, ALTER #3 left BE holding a higher version than FE, with FE logging "BE flat_json_config version is N greater than version of FE". With post-fix FE the same sequence applied versions 1 and 2 cleanly and produced no such warning. - Orphan reconciliation: with pre-fix FE, an ALTER fired while BE was down (so the initial push timed out) left the tablet without flat_json_config and the FE skipped it on every subsequent tablet report. With post-fix FE the same setup, after restarting FE to drain any in-memory agent task queue, BE applied the config 21 ms after its first post-restart tablet report -- proving heartbeat reconciliation alone reaches orphans. Co-Authored-By: 김영진 <jini.kim@navercorp.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: 이모원[Log data] <moweon.lee@navercorp.com>
moweonlee
added a commit
that referenced
this pull request
May 26, 2026
…le orphans Address two follow-up review comments on flat_json ALTER propagation: - Round-trip flat_json.version through FlatJsonConfig.toProperties() and TableProperty.buildFlatJsonConfig() so FE failover preserves the version monotonicity the BE version-gate depends on. Without this, OP_MODIFY_FLAT_JSON_CONFIG replay reset version to 0 while BEs already held higher versions, causing subsequent ALTERs to be silently dropped by BE (new_version <= old_version). - Treat an unset flat_json_config_version on a tablet report as 0 in ReportHandler so tablets that never received an initial ALTER push (orphans from missed pushes) are picked up by heartbeat reconciliation instead of being skipped indefinitely. Verified both bugs end-to-end on a local FE+BE cluster by swapping between pre-fix and post-fix FE jars against the same BE: - EditLog persistence: with pre-fix FE, ALTER #1, ALTER #2, FE restart, ALTER #3 left BE holding a higher version than FE, with FE logging "BE flat_json_config version is N greater than version of FE". With post-fix FE the same sequence applied versions 1 and 2 cleanly and produced no such warning. - Orphan reconciliation: with pre-fix FE, an ALTER fired while BE was down (so the initial push timed out) left the tablet without flat_json_config and the FE skipped it on every subsequent tablet report. With post-fix FE the same setup, after restarting FE to drain any in-memory agent task queue, BE applied the config 21 ms after its first post-restart tablet report -- proving heartbeat reconciliation alone reaches orphans. Co-Authored-By: 김영진 <jini.kim@navercorp.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: 이모원[Log data] <moweon.lee@navercorp.com>
moweonlee
added a commit
that referenced
this pull request
May 28, 2026
…le orphans Address two follow-up review comments on flat_json ALTER propagation: - Round-trip flat_json.version through FlatJsonConfig.toProperties() and TableProperty.buildFlatJsonConfig() so FE failover preserves the version monotonicity the BE version-gate depends on. Without this, OP_MODIFY_FLAT_JSON_CONFIG replay reset version to 0 while BEs already held higher versions, causing subsequent ALTERs to be silently dropped by BE (new_version <= old_version). - Treat an unset flat_json_config_version on a tablet report as 0 in ReportHandler so tablets that never received an initial ALTER push (orphans from missed pushes) are picked up by heartbeat reconciliation instead of being skipped indefinitely. Verified both bugs end-to-end on a local FE+BE cluster by swapping between pre-fix and post-fix FE jars against the same BE: - EditLog persistence: with pre-fix FE, ALTER #1, ALTER #2, FE restart, ALTER #3 left BE holding a higher version than FE, with FE logging "BE flat_json_config version is N greater than version of FE". With post-fix FE the same sequence applied versions 1 and 2 cleanly and produced no such warning. - Orphan reconciliation: with pre-fix FE, an ALTER fired while BE was down (so the initial push timed out) left the tablet without flat_json_config and the FE skipped it on every subsequent tablet report. With post-fix FE the same setup, after restarting FE to drain any in-memory agent task queue, BE applied the config 21 ms after its first post-restart tablet report -- proving heartbeat reconciliation alone reaches orphans. Co-Authored-By: 김영진 <jini.kim@navercorp.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: 이모원[Log data] <moweon.lee@navercorp.com>
moweonlee
added a commit
that referenced
this pull request
May 28, 2026
…le orphans Address two follow-up review comments on flat_json ALTER propagation: - Round-trip flat_json.version through FlatJsonConfig.toProperties() and TableProperty.buildFlatJsonConfig() so FE failover preserves the version monotonicity the BE version-gate depends on. Without this, OP_MODIFY_FLAT_JSON_CONFIG replay reset version to 0 while BEs already held higher versions, causing subsequent ALTERs to be silently dropped by BE (new_version <= old_version). - Treat an unset flat_json_config_version on a tablet report as 0 in ReportHandler so tablets that never received an initial ALTER push (orphans from missed pushes) are picked up by heartbeat reconciliation instead of being skipped indefinitely. Verified both bugs end-to-end on a local FE+BE cluster by swapping between pre-fix and post-fix FE jars against the same BE: - EditLog persistence: with pre-fix FE, ALTER #1, ALTER #2, FE restart, ALTER #3 left BE holding a higher version than FE, with FE logging "BE flat_json_config version is N greater than version of FE". With post-fix FE the same sequence applied versions 1 and 2 cleanly and produced no such warning. - Orphan reconciliation: with pre-fix FE, an ALTER fired while BE was down (so the initial push timed out) left the tablet without flat_json_config and the FE skipped it on every subsequent tablet report. With post-fix FE the same setup, after restarting FE to drain any in-memory agent task queue, BE applied the config 21 ms after its first post-restart tablet report -- proving heartbeat reconciliation alone reaches orphans. Co-Authored-By: 김영진 <jini.kim@navercorp.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: 이모원[Log data] <moweon.lee@navercorp.com>
moweonlee
added a commit
that referenced
this pull request
Jun 9, 2026
…le orphans Address two follow-up review comments on flat_json ALTER propagation: - Round-trip flat_json.version through FlatJsonConfig.toProperties() and TableProperty.buildFlatJsonConfig() so FE failover preserves the version monotonicity the BE version-gate depends on. Without this, OP_MODIFY_FLAT_JSON_CONFIG replay reset version to 0 while BEs already held higher versions, causing subsequent ALTERs to be silently dropped by BE (new_version <= old_version). - Treat an unset flat_json_config_version on a tablet report as 0 in ReportHandler so tablets that never received an initial ALTER push (orphans from missed pushes) are picked up by heartbeat reconciliation instead of being skipped indefinitely. Verified both bugs end-to-end on a local FE+BE cluster by swapping between pre-fix and post-fix FE jars against the same BE: - EditLog persistence: with pre-fix FE, ALTER #1, ALTER #2, FE restart, ALTER #3 left BE holding a higher version than FE, with FE logging "BE flat_json_config version is N greater than version of FE". With post-fix FE the same sequence applied versions 1 and 2 cleanly and produced no such warning. - Orphan reconciliation: with pre-fix FE, an ALTER fired while BE was down (so the initial push timed out) left the tablet without flat_json_config and the FE skipped it on every subsequent tablet report. With post-fix FE the same setup, after restarting FE to drain any in-memory agent task queue, BE applied the config 21 ms after its first post-restart tablet report -- proving heartbeat reconciliation alone reaches orphans. Co-Authored-By: 김영진 <jini.kim@navercorp.com> Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: 이모원[Log data] <moweon.lee@navercorp.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…always columnized
Why I'm doing:
What I'm doing:
Fixes #issue
What type of PR is this:
Does this PR entail a change in behavior?
If yes, please specify the type of change:
Checklist:
Bugfix cherry-pick branch check: