[BugFix] Fix pre-1970 Parquet timestamp load corrupting sub-second DATETIME by xiangguangyxg · Pull Request #75207 · StarRocks/starrocks

xiangguangyxg · 2026-06-23T07:48:38Z

Why I'm doing:

When loading a Parquet INT64 column annotated TIMESTAMP (isAdjustedToUTC=false) into a StarRocks DATETIME, a pre-1970 value with a nonzero sub-second part was decoded to a corrupt garbage value instead of the real wall clock.

Int64ToDateTimeConverter::convert splits the signed epoch tick with C++ truncating division:

int64_t seconds     = src_data[i] / _second_mask;
int64_t nanoseconds = (src_data[i] % _second_mask) * _scale_to_nano_factor;

For a negative tick whose sub-second remainder is nonzero, nanoseconds is negative. timestamp::of_epoch_second then packs the result via a bitwise OR (from_julian_and_time), so the negative microsecond corrupts the packed Julian field — e.g. 1969-12-31 23:59:59.500 loaded as a year-41222 garbage value. Whole-second negatives and all post-1970 values were unaffected.

What I'm doing:

Borrow a whole second when the sub-second remainder is negative, so nanoseconds stays in [0, NANOSECS_PER_SEC) — the floor split the FE boundary computation already uses (Math.floorDiv/Math.floorMod). of_epoch_second then receives a non-negative sub-second and packs the correct value. The borrow is unit-agnostic (MILLIS/MICROS/NANOS) and runs before the UTC whole-second offset, so it composes with the timezone-adjusted branch unchanged.

Added a regression test (Int64PreEpochTimestampSubSecond) that loads a Parquet file holding 1969-12-31 23:59:59.500000 in both MILLIS and MICROS units: it decoded to a garbage value before the fix and to the correct wall clock after. The existing post-1970 Int64_2_Timestamp test is unchanged (14/14 ColumnConverterTest pass).

What type of PR is this:

Does this PR entail a change in behavior?

Yes, this PR will result in a change in behavior.
No, this PR will not result in a change in behavior.

Checklist:

I have added test cases for my bug fix or my new feature
This pr needs user documentation (for new or modified features or behaviors)
- I have added documentation for my new feature or new function
- This pr needs auto generate documentation
This is a backport pr

Bugfix cherry-pick branch check:

I have checked the version labels which the pr will be auto-backported to the target branch
- 4.1
- 4.0
- 3.5

…TETIME When loading a Parquet INT64 TIMESTAMP (isAdjustedToUTC=false) into a DATETIME, Int64ToDateTimeConverter split the signed epoch tick with C++ truncating division. For a pre-1970 tick with a nonzero sub-second part the remainder is negative, so nanoseconds was negative; timestamp::of_epoch_second then packs it via a bitwise OR (from_julian_and_time), and the negative microsecond corrupts the packed Julian field — the loaded DATETIME became a garbage value (e.g. year 41222) instead of the real wall clock. Borrow a whole second when the sub-second remainder is negative so nanoseconds stays in [0, NANOSECS_PER_SEC), matching the floor split the FE boundary computation uses. Whole-second and post-1970 values are unaffected. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: xiangguangyxg <xiangguangyxg@gmail.com>

CelerData-Reviewer · 2026-06-23T07:49:33Z

@codex review

xiangguangyxg · 2026-06-23T07:49:45Z

@codex review

chatgpt-codex-connector · 2026-06-23T07:53:35Z

Codex Review: Didn't find any major issues. Delightful!

Reviewed commit: 3abdcc04ec

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

github-actions · 2026-06-24T10:34:55Z

[Java-Extensions Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-06-24T10:34:59Z

[FE Incremental Coverage Report]

✅ pass : 0 / 0 (0%)

github-actions · 2026-06-24T10:52:09Z

[BE Incremental Coverage Report]

✅ pass : 3 / 3 (100.00%)

file detail

	path	covered_line	new_line	coverage	not_covered_line_detail
🔵	be/src/formats/parquet/column_converter.cpp	3	3	100.00%	[]

mergify · 2026-06-26T02:07:57Z

Tick the box to add this pull request to the merge queue (same as @mergifyio queue).

Queue this pull request

github-actions · 2026-06-26T02:11:46Z

@Mergifyio backport branch-4.1

mergify · 2026-06-26T02:11:50Z

backport branch-4.1

✅ Backports have been created

Details

#75385 [BugFix] Fix pre-1970 Parquet timestamp load corrupting sub-second DATETIME (backport #75207) has been created for branch branch-4.1

…TETIME (backport #75207) (#75385) Signed-off-by: xiangguangyxg <xiangguangyxg@gmail.com> Co-authored-by: xiangguangyxg <110401425+xiangguangyxg@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

OrcTimestampHelper::orc_ts_to_native_ts_before_unix_epoch hardcoded the microsecond argument to 0, so loading any pre-1970 ORC TIMESTAMP with a non-zero sub-second dropped the fraction (e.g. 1965-03-02 12:00:00.500000 loaded as 1965-03-02 12:00:00.000000) on both the plain and the instant (timestamp with local time zone) read paths. Pass nanoseconds / NANOSECS_PER_USEC instead, mirroring the after-epoch path. The shared helper is also used by the ORC stripe min/max statistics decoder; keep its pre-1970 bounds byte-for-byte unchanged by dropping the sub-second for negative-epoch bounds there, leaving the decoder's separate pre-existing stats-nanos handling for a follow-up. Companion to the merged Parquet fix StarRocks#75207. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: xiangguangyxg <xiangguangyxg@gmail.com>

github-actions Bot added the 4.1 label Jun 23, 2026

mergify Bot assigned xiangguangyxg Jun 23, 2026

github-actions Bot requested review from dirtysalt and trueeyu June 23, 2026 07:50

xiangguangyxg requested a review from kevincai June 24, 2026 13:20

kevincai approved these changes Jun 25, 2026

View reviewed changes

srlch approved these changes Jun 26, 2026

View reviewed changes

xiangguangyxg merged commit 0f52fa0 into StarRocks:main Jun 26, 2026
109 of 112 checks passed

xiangguangyxg deleted the be-parquet-pre1970-ts branch June 26, 2026 02:11

github-actions Bot removed the 4.1 label Jun 26, 2026

mergify Bot mentioned this pull request Jun 26, 2026

[BugFix] Fix pre-1970 Parquet timestamp load corrupting sub-second DATETIME (backport #75207) #75385

Merged

18 tasks

github-actions Bot added the 4.1-merged label Jun 26, 2026

This was referenced Jun 26, 2026

[Enhancement] Widen meta-tier DATETIME pre-split window to year 1 #75427

Open

[BugFix] Preserve sub-second when loading pre-1970 ORC TIMESTAMP #75432

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[BugFix] Fix pre-1970 Parquet timestamp load corrupting sub-second DATETIME#75207

[BugFix] Fix pre-1970 Parquet timestamp load corrupting sub-second DATETIME#75207
xiangguangyxg merged 1 commit into
StarRocks:mainfrom
xiangguangyxg:be-parquet-pre1970-ts

xiangguangyxg commented Jun 23, 2026

Uh oh!

CelerData-Reviewer commented Jun 23, 2026

Uh oh!

xiangguangyxg commented Jun 23, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

Uh oh!

mergify Bot commented Jun 26, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

mergify Bot commented Jun 26, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

xiangguangyxg commented Jun 23, 2026

Why I'm doing:

What I'm doing:

What type of PR is this:

Checklist:

Bugfix cherry-pick branch check:

Uh oh!

CelerData-Reviewer commented Jun 23, 2026

Uh oh!

xiangguangyxg commented Jun 23, 2026

Uh oh!

chatgpt-codex-connector Bot commented Jun 23, 2026

Uh oh!

github-actions Bot commented Jun 24, 2026

[Java-Extensions Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jun 24, 2026

[FE Incremental Coverage Report]

Uh oh!

github-actions Bot commented Jun 24, 2026

[BE Incremental Coverage Report]

file detail

Uh oh!

mergify Bot commented Jun 26, 2026

Uh oh!

Uh oh!

github-actions Bot commented Jun 26, 2026

Uh oh!

mergify Bot commented Jun 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

✅ Backports have been created

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

mergify Bot commented Jun 26, 2026 •

edited

Loading