Skip to content

Read Arrow rows directly from vectors with per-batch converters - #628

Merged
vuanhphung merged 4 commits into
mainfrom
perf/row-convert
Nov 19, 2026
Merged

vuanhphung merged 4 commits into
mainfrom
perf/row-convert

Conversation

@vuanhphung

@vuanhphung vuanhphung commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Arrow object materialization spends most CPU repeatedly dispatching types and expanding StructRow proxies: a decoded-batch profile attributes 76% of sampled time to the recursive converter. Both Thrift and kernel results use this path.

Read top-level vectors directly into final rows and resolve scalar converters once per batch. Keep nested conversion and the legacy object path for duplicate/special names unchanged. Values, precision options, logging, disabled materialization and IPC/fetching behavior stay the same.

Verification: 1,831 unit tests pass (agent markers cleared; the unrelated default-environment user-agent failure also reproduces on main), all static checks pass, and 47 live e2e cases plus type/limit/lifecycle checksum matrices pass. Fresh-process live medians below use five interleaved pairs on Node 24; p95/p99 and raw runs are in the factory handoff. Decoded DOUBLE/object CPU: 480 → 45 ms/million rows. Array-mode heap peaks can increase with GC slack; bounded-heap runs still complete.

Follow-up verification: fallback tests now reject both guard-removal mutants cleanly; all 1,831 units and static checks pass, and the shared Thrift/kernel row-materialization specs each pass 8 live cases with no skips; kernel CI now detects shared result changes.

Live query / row mode Thrift main → branch Kernel main → branch
DOUBLE 30M / object 20.52 → 5.30 s 23.58 → 5.22 s
DOUBLE 30M / array 8.15 → 5.87 s 8.13 → 5.99 s
lineitem 2M / object 29.60 → 20.30 s 27.49 → 18.35 s
lineitem 2M / array 21.58 → 19.71 s 20.65 → 16.75 s

Adversarial review: converged in round 1; gains and parity independently reproduced, fallback/precision/binary tests strengthened and mutation-checked, array-mode memory caveat documented.

This pull request and its description were written by Isaac.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@vuanhphung
vuanhphung marked this pull request as ready for review November 19, 2026 09:25

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

Signed-off-by: Vu Anh Phung <vu.phung@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>

@peco-review-bot peco-review-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).

✅ No issues identified.

@aakash-saravanan-db aakash-saravanan-db left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Resolving converters once per record batch and reading vectors directly removes the repeated per-cell work at the right boundary. The guarded fallback and regression coverage give good confidence that existing result semantics are preserved.

Non-blocking test suggestion: could we extend the parity tests with nested and dictionary-encoded BIGINT/DECIMAL values under preserveBigNumericPrecision: true, including nulls and values beyond the safe integer range, and compare against the legacy conversion path? Top-level precision coverage is strong; these combinations would further protect future converter changes.

@vuanhphung
vuanhphung added this pull request to the merge queue Nov 19, 2026
Merged via the queue into main with commit d9bd91f Nov 19, 2026
23 checks passed

This branch was previously deployed

1 inactive deployment
azure-prod — 2d394f14 Deployed Nov 19, 2026 by vuanhphung via e2e-test (26.12.1, 26_12_1) #2154
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants