SIGSEGV race: worker threads execute a corrupted task clone in the arrow result collector path (ArrowTest/CSR tests fail in CI)

[adsharma]

Summary

The linux minimal test CI job crashes with SIGSEGV in every test that builds an ArrowQueryResult:

• ArrowTest.queryAsArrow, ArrowTest.getArrowResult
• ArrowTest.queryAsArrowDirectCSRRowIDProjection (+ KeepsCSRMetadataWithFourThreads)
• ArrowTest.queryAsArrowTracksCSRMetadataWithoutRelIDs / WithRelIDsAndExtraColumns / DoesNotTrackCSRMetadataForNonCSRShape
• ProjectGraphCsrTest.materializesArrowCsr, ProjectGraphCsrTest.materializedCsrSurvivesConsumingQueries
• ReadOnlyTest.ProjectGraphOnReadOnlyDatabase

Some crash with Fatal signal 11 right after the test's COPY setup finishes (i.e. during the arrow query itself); the ProjectGraphCsrTest ones fail on entry.relCsrResults[0] == nullptr — queryAsArrow returned a result without CSR metadata because the collector's tracking state was corrupted.

The crash is timing-dependent: it reproduces on nearly every CI run (small/slow 2-4 core runners) but rarely on larger local machines. It is pre-existing on main (verified as described below), not caused by any specific PR — first noticed in #878 CI.

Evidence (instrumented ASAN run)

With env-gated tracing added to the arrow collector pipeline (LBUG_ARROW_DEBUG=1), the crash reproduces deterministically, also on pristine main (dd9ca6cc7, clean ASAN worktree, no other changes). Instrumented commit (patch on top of main or #878, identical signature both ways): 0a3ad1d97 on branch fix/877-cached-plan-reuse-state.

[arrow-dbg] copy: src=0x51500001e800 new=0x515000031e00 srcShared=0x50f000007df0   ← valid clone #1 (worker A)
[arrow-dbg] copy: src=0x51500001e800 new=0x515000023800 srcShared=0x50f000007df0   ← valid clone #2 (worker B)
[arrow-dbg] init-ft: this=0x50f000007df0 shared=(nil) batch=<garbage>              ← executed object is the SHARED STATE address
[arrow-dbg] exec-ft: this=0x50f000007df0 shared=0x6b53b39fe630                     ← members are garbage
Fatal signal 11

Facts:

1. Both worker threads create valid ArrowResultCollector clones via ProcessorTask::run() → sink->copy() (new=0x515000031e00 / 0x515000023800).
2. The worker threads then execute Sink::execute → initLocalStateInternal / executeInternal on an object whose this equals the ArrowResultCollectorSharedState heap block address (srcShared), not their clone.
3. On that bogus this, members read as garbage: sharedState = 0x1 / (nil), batchIndex = garbage, and csrMetadata.has_value() == true for a CSR-free query.
4. taskRoot (raw pointer) is printed unchanged immediately after copy() and immediately before taskRoot->ptrCast<Sink>()->execute(...), yet executeInternal receives a different this — so the clone's memory (vtable region included) is corrupted concurrently between task creation and task execution, i.e. a data race, and the dynamic/virtual dispatch on the corrupted object produces the bogus this.

Interpretation

Something writes into the freshly created task clone (or the plan root's memory that it was cloned from) between ProcessorTask::run() cloning the sink and the worker executing it. Prime suspects:

• The Phase-3 thread-local ResultSet cache in ProcessorTask::run() (keyed by ResultSetDescriptor::id) interacting with task/plan lifetime.
• The plan/PhysicalPlan (and the original sink) being torn down or mutated while worker threads still clone/execute from it.
• ArrowRowBatch::toArray() buffers aliasing storage that is freed when the producing pipeline's ResultSet/vectors die, with a later writer reusing that heap.

Reproducing

1. Check out main (confirmed on dd9ca6cc7).
2. Apply the instrumentation patch from commit 0a3ad1d97 (branch fix/877-cached-plan-reuse-state, now removed from the branch tip — patch is purely additive, marked REMOVE ME).
3. Build api_test with ASAN and run:

LBUG_ARROW_DEBUG=1 ASAN_OPTIONS=detect_leaks=0 ./build/asan/test/api/api_test --gtest_filter='ArrowTest.queryAsArrow'

The crash reproduces deterministically with instrumentation (the extra fprintf latency widens the race window) and intermittently in CI without it. All instrumentation output is prefixed [arrow-dbg] with a per-thread id.

Suggested next steps

• Run the api test suite under TSAN — the corruption is a race, so TSAN should point at the exact pair of accesses.
• Audit the lifetime of the mapped plan / original sink vs. worker-thread clones in QueryProcessor::execute() and ProcessorTask::run(), and the ResultSetDescriptor-keyed thread-local ResultSet reuse introduced by the plan-caching work (Fix state reuse on the cached-physical-plan fast path (0-row scans, live-result clobbering) #870).