GH-49272: [C++][CI] Fix intermittent segfault in arrow-json-test on M…#49462
Open
vanshaj2023 wants to merge 1 commit intoapache:mainfrom
Open
GH-49272: [C++][CI] Fix intermittent segfault in arrow-json-test on M…#49462vanshaj2023 wants to merge 1 commit intoapache:mainfrom
vanshaj2023 wants to merge 1 commit intoapache:mainfrom
Conversation
kou
reviewed
Mar 6, 2026
cpp/src/arrow/util/thread_pool.cc
Outdated
Comment on lines
+29
to
+31
| #if defined(__MINGW32__) || defined(__MINGW64__) | ||
| # include "arrow/util/windows_compatibility.h" | ||
| #endif |
Member
There was a problem hiding this comment.
We can always include windows_compatibility.h because it does nothing on non Windows.
Suggested change
| #if defined(__MINGW32__) || defined(__MINGW64__) | |
| # include "arrow/util/windows_compatibility.h" | |
| #endif | |
| #include "arrow/util/windows_compatibility.h" |
cpp/src/arrow/util/thread_pool.cc
Outdated
| static DWORD index = [] { | ||
| DWORD i = TlsAlloc(); | ||
| if (i == TLS_OUT_OF_INDEXES) { | ||
| ARROW_LOG(FATAL) << "TlsAlloc failed for thread pool TLS"; |
Member
There was a problem hiding this comment.
Can we show error details?
Suggested change
| ARROW_LOG(FATAL) << "TlsAlloc failed for thread pool TLS"; | |
| ARROW_LOG(FATAL) << "TlsAlloc failed for thread pool TLS: " << internal::WinErrorMessage(GetLastError()); |
cpp/src/arrow/json/reader_test.cc
Outdated
Comment on lines
+293
to
+317
| // Stress test for intermittent threading crashes on MinGW. | ||
| // See https://github.com/apache/arrow/issues/49272 | ||
| TEST(ReaderTest, MultipleChunksParallelStress) { | ||
| constexpr int kTrials = 20; | ||
| for (int trial = 0; trial < kTrials; ++trial) { | ||
| int64_t count = 1 << 10; | ||
| ParseOptions parse_options; | ||
| parse_options.unexpected_field_behavior = UnexpectedFieldBehavior::InferType; | ||
| ReadOptions read_options; | ||
| read_options.block_size = static_cast<int>(count / 2); | ||
| read_options.use_threads = true; | ||
|
|
||
| std::string json; | ||
| for (int i = 0; i < count; ++i) { | ||
| json += "{\"a\":" + std::to_string(i) + "}\n"; | ||
| } | ||
|
|
||
| std::shared_ptr<io::InputStream> input; | ||
| ASSERT_OK(MakeStream(json, &input)); | ||
| ASSERT_OK_AND_ASSIGN(auto reader, TableReader::Make(default_memory_pool(), input, | ||
| read_options, parse_options)); | ||
| ASSERT_OK_AND_ASSIGN(auto table, reader->Read()); | ||
| ASSERT_EQ(table->num_rows(), count); | ||
| } | ||
| } |
Member
There was a problem hiding this comment.
Is this a heavy test?
If so, we don't want to this test in our CI. Can we disable it by default something like the following?
TEST(...) {
if (!std::getenv("ARROW_SLOW_TEST_ENABLE")) {
GTEST_SKIP() << ...;
}
...
}
Author
There was a problem hiding this comment.
No, it is not too heavy, each iteration is identical to the existing MultipleChunksParallel test (1024 small JSON lines), just repeated 20 times to increase the chance of hitting the race condition. It finishes in a few seconds.
Author
There was a problem hiding this comment.
Also we reduce the iteration count if you think that's better for CI.
Member
There was a problem hiding this comment.
OK. Can we remove for (int trial = 0; trial < kTrials; ++trial) {? We can reproduce this by running this test multiple times manually, right? (e.g. while build/debug/debug/arrow-json-test ...; do :; done)
Author
There was a problem hiding this comment.
Sure, I will update the code
Author
|
Thanks for the review @kou |
github-actions
bot
added
awaiting changes
…t on MinGW MinGW's __emutls implementation for C++ thread_local has known race conditions during thread creation. When ThreadPool::LaunchWorkersUnlocked creates a new worker thread, the thread writes to the thread_local current_thread_pool_ variable. If __emutls hasn't finished initializing TLS for the new thread, this dereferences an invalid pointer, causing a segfault. Replace thread_local with native Win32 TLS API (TlsAlloc/TlsGetValue/ TlsSetValue) on MinGW to bypass the buggy __emutls emulation. Non-MinGW platforms continue using standard thread_local. Also add a stress test (MultipleChunksParallelStress) that runs parallel JSON reading 20 times to help expose intermittent threading races.
vanshaj2023
force-pushed
the
fix-mingw-json-test-segfault
branch
from
62da992 to
1e0fba3
Compare
github-actions
bot
added
awaiting review
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
The
arrow-json-testintermittently segfaults on AMD64 Windows MinGW CI (both CLANG64 and MINGW64 environments), causing false CI failures. The crash occurs at 0.03 seconds into test execution duringReaderTest.MultipleChunksParallel. See #49272.Example failing runs:
The root cause is MinGW's
__emutlsimplementation for C++thread_local, which has known race conditions during thread creation. WhenThreadPool::LaunchWorkersUnlockedcreates a new worker thread, the thread immediately writes to thethread_localcurrent_thread_pool_variable. If__emutlshasn't finished initializing TLS for the new thread, this dereferences an invalid pointer, causing a segfault.What changes are included in this PR?
Replace
thread_localwith native Win32 TLS API on MinGW (thread_pool.cc): UsesTlsAlloc/TlsGetValue/TlsSetValuevia Arrow'swindows_compatibility.hinstead ofthread_localon MinGW to bypass the buggy__emutlsemulation. Non-MinGW platforms are unchanged. Includes error handling forTlsAllocfailure.Add
MultipleChunksParallelStresstest (reader_test.cc): Runs parallel JSON reading 20 times to help expose intermittent threading races.Are these changes tested?
Yes. A new stress test (
ReaderTest.MultipleChunksParallelStress) is added that repeatedly exercises the parallel JSON reading path. The existingReaderTest.MultipleChunksParalleltest also covers the affected code path.Are there any user-facing changes?
No.
This PR contains a "Critical Fix". The changes fix a bug that causes a crash (segfault) in
arrow-json-teston MinGW Windows CI due to a race condition in MinGW's__emutlsthread-local storage emulation during thread pool worker creation.