GH-49382: [Python] Enable OpenTelemetry on PyArrow wheels#49383
GH-49382: [Python] Enable OpenTelemetry on PyArrow wheels#49383raulcd wants to merge 9 commits intoapache:mainfrom
Conversation
|
|
|
@github-actions crossbow submit wheel-manylinux-2-28-cp313-cp313-amd64 |
|
Revision: 7426f25 Submitted crossbow builds: ursacomputing/crossbow @ actions-1e10ae4c04
|
…ple and fix using the proper arrow tracer singleton
| options.parent = otel::trace::GetSpan(new_otel_context)->GetContext(); | ||
|
|
||
| auto tracer = otel::trace::Provider::GetTracerProvider()->GetTracer("arrow"); | ||
| auto tracer = arrow::internal::tracing::GetTracer(); |
There was a problem hiding this comment.
@lidavidm I am not entirely sure why this is necessary but using the example middleware.py enabling otel as updated below with a running server:
$ export OTEL_SERVICE_NAME=arrow-raul
$ ARROW_TRACING_BACKEND=otlp_http python examples/flight/middleware.py server --listen grpc://localhost:5050 --otel
and client:
python examples/flight/middleware.py client grpc://localhost:5050
I wasn't able to get traces unless I update this to use arrow::internal::tracing::GetTracer. Example of local traces with the built wheel and this changes:
Initially I thought it might be due to the addition of CMAKE_INTERPROCEDURAL_OPTIMIZATION to the wheels but I've validated is unrelated.
Without this change I haven't been able to get opentelemetry traces using the flight.TracingServerMiddlewareFactory
There was a problem hiding this comment.
Hmm, it's been a while since I've worked with OTel...possibly the OTel trace provider isn't being configured (vs the hardcoded Arrow one)? The Arrow util is about equivalent, but does some static initialization first
There was a problem hiding this comment.
yeah, that was my feeling too, either not being initialized or something to do with libarrow.so and libarrow_flight.so both linking opentelementry statically and getting their own copy of the static opentelemetry provider that's why I thought CMAKE_INTERPROCEDUREAL_OPTIMIZATION might be related (but proved it wasn't).
The Arrow util one should cross the shared object boundary and use the already initialized one (or initialize it) which I think is correct.
I just wanted to validate the change looks ok to you or whether do you think it would be better to keep investigating and try to configure when calling MakeTracingServerMiddlewareFactory instead of using the one initialized via libarrow.so?
github-actions
bot
added
awaiting changes
github-actions
bot
added
awaiting change review
|
@github-actions crossbow submit wheel-macos-monterey-cp313-cp313-amd64 |
|
Revision: 9ccb832 Submitted crossbow builds: ursacomputing/crossbow @ actions-c6d347423f
|
|
@github-actions crossbow submit wheel-windows-cp313-cp313-amd64 wheel-musllinux-1-2-cp313-cp313-amd64 |
|
Revision: 8d2daf8 Submitted crossbow builds: ursacomputing/crossbow @ actions-4d34d6d1e1
|
github-actions
bot
added
awaiting changes
|
@github-actions crossbow submit wheel-windows-cp313-cp313-amd64 |
github-actions
bot
added
awaiting change review
|
Revision: be1de48 Submitted crossbow builds: ursacomputing/crossbow @ actions-3168ee0ef2
|
…/exec_plan.cc uses UnwrapSpan
|
@github-actions crossbow submit wheel-windows-cp313-cp313-amd64 |
|
Revision: 2bbd485 Submitted crossbow builds: ursacomputing/crossbow @ actions-2be70cfc4f
|
| }; | ||
|
|
||
| opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span>& UnwrapSpan( | ||
| ARROW_EXPORT opentelemetry::nostd::shared_ptr<opentelemetry::trace::Span>& UnwrapSpan( |
There was a problem hiding this comment.
These symbols seem to be required to be exported for Windows:
2026-02-25T17:46:03.6802906Z
2026-02-25T17:46:03.6803059Z Finished searching libraries
2026-02-25T17:46:03.6813183Z unity_3_cxx.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::RewrapSpan(class arrow::util::tracing::SpanDetails *,class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span>)" (?RewrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@V4567@@Z) [C:\arrow-build\src\arrow\acero\arrow_acero_shared.vcxproj]
2026-02-25T17:46:03.6817761Z unity_2_cxx.obj : error LNK2001: unresolved external symbol "class opentelemetry::v1::nostd::shared_ptr<class opentelemetry::v1::trace::Span> & __cdecl arrow::internal::tracing::UnwrapSpan(class arrow::util::tracing::SpanDetails *)" (?UnwrapSpan@tracing@internal@arrow@@YAAEAV?$shared_ptr@VSpan@trace@v1@opentelemetry@@@nostd@v1@opentelemetry@@PEAVSpanDetails@1util@3@@Z) [C:\arrow-build\src\arrow\acero\arrow_acero_shared.vcxproj]
2026-02-25T17:46:03.6820710Z C:\arrow-build\release\Release\arrow_acero.dll : fatal error LNK1120: 2 unresolved externals [C:\arrow-build\src\arrow\acero\arrow_acero_shared.vcxproj]
|
@github-actions crossbow submit -g wheel |
|
Revision: 2bbd485 Submitted crossbow builds: ursacomputing/crossbow @ actions-b2c694ec7c
|
raulcd
requested review from
AlenkaF,
assignUser,
jonkeane,
kou and
rok
as code owners
github-actions
bot
added
awaiting merge
python/pyarrow/__init__.py
Outdated
| for module in modules: | ||
| status = "Enabled" if _module_is_available(module) else "-" | ||
| print(f" {module: <20}: {status: <8}") | ||
| print(f" {"opentelemetry": <20}: {"Enabled" if is_opentelemetry_enabled() else "-": <8}") |
There was a problem hiding this comment.
Example of output:
>>> pyarrow.show_info()
pyarrow version info
--------------------
Package kind : python-wheel-manylinux228
Arrow C++ library version : 24.0.0-SNAPSHOT
Arrow C++ compiler : GNU 14.2.1
Arrow C++ compiler flags : -Wno-noexcept-type -Wno-self-move -Wno-subobject-linkage -fdiagnostics-color=always -Wall -fno-semantic-interposition -msse4.2
Arrow C++ git revision :
Arrow C++ git description :
Arrow C++ build type : release
PyArrow build type : release
Platform:
OS / Arch : Linux x86_64
SIMD Level : avx2
Detected SIMD Level : avx2
Memory:
Default backend : mimalloc
Bytes allocated : 0 bytes
Max memory : 0 bytes
Supported Backends : mimalloc, jemalloc, system
Optional modules:
csv : Enabled
cuda : -
dataset : Enabled
feather : Enabled
flight : Enabled
fs : Enabled
gandiva : -
json : Enabled
orc : Enabled
parquet : Enabled
opentelemetry : Enabled
Filesystems:
AzureFileSystem : Enabled
GcsFileSystem : Enabled
HadoopFileSystem : Enabled
S3FileSystem : Enabled
Compression Codecs:
brotli : Enabled
bz2 : Enabled
gzip : Enabled
lz4_frame : Enabled
lz4 : Enabled
snappy : Enabled
zstd : Enabled
…dout when using flight.TracingServerMiddlewareFactory
github-actions
bot
added
awaiting change review
|
@github-actions crossbow submit -g wheel -g python |
This comment was marked as outdated.
This comment was marked as outdated.
|
@github-actions crossbow submit -g wheel -g python |
|
Revision: 0357767 Submitted crossbow builds: ursacomputing/crossbow @ actions-2156854d96
|
3 participants
Rationale for this change
Currently we are not building wheels with OpenTelemetry.
What changes are included in this PR?
vcpkg.jsonto install required opentelemetry-cppARROW_WITH_OPENTELEMETRY=ONon the wheel build scriptspython/examples/flight/middleware.pyto be able to easily enable opentelemetry middleware tracing for testing.server_tracing_middleware.ccto retrieve tracer fromarrow::internal::tracing::GetTracer();instead ofotel::trace::Provider::GetTracerProvider()->GetTracer("arrow");is_opentelemetry_enabledutility function in pyarrow to validate whether OpenTelemetry was enabled or not when building Arrow C++. Expose this information onpa.show_info().Are these changes tested?
Yes, wheels are being successfully built and tested locally with a built wheel that traces are being generated when properly configured.
Added test to validate traces appear on stdout and I've also validated Acero traces are being correctly generated by running the test suite and seeing the opentelemetry traces on Jaeger UI.
Are there any user-facing changes?
Yes, PyArrow wheel users will be able to enable OpenTelemetry traces.