libclang_rt.builtins.a

[mayeut]

From scikit-build/ninja-python-distributions#320 (comment)

Do the static clang builds include the libclang_rt builtins for the target architecture, or if not, could they be added?
I know the C library I'm trying to build needs them, and at a quick glance I couldn't seem to find the file in the static-clang-linux-* archives...

cc @mara004

[mayeut]

The default built-ins used here is libgcc so you don't likely need libclang_rt.builtins.a

[mara004]

The default built-ins used here is libgcc so you don't likely need libclang_rt.builtins.a

I don't know much about the topic, except that google's build system expects the libclang_rt.builtins.a.
When I pull a build from here and set it as clang_base_path, I'm getting this error, as anticipated:

['gn', 'gen', 'out/Default'] (cwd=PosixPath('/home/me/projects/pypdfium2/sbuild/native/pdfium'))
Done. Made 486 targets from 122 files in 786ms
['ninja', '-C', 'out/Default', 'pdfium'] (cwd=PosixPath('/home/me/projects/pypdfium2/sbuild/native/pdfium'))
ninja: Entering directory `out/Default'
ninja: error: '../../../../../clang/lib/clang/21/lib/x86_64-unknown-linux-gnu/libclang_rt.builtins.a', needed by 'libpdfium.so.TOC', missing and no known rule to make it
Traceback (most recent call last):
  File "/home/me/projects/pypdfium2/setupsrc/build_native.py", line 405, in <module>
    main_cli()
  File "/home/me/projects/pypdfium2/setupsrc/build_native.py", line 401, in main_cli
    main(**vars(args))
  File "/home/me/projects/pypdfium2/setupsrc/build_native.py", line 334, in main
    build(with_tests, build_dir, n_jobs)
  File "/home/me/projects/pypdfium2/setupsrc/build_native.py", line 267, in build
    run_cmd(["ninja", *ninja_args, "-C", str(build_dir_rel), *targets], cwd=PDFIUM_DIR)
  File "/home/me/projects/pypdfium2/setupsrc/base.py", line 647, in run_cmd
    comp_process = subprocess.run(command, cwd=cwd, check=check, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib64/python3.11/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['ninja', '-C', 'out/Default', 'pdfium']' returned non-zero exit status 1.
error: Recipe `build-native` failed on line 44 with exit code 1

[mayeut]

I don't know Google build system to provide help.
Either google uses the -rtlib=compiler-rt flag which overrides the default and/or they explicitly link against the builtins which should not be needed with default settings or -rtlib=libgcc.
You could try to copy libgcc in that place just to check if it's indeed useless but it would be better to setup the toolchain properly.
Please do check the output of:

/opt/clang/bin/clang -print-libgcc-file-name
/opt/clang/bin/clang -rtlib=libgcc -print-libgcc-file-name
/opt/clang/bin/clang -rtlib=compiler-rt -print-libgcc-file-name

[mayeut]

I forgot to ask, did you setup the configuration files as is done in scikit-build/ninja-python-distributions#320 ?

[mara004]

I forgot to ask, did you setup the configuration files as is done in scikit-build/ninja-python-distributions#320 ?

I had just unpacked one of the archives locally to check the libclang_rt.builtins.a situation, not used a container with full install-static-clang.sh setup yet. Thanks for the pointer, but I don't think this is relevant yet, when ninja fails due to missing dependency before even entering the compilation phase.

FWIW I'm not sure trying to patch out the libclang_rt dependency is a fruitful route, nor do I know how to do that.
Couldn't libclang_rt just be added to the builds here?

[mayeut]

Couldn't libclang_rt just be added to the builds here?

No it can't easily be done, building tools is different than building libraries for various targets:

• musl & glibc are different
• need to check if there are issues depending on the version of those
• need to package all target libraries in each deployed clang

[mara004]

Oh, right, I see.

I've asked on pdfium's mailing list about building with clang + libgcc rather than libclang_rt:
https://groups.google.com/g/pdfium/c/HLQUNdMqxCU/m/nG0duDYbCAAJ

[mayeut]

I tested building PDFium per https://github.com/pypdfium2-team/pypdfium2 instructions on ghcr.io/mayeut/manylinux_2_39:latest (Ubuntu 24.04).
using --compiler clang also seems to enable libc++ which is not supported either (I'm on aarch64 which might explain the difference):

['gn', 'gen', 'out/Default'] (cwd=PosixPath('/root/pypdfium2/sbuild/native/pdfium'))
ERROR at //build/config/BUILDCONFIG.gn:678:11: Unable to load "/root/pypdfium2/sbuild/native/pdfium/buildtools/third_party/libc++/BUILD.gn".
          "//buildtools/third_party/libc++:std_ctype_h",
          ^--------------------------------------------

Just symlinking clang/clang++/... as gcc/gcc++/... (with arch prefix) and building using --compiler gcc seems to work (clang being mostly compatible with gcc, not that surprising) meaning nothing is missing here, it's a build configuration issue.

[mara004]

We have a patch for the error you mention: https://github.com/pypdfium2-team/pypdfium2/blob/main/patches/system_libcxx_with_clang.patch (cf. https://crbug.com/410883044)

The error occurs when the initial checkout was run without --compiler clang, so you may need to pass --reset --compiler clang for the patch to be applied.

Just symlinking clang/clang++/... as gcc/gcc++/... (with arch prefix) and building using --compiler gcc seems to work (clang being mostly compatible with gcc, not that surprising) meaning nothing is missing here, it's a build configuration issue.

Interesting. Though that sounds like a hack, and I might prefer a different approach.
I know it's the build config, but I'm only a downstream and no expert at tweaking their build files.
pdfium's build system is a bit difficult to deal with for an outsider, as it does not respect the usual env vars.

[mara004]

Aah, it was actually way easier than thought, I just had to comment out the wohle clang_lib("compiler_builtins") block in //build/config/clang/BUILD.gn, and comment out this use in //build/config/compiler/BUILD.gn:

  if (is_clang && !is_nacl && !is_cronet_build) {
    configs += [ "//build/config/clang:compiler_builtins" ]
  }

I'm currently at [300/668], let's see if it goes through. This build mode generates a lot of warnings though, probably because upstream aren't using this config at all.

[mara004]

Ok, the x86_64 build did indeed work, after yet applying a symlink to fix the linkage step:
ln -s $PWD/clang/bin/readelf $PWD/clang/bin/llvm-readelf.

I plan to integrate the new patches into build_native.py, and then test cross-compilation in a container, with your install script.

Thank you for the help here, very much appreciated!
Given that building libclang_rt.builtins appears to be out of scope for this project, this issue could be closed from my part.

[mayeut]

I came too the same conclusion.
It will fail because it looks for llvm-readelf which is installed as readelf
ln -s readelf /opt/clang/bin/llvm-readelf

[mayeut]

PS, building on ppc64le will fail as it's not supported upstream.
simple patch that also takes care of not trying to link any clang libraries (including builtins) since none are provided here:

pypdfium2/sbuild/native/pdfium/build/config/clang/BUILD.gn, added || true

template("clang_lib") {
  if (!defined(invoker.libname) || is_wasm || true) {
    not_needed(invoker, "*")
    config(target_name) {
    }

ppc64le build on aarch64 in less than a minute (excluding clone/sync, just building after removing out folder).

time python3.12 ./setupsrc/build_native.py --compiler clang --clang-path /opt/clang
....
[759/759] SOLINK ./libpdfium.so
Packing data files for sourcebuild...
Using reference bindings - this will bypass all bindings params. If this is not intentional, make sure ctypesgen is installed.
Warning: binary/bindings version mismatch (7191 != 7483). This is ABI-unsafe!

real	0m57.600s
user	7m55.426s
sys	0m44.646s