Convert remaining JLL stdlibs to LazyLibraries #58444

IanButterworth · 2025-05-17T00:38:26Z

Builds on #58405 (which I rebased, so the merge base here is master for now)

TODO:

Convert the new Zstd_jll from [deps] enable zstd support #58344
~~Convert LLD_jll?~~ No, it's an executable not a lib
~~Convert p7zip_jll?~~ No, it's an executable not a lib
~~Blocked by --trim failure~~ trimming: no way to ccall into a dynamically-computed library name #57707 Instead it copies libzstd over for testing with a const path
Fix linux RR test failure - release Fix _ossl_modules_path() returning wrong path to ossl-modules JuliaWeb/OpenSSL.jl#40

topolarity · 2025-05-17T01:57:20Z

Unfortunately the --trim failure is expected until we resolve #57707:

Core.MethodError(f=Base.Libc.Libdl.var"#dlsym"(), args=("/cache/julia-buildkite-plugin/depots/53de1ea5-6dd3-4008-b976-161c9f34b0ad/artifacts/4db1e58d71ac6bbb35ecc832033264c630d5d3b3/lib/libzstd.so", :ZSTD_versionString), world=0x00000000000097b7)

(LinearAlgebra is already broken for --trim on master for the same reason)

We do have a plan to fix it, but I don't know when it's going to happen since @Keno would like to revamp our ccall semantics (#57931) around the same time

IanButterworth · 2025-05-17T03:41:12Z

Ah ok.
In its current state, along with JuliaSparse/SparseArrays.jl#626 this PR gives

macOS M2 compared to results in #58436

julia> @time using Plots
  0.850111 seconds (1.29 M allocations: 74.658 MiB, 6.23% gc time, 2.34% compilation time)

Linux AMD Ryzen 9 5950X

julia> @time using Plots
  0.757390 seconds (1.38 M allocations: 77.337 MiB, 4.01% gc time, 3.59% compilation time)

And basic startup appears to be faster (if that makes sense?)
master vs. PR

% hyperfine './julia --start=no -e "1+1"' '/Users/ian/.julia/juliaup/julia-nightly/bin/julia --start=no -e "1+1"'
Benchmark 1: ./julia --start=no -e "1+1"
  Time (mean ± σ):      89.3 ms ±   3.6 ms    [User: 58.6 ms, System: 29.0 ms]
  Range (min … max):    85.4 ms …  98.2 ms    30 runs

Benchmark 2: /Users/ian/.julia/juliaup/julia-nightly/bin/julia --start=no -e "1+1"
  Time (mean ± σ):      99.2 ms ±   2.1 ms    [User: 67.8 ms, System: 29.7 ms]
  Range (min … max):    97.0 ms … 107.3 ms    27 runs

Summary
  ./julia --start=no -e "1+1" ran
    1.11 ± 0.05 times faster than /Users/ian/.julia/juliaup/julia-nightly/bin/julia --start=no -e "1+1"

IanButterworth · 2025-05-17T03:42:41Z

stdlib/GMP_jll/test/runtests.jl

@@ -3,6 +3,6 @@
 using Test, Libdl, GMP_jll

 @testset "GMP_jll" begin
-    vn = VersionNumber(unsafe_string(unsafe_load(cglobal((:__gmp_version, libgmp), Ptr{Cchar}))))
+    vn = VersionNumber(unsafe_string(unsafe_load(cglobal(dlsym(libgmp, :__gmp_version), Ptr{Cchar}))))


Should/could cglobal be taught how to handle a LazyLibrary?

It could, the most annoying thing is just that cglobal() is an intrinsic, not a function, so to add an overload for that involves writing C code, not just Julia code.

stdlib/GMP_jll/src/GMP_jll.jl

stdlib/libLLVM_jll/src/libLLVM_jll.jl

stdlib/OpenLibm_jll/src/OpenLibm_jll.jl

stdlib/PCRE2_jll/src/PCRE2_jll.jl

stdlib/OpenLibm_jll/src/OpenLibm_jll.jl

stdlib/SuiteSparse_jll/src/SuiteSparse_jll.jl

topolarity · 2025-05-17T15:10:21Z

Can / should we add a test that checks the dependency list against the ldd (or DT_NEEDED entries) of the .so's?

vtjnash · 2025-05-17T15:43:33Z

BinaryBuilder should already audit that, except for a few know broken libraries

topolarity · 2025-05-17T16:43:55Z

BinaryBuilder should already audit that, except for a few know broken libraries

Right, but it would be good to make sure that we haven't fallen out-of-sync with BinaryBuilder

staticfloat · 2025-05-19T15:07:02Z

it would be good to make sure that we haven't fallen out-of-sync with BinaryBuilder

I agree, I added a test that can be run on macOS or Linux. I (or someone else feeling brave) can make one for Windows as well. It already catches plenty of minor issues:

┌ Warning: Dependency mismatch
│   jll = "SuiteSparse_jll"
│   library = "libbtf"
│   missing_deps = ""
│   extraneous_deps = "libsuitesparseconfig, libamd, libcolamd"
└ @ Main ~/src/julia/test/stdlib_dependencies.jl:169

┌ Warning: Dependency mismatch
│   jll = "SuiteSparse_jll"
│   library = "libldl"
│   missing_deps = ""
│   extraneous_deps = "libsuitesparseconfig"
└ @ Main ~/src/julia/test/stdlib_dependencies.jl:169

┌ Warning: Dependency mismatch
│   jll = "SuiteSparse_jll"
│   library = "libspqr"
│   missing_deps = "libstdc++, libgcc_s"
│   extraneous_deps = "libamd, libcolamd"
└ @ Main ~/src/julia/test/stdlib_dependencies.jl:169

┌ Warning: Dependency mismatch
│   jll = "SuiteSparse_jll"
│   library = "libumfpack"
│   missing_deps = ""
│   extraneous_deps = "libcolamd"
└ @ Main ~/src/julia/test/stdlib_dependencies.jl:169

┌ Warning: Dependency mismatch
│   jll = "MPFR_jll"
│   library = "libmpfr"
│   missing_deps = ""
│   extraneous_deps = "libgmpxx"
└ @ Main ~/src/julia/test/stdlib_dependencies.jl:169
Test Summary:               | Pass  Fail  Total  Time
Stdlib JLL dependency check |   29     5     34  3.9s
RNG of the outermost testset: Random.Xoshiro(0x1329e9815a60ab0e, 0x784c9414423a89d0, 0x1a3aea28fda405f6, 0x8c1dfb6d4dda4f7b, 0xd4081888363c92a9)
ERROR: LoadError: Some tests did not pass: 29 passed, 5 failed, 0 errored, 0 broken.
in expression starting at /home/sabae/src/julia/test/stdlib_dependencies.jl:132

Note that there are some slight differences between macOS and Linux, so you'll have to add some conditionals for those things.

IanButterworth · 2025-05-19T15:37:56Z

Nice

vtjnash · 2025-05-19T21:05:29Z

Blocked by --trim failure

That should not actually be blocking. The required code dependencies should be vendored into the test directory anyways, rather than relying on changeable state elsewhere, since otherwise we can lose test coverage simply if these package files change, which should not be true of test code.

stdlib/LibSSH2_jll/src/LibSSH2_jll.jl

stdlib/SuiteSparse_jll/src/SuiteSparse_jll.jl

staticfloat · 2025-05-20T04:52:24Z

The required code dependencies should be vendored into the test directory anyways, rather than relying on changeable state elsewhere, since otherwise we can lose test coverage simply if these package files change, which should not be true of test code.

I don't quite follow what you mean, can you be more explicit?

stdlib/LibCURL_jll/src/LibCURL_jll.jl

test/stdlib_dependencies.jl

This converts more JLL stdlibs to use lazily-loaded libraries, and as a useful side-effect, causes them to be loaded by absolute path, isolating them all from `LD_LIBRARY_PATH`-like effects. Adds tests that audits the expressed library dependencies in our LazyLibrary JLL definitions, to ensure that we don't get out of sync with the actual binaries. Co-Authored-By: Mosè Giordano <[email protected]> Co-Authored-By: Elliot Saba <[email protected]>

IanButterworth · 2025-05-27T20:52:44Z

test/trimming/basic_jll.jl

@@ -3,7 +3,7 @@ using Zstd_jll

 function @main(args::Vector{String})::Cint
    println(Core.stdout, "Julia! Hello, world!")
-    fptr = dlsym(Zstd_jll.libzstd_handle, :ZSTD_versionString)
+    fptr = dlsym(Zstd_jll.libzstd, :ZSTD_versionString)


@vtjnash @topolarity @staticfloat It'd be good to figure out what to do about the trim blocker/non-blocker. Jameson/Cody, I don't know if you've discussed offline at all?

Copy (vendor) the old files into the test/trimming directory and proceed with the changes

That's OK by me - I still consider this a --trim regression, but it's worth allowing it so that we can get LazyLibary in for now.

We've got a plan to fix-up LazyLibrary for --trim soon enough and at that point we'll bump the test to use the latest standard JLL's (incl. LazyLibrary)

Ok. Pushed.

I'm getting a segfault locally on macOS. Not sure why

/Users/ian/Documents/GitHub/julia/usr/bin/julia --depwarn=error /Users/ian/Documents/GitHub/julia/test/trimming/trimming.jl /Users/ian/Documents/GitHub/julia/usr/bin/hello [98446] signal 11 (2): Segmentation fault: 11 in expression starting at none:0 jl_lookup_generic_ at /Users/ian/Documents/GitHub/julia/src/gf.c:3613 [inlined] ijl_apply_generic at /Users/ian/Documents/GitHub/julia/src/gf.c:3686 main at /Users/ian/Documents/GitHub/julia/usr/bin/basic_jll (unknown line) _main at /Users/ian/Documents/GitHub/julia/usr/bin/basic_jll (unknown line) main at /Users/ian/Documents/GitHub/julia/usr/bin/basic_jll (unknown line) Allocations: 1 (Pool: 1; Big: 0); GC: 0 ERROR: LoadError: failed process: Process(`/Users/ian/Documents/GitHub/julia/usr/bin/basic_jll`, ProcessSignaled(11)) [0]

IanButterworth requested a review from staticfloat May 17, 2025 00:38

IanButterworth requested a review from giordano as a code owner May 17, 2025 00:38

IanButterworth mentioned this pull request May 17, 2025

Convert more stdlibs to LazyLibraries #58405

Open

IanButterworth force-pushed the ib/more_lazy branch from 6bb0540 to f8388c5 Compare May 17, 2025 03:09

IanButterworth commented May 17, 2025

View reviewed changes

IanButterworth added the latency Latency label May 17, 2025

staticfloat requested changes May 17, 2025

View reviewed changes

IanButterworth force-pushed the ib/more_lazy branch 4 times, most recently from 7fb4a84 to 46fe088 Compare May 17, 2025 12:19

This comment was marked as resolved.

Sign in to view

IanButterworth force-pushed the ib/more_lazy branch from 46fe088 to 75229fe Compare May 17, 2025 18:18

nsajko added the JLLs label May 18, 2025

IanButterworth mentioned this pull request May 18, 2025

precompile the eager_mode noop JuliaPackaging/JLLWrappers.jl#75

Merged

IanButterworth mentioned this pull request May 19, 2025

Package load time regressions in 1.12 #57436

Open

staticfloat approved these changes May 20, 2025

View reviewed changes

stdlib/LibSSH2_jll/src/LibSSH2_jll.jl Show resolved Hide resolved

stdlib/SuiteSparse_jll/src/SuiteSparse_jll.jl Show resolved Hide resolved

IanButterworth force-pushed the ib/more_lazy branch 3 times, most recently from a415b58 to 488c8e9 Compare May 20, 2025 21:38

topolarity reviewed May 20, 2025

View reviewed changes

stdlib/LibCURL_jll/src/LibCURL_jll.jl Outdated Show resolved Hide resolved

IanButterworth force-pushed the ib/more_lazy branch 2 times, most recently from cdbfd6e to 7bf2956 Compare May 22, 2025 19:08

giordano reviewed May 23, 2025

View reviewed changes

test/stdlib_dependencies.jl Outdated Show resolved Hide resolved

IanButterworth force-pushed the ib/more_lazy branch from eda06e2 to ccb1c94 Compare May 23, 2025 17:48

giordano reviewed May 23, 2025

View reviewed changes

test/stdlib_dependencies.jl Outdated Show resolved Hide resolved

IanButterworth force-pushed the ib/more_lazy branch from ccb1c94 to 35aecff Compare May 23, 2025 17:52

giordano reviewed May 23, 2025

View reviewed changes

test/stdlib_dependencies.jl Outdated Show resolved Hide resolved

IanButterworth force-pushed the ib/more_lazy branch 2 times, most recently from 1d13585 to 1afe38a Compare May 27, 2025 14:02

IanButterworth force-pushed the ib/more_lazy branch from 1afe38a to 3a9548b Compare May 27, 2025 16:57

IanButterworth commented May 27, 2025

View reviewed changes

copy libzstd over for trimming tests

dc4c2dc

IanButterworth force-pushed the ib/more_lazy branch from d8c226d to dc4c2dc Compare May 28, 2025 03:59

Uh oh!

Convert remaining JLL stdlibs to LazyLibraries #58444

Are you sure you want to change the base?

Convert remaining JLL stdlibs to LazyLibraries #58444

Conversation

IanButterworth commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

topolarity commented May 17, 2025

Uh oh!

IanButterworth commented May 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

IanButterworth May 17, 2025

Choose a reason for hiding this comment

Uh oh!

staticfloat May 17, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

topolarity commented May 17, 2025

Uh oh!

This comment was marked as resolved.

vtjnash commented May 17, 2025

Uh oh!

topolarity commented May 17, 2025

Uh oh!

staticfloat commented May 19, 2025

Uh oh!

IanButterworth commented May 19, 2025

Uh oh!

vtjnash commented May 19, 2025

Uh oh!

Uh oh!

Uh oh!

staticfloat commented May 20, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

IanButterworth May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

vtjnash May 27, 2025

Choose a reason for hiding this comment

Uh oh!

topolarity May 27, 2025

Choose a reason for hiding this comment

Uh oh!

IanButterworth May 28, 2025

Choose a reason for hiding this comment

Uh oh!

IanButterworth May 28, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

IanButterworth commented May 17, 2025 •

edited

Loading

IanButterworth commented May 17, 2025 •

edited

Loading

IanButterworth May 27, 2025 •

edited

Loading