Skip to content

fix(libSceSsl): keep the bundled mbedtls symbols out of libSceSsl's exports on macOS - #10530

Merged
adriadam10 merged 1 commit into
boykopovar:mainfrom
maximilianfv:fix/libscessl-macos-hide-mbedtls
Oct 26, 2026
Merged

adriadam10 merged 1 commit into
boykopovar:mainfrom
maximilianfv:fix/libscessl-macos-hide-mbedtls

Conversation

@maximilianfv

@maximilianfv maximilianfv commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

What

On Linux and MinGW, --exclude-libs,ALL keeps the statically linked Mbed TLS out of libSceSsl's dynamic exports. Apple's linker has no such option, so on macOS libSceSsl.prx exports 930 mbedtls_* and 155 psa_* symbols (54 of its 1,139 exports are libSceSsl's own). A program that links its own Mbed TLS can then bind to libSceSsl's copy, which allocates from libSceSsl's context pool. guest_ssl_transport_kernel and guest_ssl_transport_net fail on macOS this way: their test TLS server links Mbed TLS itself. The analysis in #8658 (commit 8a4bd1e) traced this to the server's mbedtls_ssl_setup resolving to libSceSsl's copy and failing with MBEDTLS_ERR_SSL_ALLOC_FAILED.

This builds the mbedtls, mbedcrypto, mbedx509 and p256m targets with hidden C visibility on Apple. These targets are shared, so the change also reaches libSceLibreSsl, which links mbedcrypto and exported 27 mbedtls_* symbols on macOS; it now exports none. The Linux and MinGW paths are unchanged.

Prior art, credited:

pr_overlap.py (run serially): no overlapping PRs. It could not compare #6044 because #6044 conflicts with main. #6044 changes core/libs/prx/libSceSsl/CMakeLists.txt only in the BUILD_TESTING block, not the lines changed here.

Tested

Apple M1 Pro, macOS 27.0.1, x86_64 build under Rosetta 2. main alone does not build on macOS, so I tested on mugurc's macOS series (ci/macos-series-2, 27e9689) merged with main at 1e20650, plus the macOS build parts of #10419, toggling only this change.

without with
nm -gU libSceSsl.prx 1,139 exports (930 mbedtls_, 155 psa_) 54 exports, no mbedtls_ or psa_
nm -gU libSceLibreSsl.prx 427 exports (27 mbedtls_) 400 exports, no mbedtls_
ctest -R 'guest_ssl|guest_libressl|guest_http' (14 tests) 12 pass; guest_ssl_transport_kernel and guest_ssl_transport_net fail with a server certificate without a loaded CA passed the initial verify options 14 pass

The five guest_libressl* tests pass both ways. On the series alone (27e9689, without the main merge), the same two transport tests failed without the change and all 9 ssl|http tests passed with it.

Not tested on Linux or Windows. Their CMake path is unchanged.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: n/a
  • AI-assisted: yes (Claude Code counted the exports, ran the tests and wrote this description; I am responsible for every line)

🤖 Generated with Claude Code

…xports on macOS

On Linux and MinGW, --exclude-libs keeps the statically linked mbedtls
out of libSceSsl's dynamic exports. Apple's linker has no such option,
so on macOS libSceSsl.prx exported 930 mbedtls and 155 PSA symbols, and
libSceLibreSsl.prx 27 mbedtls symbols. A program that links its own
mbedtls can then bind to libSceSsl's copy, which allocates from
libSceSsl's own pool; guest_ssl_transport_kernel and
guest_ssl_transport_net fail that way. Build the mbedtls libraries with
hidden visibility on macOS instead.

Co-authored-by: Ycaro Gabriel <280482597+ycarogabriel-lgtm@users.noreply.github.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed the exact two-line CMake change and the reported export/test comparison. It applies hidden C visibility to the four bundled Mbed TLS targets only on Apple; Linux/MinGW retain --exclude-libs,ALL and other platforms are untouched. The earlier implementation is cited, but it is not in current upstream main, and the PR provides a macOS-specific report showing the resulting export set and SSL transport regression coverage. The automatic build jobs for this PR are skipped; I am relying on the described macOS run, not treating it as hosted CI.

@ycbuzi

ycbuzi commented Oct 25, 2026

Copy link
Copy Markdown
Contributor

Windows run of f5dffcb12f merged onto main 49d9c1c751 (5090879925, clean). Windows 11 Home 10.0.26200, WinLibs MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release, BUILD_TESTING=ON.

  • Built the four guest_ssl* test executables with no errors or warnings.
  • ctest --repeat until-fail:2: guest_ssl, guest_ssl_load, guest_ssl_transport_kernel, guest_ssl_transport_net and guest_ssl_ps5_readers pass both runs.
  • The MinGW path stays as the description says: objdump -p on the built libSceSsl.prx lists 84 exports, none of them mbedtls_* or psa_*, so --exclude-libs,ALL still keeps Mbed TLS out on Windows. The Apple branch itself can't be exercised here.

Not run: a title. AI-assisted: yes (Claude Code)

@imellag imellag left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tested head f5dffcb12f on Apple M4 Pro, macOS 27.0.1, Apple clang 21, x86-64 under Rosetta 2. The tree is main 61fc572cee plus the open macOS build PRs and the parts of mugurc's series that #7201 also carries. Only this change was toggled.

  • Without it: nm -gU libSceSsl.prx lists 1,085 mbedtls_*/psa_* symbols. guest_ssl_transport_kernel and guest_ssl_transport_net fail 2 runs out of 2, at a server certificate without a loaded CA passed the initial verify options.
  • With it: libSceSsl.prx exports 55 symbols and none of them is mbedtls_* or psa_*. libSceLibreSsl.prx exports none either. Both transport tests pass 3 runs out of 3.
  • ctest -R 'ssl|http|libressl|crypto|tls|x509' (63 tests): 61 pass without this change and all 63 with it. Only the two transport tests change.

AI-assisted: yes

@krishalivaibhav

Copy link
Copy Markdown
Contributor

Test report: #10530 (hide libSceSsl's bundled mbedtls symbols on macOS)

Tested on: Apple M2 MacBook Air, macOS 27.2 (26B118), x86_64 Release build under Rosetta 2, AppleClang 21.0.0, Ninja.
Tree: origin/main a5a780e merged into a scratch branch holding our open macOS stack (integ8 029564a43: #10552, #10853 and their dependencies, plus the process-parameters, mlock-PROT_NONE and math-exact fixes). Base commit 1ddafb62d. PR head f5dffcb. Merges cleanly.

test (3 runs each) without with
guest_ssl pass 3/3 pass 3/3
guest_ssl_load pass 3/3 pass 3/3
guest_ssl_ps5_readers pass 3/3 pass 3/3
guest_ssl_transport_net fail 3/3 abort 3/3
guest_ssl_transport_kernel fail 3/3 abort 3/3

The two transport tests already fail on this stack without the PR (net: "a server certificate without a loaded CA passed the initial verify options"). With the PR they no longer start: dyld: Symbol not found: _mbedtls_ctr_drbg_free, referenced from tests/guest_ssl_transport_tests, expected in libSceSsl.prx. That test executable uses the bundled mbedtls symbols that the PR stops exporting. I did not check guest-visible behaviour; for the test target, mbedtls would have to come from somewhere else if the PR lands.

For comparison: on the same M2 and a similar stack, hiding the symbols at link time instead (target_link_options(libSceSsl PRIVATE LINKER:-unexported_symbol,_mbedtls_*), my closed #10877) let both transport tests start and pass. So link-time unexporting may avoid the dyld failure, if it fits what you intended.

Broad run: ctest -R "^(guest_|libc_|libkernel_)" = 678 tests. On the base 133 do not pass, 344 of them "Not Run" because their executables do not build here (AGC driver SpirvOptimizer.cpp needs pthread_getattr_np; guest_tls_get_addr and guest_memory test objects fail to compile). Those are Not Run both ways. With the PR: 133 do not pass; the only difference from base is those two tests going from Failed to Aborted.

Not checked: whether guest titles import any of the hidden symbols; nm export list of the built prx; Linux.

AI-assisted: yes (Claude Code)

@maximilianfv

Copy link
Copy Markdown
Contributor Author

@krishalivaibhav thanks for running it on your stack. I reran it on a tree built from main alone, without #10552: mugurc's macOS series (27e968974) merged with main at 2742004ed2, x86_64 under Rosetta 2 on an M1 Pro, 3 runs per side, every target rebuilt and relinked:

without this PR with this PR
guest_ssl_transport_kernel, guest_ssl_transport_net fail 3/3 pass 3/3
the other 14 guest_ssl*, guest_libressl* and guest_http* tests pass pass

So the dyld: Symbol not found: _mbedtls_ctr_drbg_free ... expected in libSceSsl.prx failure looks specific to your stack. On main, guest_ssl_transport_tests links mbedtls, mbedx509 and mbedcrypto itself (libSceSsl/CMakeLists.txt:82), and hidden visibility only removes the symbols from the .prx exports, so the test should bind them from its own archives. That error means the test executable was bound to libSceSsl.prx at link time. It then ran against a .prx that no longer exports the symbol.

Could you check, on your stack, where that symbol binds in the test executable (dyld_info -bind tests/guest_ssl_transport_tests | grep mbedtls_ctr_drbg_free) and whether #10552 changes how the test or the .prx is linked (link order, flat namespace, an export list, or a copied .prx)? If #10552 needs the .prx to keep exporting mbedtls, whichever of the two lands second should adapt, and I'd rather settle that here than have main break.

On -unexported_symbol,_mbedtls_*: it would leave the 155 psa_* symbols exported, and those collide the same way, and it doesn't cover libSceLibreSsl's 27 mbedtls_* exports.

AI-assisted: yes (Claude Code ran the A/B and wrote this reply).

@krishalivaibhav

Copy link
Copy Markdown
Contributor

@maximilianfv you're right, and my earlier report was wrong: the dyld failure came from a stale link on my side, not from this PR.

That run rebuilt libSceSsl.prx but not guest_ssl_transport_tests, so the old executable still bound _mbedtls_ctr_drbg_free to @rpath/libSceSsl.prx. #10552 does not change how either one is linked.

I reran it with both targets rebuilt from this PR, on the same M2 and stack:

  • libSceSsl.prx exports no mbedtls_* symbols (nm -gU);
  • guest_ssl_transport_tests now carries _mbedtls_ctr_drbg_free itself (nm -m: non-external, from its own archives);
  • guest_ssl_transport_kernel and guest_ssl_transport_net pass 3/3. Without this PR they fail 3/3 on this stack.

So this PR fixes both tests on M2 as well. Thanks also for the psa_* and libSceLibreSsl points; they're why my closed #10877 wasn't enough.

AI-assisted: yes (Claude Code)

@adriadam10 adriadam10 self-assigned this Oct 26, 2026
@adriadam10
adriadam10 merged commit 0b921c6 into boykopovar:main Oct 26, 2026
13 checks passed
@adriadam10

Copy link
Copy Markdown
Collaborator

Checked the change against the target list at libSceSsl/CMakeLists.txt:49 (same four targets), the Linux/MinGW --exclude-libs,ALL path (untouched), and the macOS export/transport-test reports from @imellag, @maximilianfv and @krishalivaibhav and the MinGW run from @ycbuzi; merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants