Skip to content

feat(ssl): run TLS 1.2 client connections over Mbed TLS - #6828

Merged
oneandonlydean merged 8 commits into
boykopovar:mainfrom
loyalburrito:claude/brave-ride-lz92qi
Oct 18, 2026
Merged

oneandonlydean merged 8 commits into
boykopovar:mainfrom
loyalburrito:claude/brave-ride-lz92qi

Conversation

@loyalburrito

@loyalburrito loyalburrito commented Oct 16, 2026 •

Copy link
Copy Markdown
Contributor

What

sceSslConnect, sceSslRead and sceSslWrite no longer throw NotImplemented: they run a TLS 1.2 client over the Mbed TLS 3.6.7 submodule.

  • Socket: the guest number resolves to the host socket through a new libkernel export, KernelLookupSocketNative_nid_no_patch. It checks libkernel's socket table, then the one libSceNet registers, so both socket() and sceNetSocket numbers work.
  • Memory: session state comes from the context's existing memory pool and is returned on sceSslDeleteConnection. Each pool now has its own lock, so a handshake or read does not hold the library lock. A connection keeps its pool alive after sceSslTerm.
  • Verification: the server is verified against the CAs loaded with sceSslLoadCert (chain, validity dates, host name, also sent as SNI). A loaded client certificate is sent when the server asks for one.
  • Config: MbedConfig.hpp turns off MBEDTLS_SSL_PROTO_TLS1_3, and libSceSsl now links the mbedtls SSL library. SharpProspero's SceSslVersion stops at TLS 1.2, and Mbed TLS's TLS 1.3 needs process-wide PSA state that can't come from a context's pool.
  • Debt: docs/dev/debt/libSceSsl.md records everything that throws or isn't verified on a console.

Questions for maintainers

Anything the sources don't settle throws for now:

  1. No CA loaded. sceSslConnect throws, because no root bundle is shipped. Should a host bundle be trusted? feat(libkernel): serve the sandbox CA bundle from host TLS roots #5280 serves the host's CA bundle to the sandbox, which could fill this.
  2. Verify callback. A non-null callback makes sceSslConnect throw. SharpProspero gives the arguments (an id, failure bits, the chain, its length, the user argument; a negative return fails the handshake), but not whether the id is the context or the connection, what the bits mean, what type the chain entries are, or when the callback runs.
  3. Error codes. Are the console codes known for handshake, verification and socket failures, would-block, timeouts, or read/write before connect? All of these throw for now. Only 0x8095F006, 0x8095F004, 0x8095F007 and sharpemu's 0x8095F008 are known.
  4. sceSslRead with flags 0. It returns what one record read delivers, the same as MsgFlagPartialRead (1). SharpProspero leaves open whether flags 0 waits to fill the buffer. Peek (3) throws.

Overlap

#5911 (verify options, min version, ALPN), #5953 and #5969 (certificate readers; #5969 also changes sceSslClose's signature) and #6511 (certificate reader arguments) edit libSceSsl/Export.cpp and its debt entry, but none implements the handshake or I/O. Whichever merges second will need a rebase; I'm happy to do it.

#6044 (test reorganization) conflicts in core/libs/prx/libSceSsl/CMakeLists.txt: it moves the SSL tests to core/libs/tests/networking/ and registers them with add_library_test. Whichever lands second moves GuestSslTransport.cpp (and SslCertificateFixture.hpp, if #6044 moves it) there and registers guest_ssl_transport_kernel and guest_ssl_transport_net the same way. #6222 also edits core/libs/prx/libkernel/Socket/src/Udp.cpp but merges cleanly with this PR.

Tested

Tested locally by the author on a machine with an NVIDIA GeForce RTX 3080 (12 GB).

Linux (Ubuntu 24.04, GCC 13), full Release build with BUILD_TESTING=ON, merged with current main:

  • New guest_ssl_transport_kernel and guest_ssl_transport_net run an in-process Mbed TLS server on loopback that requires a client certificate, about 0.3 s each:
    • handshake, a write and read round trip, and an end-of-stream read that returns 0
    • four connections on one context in parallel threads
    • wrong host name, no CA loaded, a verify callback, a non-socket descriptor, a double connect, unsupported flags
    • pool memory returning to its baseline, and the socket staying open after delete
  • Mutation checks:
    • With verification off, the wrong-host check fails.
    • With the pool lock removed, ThreadSanitizer reports data races in PoolCalloc.
  • Sanitizers:
    • ASan+UBSan, leak detection on: clean on both transport tests, guest_ssl and guest_ssl_load.
    • ThreadSanitizer: clean on 3 runs of each transport test.
  • Windows (WinLibs MinGW-w64 GCC 15.2.0 posix-seh ucrt r7, the supported toolchain): the project's own windows job of build.yml, run on a GitHub-hosted windows-latest runner with this PR merged into main (run 38911524761, job windows). Configure, test registration, cmake --build build, --target libs, the full ctest --parallel 4 and check_skips.py --job windows all pass. guest_ssl, guest_ssl_load, guest_ssl_transport_kernel, guest_ssl_transport_net and guest_memory_pool_stats ran and passed, the transport tests over Winsock loopback sockets; the 494 skips are all expected agc* and display/Vulkan tests (the runner has no Vulkan driver), and no test went over 30 s. The Linux job of that run was not run, as it needs upstream's self-hosted runner; Linux is covered above.
  • macOS: not applicable; libSceSsl is not part of the relinker-only build.
  • Full ctest: 1,281 of 1,289 pass. The 8 failures:
    • agc_driver_graphics, guest_datagram_* ×4, guest_named_shm_concurrency and guest_tcp fail the same way on unmodified main in that container (no IPv6, running as root, "VOP2 opcode is not implemented").
    • guest_kernel_fcntl aborted once under full-suite load. It then passed 30 runs alone and 3 loaded runs of all guest tests.
  • python3 tools/check_conventions.py --base origin/main passes.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: none
  • GPUs tested: n/a (no AGC, shader recompiler or VideoOut changes); the author's local run was on an NVIDIA GeForce RTX 3080 12 GB
  • AI-assisted: yes

claude and others added 4 commits October 16, 2026 11:37
sceSslConnect performs the client handshake on the host socket that the
guest number maps to in the libkernel or libSceNet socket table, and
sceSslRead and sceSslWrite carry application data. Session state comes
from the context's memory pool, and the server is verified against the
CAs loaded with sceSslLoadCert; a loaded client certificate is sent when
the server asks for one.

libkernel exports KernelLookupSocketNative_nid_no_patch, which checks
its own socket table before the one libSceNet registers. Pools take a
lock so connections can allocate outside the library lock.

TLS 1.3 is compiled out. A missing CA, a verify callback, would-block
sockets and every failure whose console code is unknown throw, as
recorded in the libSceSsl debt entry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoU3ZF76A6uR8sCA7NMLch
feat(ssl): run TLS 1.2 client connections over Mbed TLS
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoU3ZF76A6uR8sCA7NMLch
Each client thread handshakes, writes, reads the reply and the end of
the stream, so session setup and record allocations from several
threads contend for the same context pool. The pool memory must return
to its loaded size afterwards.

Initialize the pool's mutex explicitly, which -Wextra asks for.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoU3ZF76A6uR8sCA7NMLch
@MarcZX10

Copy link
Copy Markdown
Contributor

The current conventions check is failing because the PR description labels What, Tested, and Checklist are plain text; the repository template requires Markdown headings (for example ### What). Please reformat those section headers and rerun conventions. I also noticed the new transport path is a substantial addition that is not built with the WinLibs toolchain, so please keep the platform-specific build status explicit as you have in the body.

@MarcZX10

Copy link
Copy Markdown
Contributor

Thanks for fixing the template; the latest conventions run passes. I reviewed the socket ownership, per-session locking, pool lifetime and error cleanup, and do not have a concrete correctness finding to request. One validation gap remains before I can recommend merge: this adds direct Winsock send/recv, Windows test setup, and MinGW-specific Mbed TLS compile options, but neither the PR report nor current checks include a build with the supported WinLibs toolchain (the platform jobs are absent). Could you run the focused transport target and relevant SSL tests with that toolchain, or explain which known environment limitation prevents it? The Ubuntu MinGW syntax check does not validate the project’s Windows build/link setup.

@github-actions

github-actions Bot commented Oct 16, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 6511 core/libs/prx/libSceSsl/CMakeLists.txt L78, L80-81
docs/dev/debt/libSceSsl.md L7-8

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 5021 core/libs/prx/libkernel/Socket/src/Udp.cpp
PR 6039 docs/dev/debt/libSceSsl.md
PR 6044 core/libs/prx/libSceSsl/CMakeLists.txt
core/libs/tests/SslCertificateFixture.hpp
PR 6222 core/libs/prx/libkernel/Socket/src/Udp.cpp
PR 6962 core/libs/prx/libkernel/Socket/src/Udp.cpp
PR 6977 core/libs/prx/libkernel/Socket/src/Udp.cpp
PR 7075 core/libs/prx/libkernel/Socket/src/Udp.cpp
PR 7088 core/libs/prx/libkernel/Socket/src/Udp.cpp
PR 7129 core/libs/prx/libkernel/Socket/src/Udp.cpp
PR 7201 core/libs/prx/libkernel/Socket/src/Udp.cpp
PR 7332 core/libs/prx/libkernel/Socket/src/Udp.cpp

@MarcZX10

Copy link
Copy Markdown
Contributor

I attempted a project-level Windows cross-configuration on this Linux host with its installed x86_64-w64-mingw32-g++ (GCC 12-posix). CMake stops at the repository's MinGW exception-model check: this compiler selects SjLj exceptions with -fno-asynchronous-unwind-tables, while AnyPS5 requires DWARF unwinding. This toolchain therefore cannot provide the requested project build, link, or Windows runtime validation, and it is not a substitute for the supported WinLibs toolchain. The Windows validation gap remains open.

@loyalburrito

Copy link
Copy Markdown
Contributor Author

@MarcZX100 thanks for the review. The WinLibs build now exists, so the Windows gap should be closed.

I ran this repository's own windows job from build.yml (WinLibs MinGW-w64 GCC 15.2.0 posix-seh ucrt r7, windows-latest) from my fork, with this PR's head 9ce634a merged into main: run 38911524761, job windows. Every step passed:

configure, including the MinGW -fno-asynchronous-unwind-tables options on the Mbed TLS targets
check_test_registration.py
cmake --build build, which links libSceSsl.prx against Winsock and guest_ssl_transport_tests against ws2_32/bcrypt
cmake --build build --target libs
the full ctest --parallel 4
check_skips.py --job windows: "494 test(s) skipped, all expected" (only agc* and display/Vulkan tests; the runner has no Vulkan driver)
the slow-test report: nothing over 30 s
So guest_ssl, guest_ssl_load, guest_ssl_transport_kernel, guest_ssl_transport_net and guest_memory_pool_stats ran and passed on Windows. The two transport tests ran the handshake, round trip and four parallel connections over Winsock loopback sockets, using the direct send/recv path and the PATH setup for the test runs. That run's Linux job is skipped, because it needs the self-hosted runner. Linux is covered by my local full build with ASan/UBSan and TSan.

On your cross-configuration attempt: that matches what I'd expect. The repository's exception-model check rejects SjLj toolchains such as GCC 12-posix, which is why I used the pinned WinLibs zip through the project's own job instead.

I've updated the description's Windows line with these results. Since then, upstream main has moved 22 commits; none touches these files, the PR still merges cleanly, and the SSL tests pass on the merge. Upstream's own Windows and Linux jobs still need a maintainer to approve the workflows.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 16, 2026
@blawesom

Copy link
Copy Markdown
Contributor

Rebase handoff — rebased onto current main (the two authored commits, authorship unchanged; the head's empty merge flattened away). The conflicts reconciled main's newer SSL context state with this PR's Mbed TLS session design:

  • SslConnection keeps main's verifyOptions/minimumVersion/alpn fields alongside your shared_ptr<Session>
  • all three Mbed TLS includes kept (oid.h for main's cert-name readers, error.h/net_sockets.h for your Session)
  • the debt entries follow main's restructuring with your connection-entry rewrite and sceSslLoadCert tail applied; two of main's clauses were minimally reworded where your transport made them stale ("until sceSslConnect has a transport" → "nothing reads them"; "as no handshake completes" → "as the protocol negotiated by the handshake is not read") — worth your eye
  • one semantic gap noted, unchanged from either side: verifyOptions/minimumVersion/alpn are recorded but not yet consumed by Session::Configure — wiring them into the Mbed TLS config would be new work beyond both diffs

Verified on Linux (CachyOS, kernel 7.2.9, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, AMD RX 9070 XT / RADV):

To pick it up:

git fetch https://github.com/blawesom/AnyPS5 rebase/6828
git checkout <your branch>
git merge --ff-only FETCH_HEAD

Head is 925c37aa0.

Brings in main's connection verify options, minimum version, ALPN and
certificate readers. The conflicts in libSceSsl/Export.cpp and the
libSceSsl debt entry take the resolution from blawesom's rebase of this
pull request (blawesom/AnyPS5 rebase/6828 at 925c37a): SslConnection
keeps main's verifyOptions, minimumVersion and alpn next to the session,
and the debt entries follow main's structure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoU3ZF76A6uR8sCA7NMLch
With MBEDTLS_SELF_TEST, ecp.c counts point additions, doublings and
field multiplications in unsynchronized globals. Handshakes on separate
connections run in parallel, each under its own session lock, so the
counters race; ThreadSanitizer reports it on the parallel transport
test. Nothing calls the self-test functions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RoU3ZF76A6uR8sCA7NMLch
@frankea

frankea commented Oct 18, 2026

Copy link
Copy Markdown
Contributor

@loyalburrito the one gap blawesom's handoff left, the recorded verify options, minimum version and ALPN that Session::Configure never reads, I've done against your branch as a pull request to your fork: loyalburrito#3. It makes sceSslConnect return the console's SSL_ERROR_* verification codes under VERIFY_OPTIONAL instead of requiring a loaded CA and throwing, offers the ALPN list and reports the selection, floors the version at TLS 1.2, and maps would-block, reset, timeout and fatal-alert to their codes so a non-blocking socket can resume the handshake. The verify callback, sceSslClose and the initial-options default are left as they are. Built with WinLibs on Windows 11, the two transport tests rewritten and passing with guest_ssl, guest_ssl_load and the pool-stats test; details in the fork PR. Take it, cherry-pick it, or tell me what you'd rather see; it's your PR. My #7319 stays a draft on top of this.

AI-assisted: yes (Claude Code).

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 18, 2026
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 18, 2026
@oneandonlydean
oneandonlydean merged commit d704b80 into boykopovar:main Oct 18, 2026
1 check passed
@MarcZX10

Copy link
Copy Markdown
Contributor

The offered follow-up on loyalburrito#3 directly addresses the gap from the rebase review: the connection’s verify options, ALPN list, and minimum-version field are now consumed, with focused Windows transport tests reported passing. I have reviewed the changed handshake and result-code paths and am checking this independently before carrying it onto #6828; the report still identifies non-blocking socket behavior as untested end to end. I will keep the handoff separate until that review is complete.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants