[rooklz]

What

Implements the 8 libSceCesCs exports that #8993 leaves throwing on the encode side: sceCesUcs2ToMbc, sceCesUtf8ToMbc, sceCesUtf16ToMbc, sceCesUtf16beToMbc, sceCesUtf16leToMbc, sceCesUtf32ToMbc, sceCesUtf32beToMbc and sceCesUtf32leToMbc. Each converts one character into the profile it is given. simonbronk agreed on #8993 that I take this piece, so this branch carries the whole stack and only the top commit is new.

• Signatures follow the Orbital type database at b321fcd. They have the same shape as sceCesUtf8ToSJis and sceCesUcs2ToGb: UCS-2 and native UTF-32 are passed by value, UTF-32be/le by pointer, and UTF-8/UTF-16 as a buffer with its size and an optional consumed-length pointer.
• No new conversion code. Each export calls scalarToProfiled, bufferToProfiled or utf32ToProfiled, the helpers behind the per-encoding *ToSJis, *ToEucJp, *ToGb and *ToEucCn forms, with isMbcs as the profile check. That is the same check sceCesMbcTo* uses, so both directions accept the same profiles: Cp1250, Cp1251, Cp1252, CP932, CP51932, GBK, EUC-CN, EUC-KR, UHC and Big5. Each profile encodes through the existing encodeMbcs dispatch and its tables.
• The checks run in the same order as the decode side and the per-encoding siblings. An unknown profile gives INVALID_PARAMETER, then a null source INVALID_SRC_BUFFER, a zero source size SRC_BUFFER_END, a null output INVALID_DST_BUFFER, then ill-formed input INVALID_ENCODE (or SRC_BUFFER_END for a cut sequence), a character the profile does not map UNASSIGNED_CODE, and a short output DST_BUFFER_END. Input UTF goes through the library's existing decodeUtf8/decodeUtf16 and the scalar check, which reject overlong UTF-8, surrogates and anything above U+10FFFF as Unicode 3.9 requires. Nothing is written on failure.
• Still throwing: sceCesMbcsUcsSetMbcsReplacementCharCode and sceCesMbcsUcsSetMbcsReplacementCharUCode. The stack does not model a replacement character and its behaviour is unknown, so an unmapped character answers UNASSIGNED_CODE and is not replaced, as in the per-encoding forms.
• Debt: I extended feat(libSceCesCs): implement generic MBCS character, Shift-JIS code and JIS X 0213 helpers #8993's "Generic MBCS character" entry in docs/dev/debt/libSceCesCs.md to cover the encode direction, moved its unverified-errors sentence to the front, and changed the pending-exports count from 70 to 62. Nothing here is a console measurement.

Overlap: #9107 (EUC-KR, UHC and Big5 scalar conversions) edits nearby lines in Export.cpp, PendingExports.cpp and the debt file but adds no *ToMbc export, so whichever lands second will need a small rebase. #8923 doesn't touch these exports either.

Measured on a PS5

I ran homebrew payloads on a PS5 that load the console's libSceCesCs.sprx and call sceCesUtf32ToMbc on every value from U+0000 to U+110001 with each of the 15 MBCS profiles, and sceCesUcs2To* and sceCesUtf32To* of seven encodings on surrogates, C1 controls and a one-byte target. The single-character forms here now answer as the console does:

• a surrogate gives 0x805C0015, not 0x805C0014; a value past U+10FFFF stays 0x805C0014;
• DST_BUFFER_END stores the length the character needs (2 for U+3042 into a one-byte target), and any other refusal stores a zero length;
• the same holds for each encoding's own Ucs2To* and Utf32To*, which share the helper, so their tests changed with it.

The table differences the sweep found (CP932 forms, C1 and others) are other PRs' topics: #10186 measured the EUC-JP and Shift-JIS encoders, and the Shift-JIS U+0080-U+009F answer 0x805C0024 is in it, so it is not repeated here.

Tested

• Windows 11, WinLibs MinGW-w64 GCC 15.2.0 (UCRT), Release, by @ycbuzi on 12e3082520 merged onto main 25c1aa1e79: the 14 guest_ces* tests pass on two runs; with main's Export.cpp, PendingExports.cpp and Api.hpp the tests no longer build.

Ubuntu 24.04 x86-64 in a Docker container under Rosetta on an Apple M5 Mac, GCC 13.3, Release:

cmake -S . -B build -G Ninja -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=ON -DCMAKE_C_COMPILER=gcc -DCMAKE_CXX_COMPILER=g++ -DCMAKE_C_COMPILER_LAUNCHER=ccache -DCMAKE_CXX_COMPILER_LAUNCHER=ccache
ninja -C build -j4 libSceCesCs libSceCesCs-module guest_ces_{,dbcs_,unicode_,japanese_,wide_,chinese_,mbc_}{,module_}tests
cd build && ctest -R guest_ces --output-on-failure

All 14 guest_ces* tests pass. guest_ces_mbc and guest_ces_mbc_module take 0.27 s each. I extended #8993's GuestCesMbc.cpp, which is already registered in CMake, so that it checks:

• every BMP scalar through all 8 forms for each of the 10 profiles. All forms agree, every encoded character decodes back to the same code point through sceCesMbcToUtf32, the rest answer UNASSIGNED_CODE, and U+10000, U+1F600 and U+10FFFF answer UNASSIGNED_CODE in the 7 forms that can carry them;
• a round trip MBC -> UTF-x -> MBC of every one-, two- and three-byte sequence that decodes (90,964 sequences), chaining each sceCesMbcTo* output into the matching *ToMbc. For every profile, the number of cells that come back byte for byte equals the number of mapped code points: 256 for each single-byte profile, 7,517 for CP932, 13,303 for CP51932, 7,606 for EUC-CN, 21,920 for GBK, 8,354 for EUC-KR, 17,176 for UHC and 13,621 for Big5. The UHC, EUC-KR and GBK counts match Python's cp949 and gbk codecs (GBK plus the 0x80 euro);
• spot values for each profile kind (for example U+3042 to 82 A0 in CP932, U+4E02 to 8F B0 A1 in CP51932 and 81 40 in GBK but unassigned in EUC-CN, U+AC02 to 81 41 in UHC but unassigned in EUC-KR, U+4E00 to A4 40 in Big5);
• invalid input in every profile: overlong UTF-8 (C0 80, E0 80 80, F0 80 80 80 and others), encoded surrogates, values above U+10FFFF, bad leads and continuations, cut sequences, lone and reversed UTF-16 surrogates in all three byte orders, and UCS-2/UTF-32 surrogates, 0x110000 and 0xFFFFFFFF;
• the error order above for every form, byte order (the same UTF-32 bytes encode through be and are rejected through le), null length pointers, and that the output buffer is untouched after every failure. The length outputs follow the console (next section): untouched after an argument error and in the UTF-8 and UTF-16 buffer forms, zero after a refused character and the needed length after DST_BUFFER_END in the single-character forms.

Negative control: with the base Export.cpp restored, the test no longer links because the stubs are gone (undefined reference to sceCesUtf32ToMbc and the other 7). With the base PendingExports.cpp restored as well, guest_ces_mbc and guest_ces_mbc_module abort with sceCesUcs2ToMbc not implemented and the other 12 tests still pass. After restoring both files all 14 pass again.

python3 tools/progress.py --module libSceCesCs: 217/287 before, 225/287 after. python3 tools/check_conventions.py --base 15e71dbe11 passes. Not tested on a console, on Windows or with a title. I didn't run tools/pr_overlap.py; I checked the open libSceCesCs PRs by hand instead.

Hosted runs on my fork (GitHub runners), this branch 524c49e8c and its base c020ce893:

• Ubuntu 24.04 x86-64, GCC 13.3, full build with the CI configure flags, tests on lavapipe: 1811/1815 with the change, 1811/1815 on the base. The three net_dontwait_blocking_* tests fail on both.
• Hosted Linux runner, head 8ccc8df491 (verify run on my fork; Ubuntu 24.04, GCC, Release, lavapipe): every guest_ces* test passes; its one failure, agc_driver_sgpr_pattern_fill_keys, is main's on that runner. macOS (Apple clang 21, x86-64): all seven guest_ces* sources pass against this Export.cpp.
• Windows Server 2025, WinLibs GCC 15.2.0, full build: it builds, and all 14 guest_ces* tests pass (14/14; 14/14 on the base).

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: feat(libSceCesCs): implement generic MBCS character, Shift-JIS code and JIS X 0213 helpers #8993, the top of simonbronk's stack feat(libSceCesCs): implement Shift-JIS and EUC-JP scalar conversions #8967 -> feat(libSceCesCs): add UTF-16 and UTF-32 forms of the SBC and MBCS string conversions #8972 -> feat(libSceCesCs): convert GBK and EUC-CN through the CP936 table #8976 -> feat(libSceCesCs): implement generic MBCS character, Shift-JIS code and JIS X 0213 helpers #8993
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[github-actions]

Progress report (941ace5 → 12e3082)

📈 System libraries: 48.92% (+0.06%, +8 implemented functions)

✅ 8 implemented

Library	Name
libSceCesCs	sceCesUcs2ToMbc
libSceCesCs	sceCesUtf16ToMbc
libSceCesCs	sceCesUtf16beToMbc
libSceCesCs	sceCesUtf16leToMbc
libSceCesCs	sceCesUtf32ToMbc
libSceCesCs	sceCesUtf32beToMbc
libSceCesCs	sceCesUtf32leToMbc
libSceCesCs	sceCesUtf8ToMbc

Changed libraries (implemented exports)

Library	Before	After	Change
libSceCesCs	217/287 (75.61%)	225/287 (78.40%)	+2.79 pp

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 10056	docs/dev/debt/libSceCesCs.md L22-25
PR 10214	docs/dev/debt/libSceCesCs.md L24

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 9912	core/libs/prx/libSceCesCs/Export.cpp core/libs/prx/libSceCesCs/PendingExports.cpp docs/dev/debt/libSceCesCs.md

[ycbuzi]

Windows run of 524c49e merged onto main 0c6a5bd (24dd6eb041, clean), the platform the description lists as not tested. Windows 11 Home 10.0.26200, MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release, BUILD_TESTING=ON.

Built libSceCesCs and the 14 guest_ces* test executables with no errors or warnings. ctest --repeat until-fail:2 on the 14 tests these executables register on Windows: all pass on both repeats. The 14 are the base guest_ces and the guest_ces_* tests and their _module variants (dbcs, unicode, japanese, wide, chinese, mbc). Not run: a title.

[MarcZX10]

Review

I reviewed the new eight forwarding exports and the added MBCS round-trip/error cases. I did not find a correctness issue in the new wrappers. The reported test coverage is strong: exhaustive BMP checks across profiles, round trips, invalid UTF cases, output preservation on failure, Windows coverage, and hosted test runs.

Branch freshness

This head is 635 commits behind current main (merge base 27bcd6a29e). Its five commits are the stack #8967 → #8972 → #8976 → #8993 → this PR; the four prerequisite PRs are still open. Please keep the stack based on current main as required by CONTRIBUTING and rerun the focused guest_ces* tests after updating it. The test results in the description are from the earlier base c020ce893.

Overlap coordination

The overlap report lists open #9107 in Export.cpp, PendingExports.cpp, and the debt file. You already identify #9107 as nearby, distinct work, but the repo rule asks contributors to agree with overlapping authors on landing order. Please record that agreement and add a Depends on entry if this PR is actually stacked on it. The report also lists #8923 for the same debt-file line; please note that documentation-only overlap in the PR description. I found no evidence that these PRs duplicate the eight exports here.

Checks

• git diff --check against current main: passed.
• tools/check_conventions.py --base upstream/main --head refs/review/9175: passed.
• Current CI: eligibility, intake, conventions, labels, and report passed; Linux, Windows, register-context, relinker, ASan relinker, and write-watch jobs were skipped.
• I did not rebuild locally in this session; the extensive test results above are author-reported on the older base.

The top commit’s implementation and test design look sound. I’ll revisit after the stack is current and the overlap order is recorded.

[rooklz]

Rebased onto current main (8b3e39832a): head a0dd95f7fd, same diff.

[rooklz]

Rebased onto current main (15e71dbe11): head 880dce7467. https://github.com/rooklz/AnyPS5/actions/runs/39982407764 (Ubuntu 24.04, GCC, Release, lavapipe, fails the job on any ctest failure): the conflict was the debt file, where the merged CesCs PRs changed the pending count; it now reads 121 to 113 on current main, and the description's numbers are updated (progress.py --module libSceCesCs: 166/287 before, 174/287 after). The guest_ces* tests pass; the one failure, agc_driver_sgpr_pattern_fill_keys, is main's on that runner.

[rooklz]

Rebased again onto current main (da6ace485f, after the EUC-KR/UHC/Big5 PR landed): head be0749d6b3. The conflicts were the pending-exports file, where that PR removed its stubs around the eight this PR removes (both removals kept), and the debt file's count, now 76 to 68; the description's numbers follow (progress.py --module libSceCesCs: 211/287 before, 219/287 after). Hosted run https://github.com/rooklz/AnyPS5/actions/runs/40004453946 (Ubuntu 24.04, GCC, Release, lavapipe, fails the job on any ctest failure): the guest_ces* tests pass; the one failure, agc_driver_queued_index_read_order, is main's intermittent one on that runner (base side of https://github.com/rooklz/AnyPS5/actions/runs/39845397298).

[rooklz]

Rebased onto current main (ca6fbf762c, after #9204 landed): head 1f482659df, same diff; the conflict was the debt file's pending count, now 72 to 64, and the description's numbers follow (progress.py --module libSceCesCs: 215/287 before, 223/287 after). Hosted run https://github.com/rooklz/AnyPS5/actions/runs/40018006149 (Ubuntu 24.04, GCC, Release, lavapipe, fails the job on any ctest failure): the guest_ces* tests pass; the one failure, agc_driver_sgpr_pattern_fill_keys, is main's on that runner.

[rooklz]

Rebased onto current main (e7f46f5c89, after #9210 landed): head d494b14391, same diff; the conflict was the debt file's pending count, now 70 to 62, and the description's numbers follow (progress.py --module libSceCesCs: 217/287 before, 225/287 after). Hosted run https://github.com/rooklz/AnyPS5/actions/runs/40029239079 (Ubuntu 24.04, GCC, Release, lavapipe, fails the job on any ctest failure): the guest_ces* tests pass; the two failures, agc_driver_queued_index_read_order and agc_driver_sgpr_pattern_fill_keys, are main's on that runner.

[rooklz]

Pushed 8ccc8df491: the single-character encoders now answer as a PS5 does, which I measured with homebrew payloads calling the console's libSceCesCs.sprx (description, "Measured on a PS5"). A surrogate gives 0x805C0015, DST_BUFFER_END stores the needed length, and any other refusal stores a zero length, in the *ToMbc forms and in each encoding's Ucs2To* and Utf32To*. Hosted Linux run: every guest_ces* test passes.

@orezraey heads-up since #10186 carries this branch: I left the Shift-JIS U+0080-U+009F 0x805C0024 answer to your PR; my sweep agrees with it. The lines that changed here are scalarToProfiled and scalarError.

[ycbuzi]

Windows run of 8ccc8df491 merged onto main c8d34aa8f4 (76640e2a22, clean), following my run of 524c49e8cf. Windows 11 Home 10.0.26200, WinLibs MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release.

• Built libSceCesCs and the 14 guest_ces* test executables with no errors or warnings.
• ctest --repeat until-fail:2: all 14 guest_ces* tests pass both runs.
• Control: with core/libs/prx/libSceCesCs/Export.cpp and PendingExports.cpp from main and this PR's tests, six abort (0xc0000409): guest_ces_dbcs, guest_ces_japanese and guest_ces_chinese, each with its _module variant. The other eight still pass.

Not run: a title. AI-assisted: yes (Claude Code)

[rooklz]

Rebased onto current main (head 12e3082520) after #10184, #10185 and #10186 landed. Those brought in the shared scalar helpers, so this diff is now mostly the eight *ToMbc exports and their tests. The Api.hpp declarations take these exports' signatures. A few test expectations changed to match measured console behaviour:

• sceCesUtf8ToMbc now decodes with main's decodeScalarUtf8. Measured on a PS5 with CP932: overlong forms and surrogates give 0x805C0015; U+110000 and above, and 5- and 6-byte forms, give 0x805C0020 after consuming the whole sequence; bad leads give 0x805C0014 and consume 0.
• CP51932 has no JIS X 0212 encoding (fix(libSceCesCs): encode EUC-JP and Shift-JIS as measured on a PS5 #10186), so U+4E02 is unassigned there.
• Shift-JIS answers U+0080-U+009F with 0x805C0024.
• CP932's Microsoft forms are one-way (U+00A2 encodes to 81 91, which decodes to U+FFE0), and U+FEFF writes nothing.
• The CP1252/1251/1250 tables have 5, 1 and 5 unassigned bytes.

All 14 guest_ces* tests pass in a Linux x86-64 container (Ubuntu 24.04, GCC 13.3, Release).

[ycbuzi]

Windows run of 12e3082520 merged onto main 25c1aa1e79 (fdc4ba636b, clean). Windows 11 Home 10.0.26200, WinLibs MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release.

• Built libSceCesCs and the 14 guest_ces* test executables with no errors. The one warning is main's existing _sigintr_nid_postfix "initialized and declared 'extern'" in Signals.cpp.
• ctest --repeat until-fail:2: all 14 guest_ces* tests pass both runs, including guest_ces_mbc and guest_ces_mbc_module with the new expectations.
• Control: with Export.cpp, PendingExports.cpp and include/Api.hpp of libSceCesCs from main, the tests no longer build, since main declares the eight *ToMbc exports without parameters (GuestCesMbc.cpp:62: too many arguments to sceCesUcs2ToMbc()).

Not run: a title. AI-assisted: yes (Claude Code)

[rooklz]

@ycbuzi thanks for the Windows run on the rebased head, and for the control run. I've added it to the Tested section.

Dismissing a stale review: the requested changes were addressed after it (new head) and the merge queue blocks on it. Please request changes again if anything is still open.

[adriadam10]

Checked the eight *ToMbc wrappers against the existing helpers, the removed stubs and the pending count (62), and the shared scalarToProfiled change against the updated Chinese/Dbcs/Japanese expectations and the PS5 measurements. GuestCesMbc is registered, pins values across ten profiles and links only with the new exports. Windows run by @ycbuzi covers the 14 guest_ces* tests. The stack is now a single commit on main, so the earlier freshness request is met; merged.