[orezraey]

What

sceCesSbcToUtf8 and sceCesUtf8ToSbc used assumed tables and errors. I measured them on a retail PS5 (firmware 13.60) with homebrew probes, from a payload process loading libSceCesCs.sprx. The sources and console logs are in this gist (pces3 to pces6).

Tables. All 128 high bytes of Cp1252, Cp1251 and Cp1250 match the console, except the bytes the console leaves unassigned (0x805C0020), which we mapped to C1 control characters:

Profile	Unassigned bytes
Cp1252	0x81, 0x8D, 0x8F, 0x90, 0x9D
Cp1251	0x98
Cp1250	0x81, 0x83, 0x88, 0x90, 0x98

They are unassigned in both directions now (U+0081 and the others no longer map back), and the MBCS string conversions that share these tables follow. The console's sceCesMbcsStrToUtf8Str also returned 0x805C0020 for Cp1252 byte 0x81.

sceCesSbcToUtf8.

• The length is cleared to 0 first.
• A null profile returns 0x805C0004 (we returned 0x805C0001).
• An unassigned byte returns 0x805C0020 before the output is checked.
• The length is then set to the UTF-8 size, even when the output is null (0x805C0030) or too small (0x805C0031).

sceCesUtf8ToSbc.

• The length and output byte are cleared first.
• A null source or a zero size returns 0x805C0010 (we returned 0x805C0011 for a zero size).
• Then it decodes the sequence:
  • leads 0xF8–0xFB and 0xFC–0xFD take 5 and 6 bytes;
  • a continuation lead, 0xFE and 0xFF return 0x805C0014 with length 0;
  • a short sequence returns 0x805C0011 with the full length, unless a byte already present is not a continuation byte;
  • a bad continuation byte returns 0x805C0014 with the length set to that byte's index, or the index plus one when the sequence is also short;
  • overlong forms and surrogates return 0x805C0015 (we returned 0x805C0014).
• The length is reported as above for every case, before the profile is checked.
• A null profile returns 0x805C0004.
• Values with no byte in the table, scalars above U+10FFFF included, return 0x805C0020.
• A null output returns 0x805C0030 last.

The test pins 42 measured sequences with their results and lengths.

Not changed. The console reads any non-null profile pointer as a table: a zero-filled buffer and an EUC-JP profile both converted. Here a pointer that is not one of the three profiles still returns 0x805C0001. Which comes first between a null profile and a scalar above U+10FFFF was not measured. The UTF-8 decoder of the string conversions is unchanged, since only the scalar functions were measured. The debt entries record all of this.

Also found: on this firmware libSceCesCs.sprx does not export the Unicode scalar functions in Unicode.cpp (sceCesUtf8ToUtf32, sceCesUtf16ToUtf8, sceCesUcs2ToUtf8 and the others; pces2), so they could not be measured.

On top of #8923 and #8972. The 27 new profiles keep their tables. The UTF-16 and UTF-32 forms from #8972 share the lookup, so they now answer UNASSIGNED_CODE for the unassigned bytes too, and guest_ces_wide expects that for Cp1251 0x98.

Tested

• Head after merging main (f26434afa3, with refactor(libs): add full export headers for libs #10706): f7f39ad89c. The test now takes its declarations from libSceCesCs/include/Api.hpp (sceCesRefersUcsProfileCp1250 returns const void* there, so the test casts it). The 14 guest_ces* tests pass three runs in a row on Windows (MinGW) and on Linux (WSL Ubuntu, GCC); public_headers audit and check_conventions pass.
  Windows 11, MinGW (winlibs).
• guest_ces checks:
  • every byte of the three profiles: a round trip for assigned bytes, 0x805C0020 with length 0 for the unassigned ones;
  • the 42 measured sceCesUtf8ToSbc sequences;
  • the null, size and output cases above.
• It fails on main and passes with the change. The ten guest_ces* tests pass three runs in a row.
• tools/check_conventions.py passes.
• tools/pr_overlap.py lists feat(libSceCesCs): add 27 single-byte code page profiles #8923, feat(libSceCesCs): add the Windows-1253, 1255 and 1257 profiles #9912 and feat(libSceCesCs): add the GB18030-2000 and GB18030-2005 profiles #9210, which add other code pages and rework sbcToUnicode. They do not change these two functions or these three tables, but whichever merges second needs a rebase. It also lists PRs that touch other lines of docs/dev/debt/libSceCesCs.md.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: feat(libSceCesCs): encode UCS-2, UTF-8, UTF-16 and UTF-32 characters into any MBCS profile #9175, feat(libSceCesCs): add the ISO 646 national variants and JIS X 0201 profiles #10430, feat(libSceCesCs): select the single-byte set of an MBCS profile and a replacement character for a context #10431 (merge order only: they edit the same lines; this PR is refreshed after them); refactor(libs): add full export headers for libs #10706, fix(libSceCesCs): report the length of a rejected MBCS character as the console does #10722, fix(libSceCesCs): map the GB 18030 profiles as the console does #10739, feat(libSceCesCs): report the JIS X 0213 level of a cell as the console does #10751 edit the same lines too (merge order only)
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[MarcZX10]

Reviewed the scalar Cp1252/Cp1251/Cp1250 changes at head b41fdbc. The table entries marked unassigned now fail in both conversion directions, and the UTF-8 decoder’s lengths/errors for truncated sequences, malformed continuations, overlong encodings, surrogates, and out-of-table values match the measurements described. The 42 measured-sequence tests and byte-table round trips are appropriately focused. This exact commit is included in the descendant #10186 C++20 syntax/conventions check; hosted eligibility, conventions, intake, labels and report pass, while product builds are path-skipped. The author reports the ten CES tests passing three consecutive runs. I could not run the executable locally because full CMake configuration is blocked by missing SDL2 development files.

[MarcZX10]

Correction to my prior approval: I had not evaluated the direct overlap with the newly refreshed #10056 head. git merge-tree of #10183 and #10056 reports a content conflict in core/libs/prx/libSceCesCs/Export.cpp, where both alter the single-byte decode/error behavior. Please coordinate the landing order with #10056's author, declare the dependency on the checklist if this PR lands second, and preserve both changes after rebasing. #10185 and #10186 already stack on this conversion work, so keep that dependency chain intact. The exact-head tests previously reviewed remain useful; hosted product checks are path-skipped. This is a keep-both sequencing request, not a request to close either PR.

[orezraey]

@MarcZX10 @atharvjairath agreed on #10056 that this stack lands first (#10183/#10184, then #10185, then #10186) and #10056 rebases over it, so this PR keeps Depends on: none; #10185 and #10186 keep their chain.

[MarcZX10]

Re-review of exact head 8e10ddbb5ce16446c68d71054b6a9ffec76049d6. The previous review asked for an agreed order with #10056 because both changed the single-byte conversion behavior. The authors have now explicitly agreed that #10183/#10184 land first, followed by #10185 and #10186, then #10056; #10056 records its dependency on #10186, and the CES stack PRs record their dependencies. The #10183 checklist correctly remains Depends on: none because it is first.

The implementation diff against current main is limited to Export.cpp, the CES tests, and the CES debt entry; the new commit only merges the same current main base. Exact-head conventions and git diff --check pass. I also ran GCC C++20 -Wall -Wextra -Werror -fsyntax-only on Export.cpp successfully. Hosted eligibility, conventions, labels and report pass; product test/build jobs are skipped by path policy. The author-reported CES test runs remain the available runtime evidence; I could not run the full CMake test locally. The prior overlap blocker is resolved. Approving this exact head.

[orezraey]

@oneandonlydean this was closed a second after #9107 merged, without a note, so I wanted to check whether that was intended. #9107 adds the Korean and Big5 profiles; this PR changes the single-byte conversion the PS5 measurement showed, which #9107 does not cover. The head still merges cleanly into current main (with #9107), MarcZX10 approved it, and #10185 and #10186 build on it, with #10056 agreed to land after the stack. If it was closed by mistake, could you reopen it? If there is a reason to drop it, let me know and I'll rework the stack.

[oneandonlydean]

@orezraey not intended, sorry, and thanks for asking. #9107's description mentions "the four PS5 measurement fixes #10183, …", and GitHub read "fixes #10183" as a closing keyword, so merging #9107 closed this one automatically. Reopened; it's separate work and stays in review as before.

[oneandonlydean]

@orezraey #9210 has landed and changes libSceCesCs/Export.cpp and docs/dev/debt/libSceCesCs.md, so this PR now conflicts in both. Please rebase onto main, keeping the GB18030 supplementary mappings and encodeUtf16 from #9210 alongside your changes, then rerun the guest_ces* tests on Linux and Windows.

[orezraey]

Merged main (with #9210) in a349c9d instead of rebasing, since the PR has reviews. sceCesSbcToUtf8 keeps this PR's measured checks and takes #9210's 4-byte UTF-8 buffer; the debt file keeps #9210's GBK and GB 18030 entries next to this PR's Cp1250 and sceCesSbcToUtf8 entries. All 14 guest_ces* tests pass three runs in a row on Windows (MinGW GCC 15.2) and Linux (WSL, GCC) at that head.

[orezraey]

@rooklz: #9175, #10430 and #10431 change the same lines as this PR in core/libs/prx/libSceCesCs/Export.cpp and docs/dev/debt/libSceCesCs.md (git merge-tree of both heads over current main conflicts there). I propose they land first; I will then merge main into this branch, keeping both sides' behaviour, and rerun its tests. Depends on now lists them (merge order only).

[orezraey]

@rooklz @boykopovar: git merge-tree of this stack (#10183, #10184, #10185, #10186) with your open PRs:

• feat(libSceCesCs): report the JIS X 0213 level of a cell as the console does #10751: core/libs/prx/libSceCesCs/Export.cpp conflicts in all three that carry this PR: both name 0x805C0015 (CES_ERROR_ILLEGAL_CODE here, CES_ERROR_OUT_OF_CODE_SPACE there).
• fix(libSceCesCs): report the length of a rejected MBCS character as the console does #10722: conflicts with fix(libSceCesCs): terminate and count MBCS string conversions as measured on a PS5 #10185 and fix(libSceCesCs): encode EUC-JP and Shift-JIS as measured on a PS5 #10186 in the EUC-JP decoder in Export.cpp, the DecodeFails/DecodeRejects checks in GuestCesJapanese.cpp and docs/dev/debt/libSceCesCs.md; with this PR only in the debt file.
• fix(libSceCesCs): map the GB 18030 profiles as the console does #10739: docs/dev/debt/libSceCesCs.md (and GuestCesChinese.cpp for fix(libSceCesCs): terminate and count MBCS string conversions as measured on a PS5 #10185 and fix(libSceCesCs): encode EUC-JP and Shift-JIS as measured on a PS5 #10186).
• refactor(libs): add full export headers for libs #10706: core/libs/tests/GuestCes.cpp in fix(libSceCesCs): convert single-byte characters as measured on a PS5 #10183 and fix(libSceCesCs): convert single EUC-JP characters as measured on a PS5 #10184 (already listed for fix(libSceCesCs): terminate and count MBCS string conversions as measured on a PS5 #10185 and fix(libSceCesCs): encode EUC-JP and Shift-JIS as measured on a PS5 #10186).

I propose those land first; each PR of this stack will then be refreshed over them, keeping your rejected-length and JIS X 0213 behaviour and this stack's decoders, and its tests rerun. I listed them in Depends on of each PR (merge order only).

[rooklz]

@orezraey data from a second console that may help this PR: I swept sceCesSbcToUtf32 over every byte and sceCesUtf32ToSbc over every BMP scalar for all 42 single-byte profiles on main plus #10430's. Your Cp1252, Cp1251 and Cp1250 holes match mine exactly. The two other Windows code pages on main behave the same way and still map their holes to C1 here:

• Cp1254: 81 8D 8E 8F 90 9D 9E answer 0x805C0020 both ways
• Cp1258: 81 8A 8D 8E 8F 90 9A 9D 9E answer 0x805C0020 both ways
• Cp1256 has no holes, and the ISO 8859, PC code page and KOI8 tables on main match the console byte for byte.

Every single-byte profile also answers 0x805C0015 for a surrogate in sceCesUtf32ToSbc, where main gives 0x805C0014. If you'd like the Cp1254/1258 holes in this PR I can send the dump; otherwise I'll leave them for after it lands.

[orezraey]

@rooklz thanks, a second console agreeing on the Cp1252, Cp1251 and Cp1250 holes is useful evidence for this PR. I would keep this PR to those three measured profiles, since #10184, #10185 and #10186 stack on it and it already waits on several overlapping PRs. The Cp1254 and Cp1258 holes and the 0x805C0015 surrogate result of sceCesUtf32ToSbc fit better in your own follow-up after the stack lands, with your dump as its evidence; the shared unassigned-byte lookup here should make the table change small.

[orezraey]

Everything in this PR landed with #10185, which contained this branch. I merged main (f4aabbb122) in, and the branch now has no diff against main, so nothing is left to review here.

[oneandonlydean]

Thanks! After the latest merge from main this PR has no changes left: the conversion work landed with #10185. Closing it as done.