[orezraey]

What

The MBCS string conversions (sceCesUtf8StrToMbcsStr, sceCesUcs2StrToMbcsStr, sceCesMbcsStrToUtf8Str, sceCesMbcsStrToUcs2Str and their Get*Len forms) wrote no terminator and set no length on an error. I measured them on a retail PS5 (firmware 13.60) with homebrew probes, from a payload process loading libSceCesCs.sprx, with the Cp1252 and EUC-JP profiles. The sources and console logs are in this gist (pces8, pces9, pces10).

On the console:

• Terminator. The output always gets a terminating zero, which is not counted. A character is written only if it and the terminator fit. Converting ABC into 3 bytes gives AB, a terminator and 0x805C0031 with lengths 2 and 2. With a capacity of 0, nothing is written.
• Errors keep what was converted. On an error mid-string, the characters before it stay written, terminated and counted. For example, A, 0x81, B in Cp1252 returns 0x805C0020 with lengths 1 and 1 and the output A, terminator.
• Source limit 0 reads up to the terminating zero. We converted nothing.
• Order. The lengths are cleared and a terminator is written first. Then the checks run in this order: context (0x805C0001), source (0x805C0010), output (0x805C0030).
• Decoding.
  • UTF-8 sources decode as sceCesUtf8ToSbc does: overlong forms and surrogates give 0x805C0015, and a zero byte inside a sequence is an invalid byte.
  • A UTF-16 surrogate gives 0x805C0015.
  • EUC-JP sources decode as sceCesEucJpToUcs2 does, so 8F A2 AF gives 0x805C0014.
• No JIS X 0212 encoding either. U+02D8, U+0386, U+0402 and U+4E02 give 0x805C0020 with the EUC-JP profile. We encoded them as 0x8F triples.

This PR implements all of the above with one conversion loop shared by the eight functions. sceCesMbcsUcsContextInit with a null profile now returns 0x805C0001 as well, as the console did; before, it stored the null profile.

Not changed, recorded in the debt file. The console encodes U+00A5, U+203E and U+0080–U+009F to EUC-JP bytes where we answer UNASSIGNED_CODE. Its Shift-JIS encoder answers 0x805C0024 for U+0080–U+009F and accepts the JIS forms of a few Microsoft characters. These table choices are a separate change. A zero-filled context converted nothing and returned 0 on the console; here it returns 0x805C0001.

Depends on the two scalar PRs, for decodeScalarUtf8 and decodeEucJpScalar. This branch carries their commits until they merge.

On top of #8967 and #8972. The UTF-16 and UTF-32 string forms from #8972 now run through the same loop, so they stop, terminate and count the same way. sceCesUtf8ToSJis, sceCesUtf8ToEucJp and the GBK and EUC-CN forms from #8976 decode UTF-8 like sceCesUtf8ToSbc. JIS X 0212 is neither decoded nor encoded, as measured, so two guest_ces_japanese checks that expected 0x8FB0A1 for U+4E02 now expect INVALID_ENCODE and UNASSIGNED_CODE.

Tested

• Head after merging main (f26434afa3, with refactor(libs): add full export headers for libs #10706): 4c3c3231d6. The test now takes its declarations from libSceCesCs/include/Api.hpp (sceCesRefersUcsProfileCp1250 returns const void* there, so the test casts it). The 14 guest_ces* tests pass three runs in a row on Windows (MinGW) and on Linux (WSL Ubuntu, GCC); public_headers audit and check_conventions pass.
  Windows 11, MinGW (winlibs).
• guest_ces checks:
  • 30 measured string cases with result, both lengths and the exact output bytes up to the terminator;
  • the order and terminator cases;
  • the Get*Len forms;
  • the UCS-2 forms.
• It fails without the change and passes with it.
• The existing checks now expect the partial output after a failed call.
• The EUC-KR, UHC and Big5 round trip in guest_ces_dbcs had output buffers with no room for a terminator. It now keeps that room; before, it passed without converting anything.
• The ten guest_ces* tests pass three runs in a row.
• tools/check_conventions.py passes.
• tools/pr_overlap.py lists feat(libSceCesCs): add 27 single-byte code page profiles #8923, feat(libSceCesCs): add the Windows-1253, 1255 and 1257 profiles #9912 and feat(libSceCesCs): add the GB18030-2000 and GB18030-2005 profiles #9210 (other code pages in Export.cpp and the tests) and test: organize library tests and label CTest suites #6044 (moves test files).

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: fix(libSceCesCs): convert single-byte characters as measured on a PS5 #10183, fix(libSceCesCs): convert single EUC-JP characters as measured on a PS5 #10184, feat(libSceCesCs): encode UCS-2, UTF-8, UTF-16 and UTF-32 characters into any MBCS profile #9175, feat(libSceCesCs): add the EUC-JIS-2004 and Shift_JIS-2004 profiles #10426, feat(libSceCesCs): add the ISO 646 national variants and JIS X 0201 profiles #10430, feat(libSceCesCs): select the single-byte set of an MBCS profile and a replacement character for a context #10431; feat(libSceCesCs): encode UCS-2, UTF-8, UTF-16 and UTF-32 characters into any MBCS profile #9175, feat(libSceCesCs): add the EUC-JIS-2004 and Shift_JIS-2004 profiles #10426, feat(libSceCesCs): add the ISO 646 national variants and JIS X 0201 profiles #10430, feat(libSceCesCs): select the single-byte set of an MBCS profile and a replacement character for a context #10431 edit the same lines too (merge order only); fix(libSceCesCs): map the strict JIS X 0208 profiles' unsettled cells as the console does #10718 rewrites the same EUC-JP decoder and debt entries, and refactor(libs): add full export headers for libs #10706 removes the test declarations this PR adds to GuestCes.cpp (merge order only); fix(libSceCesCs): report the length of a rejected MBCS character as the console does #10722, fix(libSceCesCs): map the GB 18030 profiles as the console does #10739, feat(libSceCesCs): report the JIS X 0213 level of a cell as the console does #10751 edit the same lines too (merge order only)
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[MarcZX10]

Reviewed the MBCS conversion-loop change at head 00b45d4, stacked on #10183 and #10184. The shared loop clears lengths and initializes the terminator before validation, preserves converted prefixes on error, reserves terminator capacity, and handles sourceMax=0 as the measured NUL-terminated mode. UTF-8/EUC-JP/UCS-2 decode paths and the new focused cases align with the described firmware measurements. This exact head is included in the descendant #10186 syntax check; its hosted eligibility, conventions, intake, labels and report checks pass, while product builds are path-skipped. The author reports the ten focused tests passing three consecutive runs on Windows. I could not execute them here because full CMake configuration is blocked by the missing SDL2 development package. The declared dependencies match the branch stack.

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 10428	core/libs/prx/libSceCesCs/Export.cpp L27-28, L631-657 core/libs/tests/GuestCesJapanese.cpp L158-163 docs/dev/debt/libSceCesCs.md L12-14, L18-19
PR 10718	core/libs/prx/libSceCesCs/Export.cpp L631-657 core/libs/tests/GuestCesJapanese.cpp L158-163 docs/dev/debt/libSceCesCs.md L12-14
PR 10722	core/libs/prx/libSceCesCs/Export.cpp L631-657 core/libs/tests/GuestCesJapanese.cpp L69, L72-73, L159-163, L173-174, L182-183, L202, L220, L223-224, L234 docs/dev/debt/libSceCesCs.md L18-19
PR 10739	core/libs/tests/GuestCesChinese.cpp L228-231 docs/dev/debt/libSceCesCs.md L17-18
PR 10751	core/libs/prx/libSceCesCs/Export.cpp L27, L572-573

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 9912	core/libs/prx/libSceCesCs/Export.cpp core/libs/tests/GuestCes.cpp docs/dev/debt/libSceCesCs.md

[MarcZX10]

Re-review of exact current head e4ed848821e95fb162923e68991d6c32d9d5cfb8. The source/test/debt diff against current main is unchanged by this head's merge-main refresh. Its checklist records the agreed #10183/#10184 prerequisites, and #10186 records #10185 as its prerequisite; #10056 then depends on #10186. Exact-head repository conventions and git diff --check pass, and GCC C++20 strict syntax-only compilation of the CES export passes. Hosted eligibility, conventions, labels and report pass; product test/build jobs are path-skipped. The author reports the focused CES tests passing three runs; no full runtime suite was available locally. Approving this exact head in the agreed stack order.

[oneandonlydean]

@orezraey #9204 and #9210 are landing ahead of this PR and both change libSceCesCs/Export.cpp, so this needs a rebase. Taking either side whole drops behaviour. #9204 adds the strict JIS/SS2/SS3 profiles and the JIS X 0212 reverse cells, and #9210 adds the GB18030 supplementary mappings and encodeUtf16. This PR replaces the profile-aware reverse map and adds convertString. Please keep the strict-profile selection and the GB18030/supplementary paths inside your new helpers, with the measured CP51932 and string-length semantics on top. Then rerun the Japanese, Chinese, Unicode and string guest_ces* tests on Linux and Windows.

[orezraey]

Merged main (with #9204) in cce3beb. A new decodeEucJp keeps the profile selection: the measured CP51932 profile goes to decodeEucJpScalar, and the strict X0208, X0208Ss2 and X0208Ss2Ss3 profiles keep #9204's SS2/SS3 and strict-table rules, both for the string loop and for sceCesEucJpToUcs2/sceCesEucJpToUtf8. The reverse map keeps #9204's strict and supplementary cells, but only X0208Ss2Ss3 encodes JIS X 0212, because CP51932 answered 0x805C0020 for those code points on the console. Shift-JIS still never encodes a supplementary cell, and the new EUC-KR, UHC and Big5 UTF-8 forms decode with decodeScalarUtf8 like the other scalar forms. Three tests from main changed to this PR's semantics: the strict-profile sceCesEucJpToUcs2 failures now expect the cleared output and the console-style length, the EUC-KR overlong case expects 0x805C0015, and the UCS-2 string comparison in guest_ces_dbcs leaves room for the terminator. All 14 guest_ces* tests pass three runs in a row on Windows; I have no Linux machine for this, so Linux is covered by CI. #9210 has not landed yet; I will fold its GB18030 and encodeUtf16 paths into convertString when it does.

[oneandonlydean]

@orezraey The #9204 resolution preserves strict JIS and the measured CP51932 split. Please retain the #9210 prerequisite: after it lands, fold supplementary GB18030 decode/encode and encodeUtf16 into Ces.cpp’s convertString without losing that split, then rerun all 14 guest_ces cases on Linux and Windows and report the final head. The current branch still lacks that integration, so the earlier request is only partly resolved.

[orezraey]

Agreed, #9210 stays the prerequisite, and it is still open. Once it lands I will fold its supplementary GB18030 decode/encode and encodeUtf16 into convertString in Ces.cpp keeping the strict JIS and CP51932 split, then report all 14 guest_ces cases on Linux and Windows with the final head. The AI-assisted line now reads yes.

[orezraey]

#9210 has landed, so this branch now merges #10183 and #10184 refreshed on main (head b51910b). The MBCS string forms keep convertString: the UTF-16 forms encode with #9210's encodeUtf16, so GB 18030 supplementary characters become surrogate pairs, and UCS-2 answers UNASSIGNED_CODE for them. #9210's GB 18030 string test now checks the short-buffer error in its own buffer, since a failed conversion terminates its output here. All 14 guest_ces* tests pass three runs in a row on Windows (MinGW GCC 15.2) and Linux (WSL, GCC) at b51910b.

[orezraey]

@rooklz: #9175, #10426, #10430 and #10431 change the same lines as this PR in core/libs/prx/libSceCesCs/Export.cpp and docs/dev/debt/libSceCesCs.md (git merge-tree of both heads over current main conflicts there). I propose they land first; I will then merge main into this branch, keeping both sides' behaviour, and rerun its tests. Depends on now lists them (merge order only).

[orezraey]

@rooklz #10718 and this PR both rewrite the EUC-JP decoder in core/libs/prx/libSceCesCs/Export.cpp (this PR moves it into decodeEucJp, #10718 changes its strict JIS X 0208 lookup) and the EUC-JP and strict-profile entries of docs/dev/debt/libSceCesCs.md. The changes do not contradict each other. I suggest #10718 lands first; this PR will then be refreshed to keep #10718's strict-profile tables inside decodeEucJp and its debt wording, with this PR's rules unchanged. Separately, #10706 removes the local test declarations this PR adds to core/libs/tests/GuestCes.cpp; this PR will be refreshed to use its headers if it lands first. I added both to Depends on (merge order only).