[orezraey]

What

sceCesEucJpToUtf8 and sceCesEucJpToUcs2 used assumed errors and decoded JIS X 0212. I measured them on a retail PS5 (firmware 13.60) with homebrew probes, from a payload process loading libSceCesCs.sprx. I swept every two-byte input and every 0x8F three-byte input with sceCesEucJpToUcs2. The sources and console logs are in this gist (pces3, pces4, pces5, pces7).

What the console does, and what changes here:

• Single bytes. Every byte below 0xA0 except 0x0E, 0x8E and 0x8F converts to itself, including 0x80–0x8D and 0x90–0x9F, which we rejected. 0x0E, 0xA0 and 0xFF return 0x805C0014 with length 1; we accepted 0x0E.
• Half-width katakana. 0x8E with a second byte E0–FE returns 0x805C0020; we returned 0x805C0014.
• JIS X 0212 is not decoded. A 0x8F form returns 0x805C0020 in the rows X 0212 leaves empty (1, 3–5, 8, 12–15, 78–94) and 0x805C0014 elsewhere, always with length 3. We decoded 6067 X 0212 cells.
• A zero byte inside a sequence returns 0x805C0014 with the length set to its index. A short sequence returns 0x805C0011 with the full length.
• Errors and lengths.
  • A null profile returns 0x805C0004; we returned 0x805C0001.
  • A zero size returns 0x805C0010; we returned 0x805C0011.
  • Both length outputs, and the sceCesEucJpToUcs2 output, are cleared first.
  • The source length is then set for every decoded result, errors included.
  • sceCesEucJpToUtf8 sets the UTF-8 length before checking the output, which returns 0x805C0030 or 0x805C0031 and leaves the output untouched.

With the change, the per-lead hashes of result, length and output match the console's for all 256 leads and all 256 0x8F second bytes. The JIS X 0208 table (including NEC row 13 and the IBM extensions in rows 89–92) was already right.

Not changed. The console reads any other profile pointer as a sheet; with the Cp1252 profile it faulted. Here a pointer that is not an EUC-JP profile still returns 0x805C0001. The MBCS string conversions keep decoding JIS X 0212, although the console's sceCesMbcsStrToUtf8Str rejected 8F A2 AF with 0x805C0014; their errors and partial output are a separate change. The debt entry records both. The strict JIS X 0208 profiles that main added later were not measured: with them, both functions keep the earlier path through decodeMbcs, so their errors leave the outputs unchanged.

On top of #8967 and #8972. The measured byte rules apply only to sceCesEucJpToUcs2 and sceCesEucJpToUtf8, through decodeEucJpScalar. sceCesEucJpToUtf16 and sceCesEucJpToUtf32 from #8967 still decode through the unchanged decodeMbcs, so this PR does not change them.

Tested

• Head after merging main (f26434afa3, with refactor(libs): add full export headers for libs #10706): 1f9a014195. The test now takes its declarations from libSceCesCs/include/Api.hpp instead of its own extern "C" block. The 14 guest_ces* tests pass three runs in a row on Windows (MinGW) and on Linux (WSL Ubuntu, GCC); public_headers audit and check_conventions pass.
  Windows 11, MinGW (winlibs).
• guest_ces pins the FNV hash of result, length and output over all 65536 two-byte inputs and all 65536 0x8F three-byte inputs. It also checks the null, size, output and length cases above. The JIS X 0212 round trip now goes through the string conversions.
• guest_ces_unicode checks the single bytes, the half-width katakana and every JIS X 0208 cell with the measured lengths.
• Both fail on main and pass with the change. The fourteen guest_ces* tests pass three runs in a row, on Windows and on Linux (WSL, GCC).
• tools/check_conventions.py passes.
• tools/pr_overlap.py lists:
  • feat(libSceCesCs): add 27 single-byte code page profiles #8923, feat(libSceCesCs): add the Windows-1253, 1255 and 1257 profiles #9912 and feat(libSceCesCs): add the GB18030-2000 and GB18030-2005 profiles #9210 (other code pages in Export.cpp and GuestCes.cpp), none of which changes these functions;
  • test: organize library tests and label CTest suites #6044, which moves test files.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: refactor(libs): add full export headers for libs #10706 (merge order only: they edit the same lines; this PR is refreshed after them)
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[MarcZX10]

Reviewed the single-character EUC-JP decoder change at head e04647b. The measured length/error handling for single bytes, half-width forms, zero bytes, short sequences, and JIS X 0212 rows is reflected in the implementation and full-input hash regressions. The branch reports the ten focused CES tests passing three consecutive runs; eligibility, conventions, intake, labels and report checks pass, while product jobs are path-skipped. The implementation and tests also pass C++20 syntax checks in the descendant #10186 tree. I could not run the executable locally because the full project configure is blocked by missing SDL2 development files.

[oneandonlydean]

@orezraey Please correct the “On top of #8967 and #8972” paragraph in the description to match the implemented scope. decodeEucJpScalar at Export.cpp:468 supplies the measured byte rules only to EucJpToUcs2 and EucJpToUtf8 (lines 1054 and 1096). Utf16/Utf32 still go through profiledToUtf16/profiledToUtf32 → decodeProfiled → the unchanged decodeMbcs (lines 696–701 and 1152–1173). No code expansion is requested; please avoid claiming those other scalar forms gain the rules here.

[orezraey]

Corrected the "On top of #8967 and #8972" paragraph: it now says the measured byte rules reach only sceCesEucJpToUcs2 and sceCesEucJpToUtf8 through decodeEucJpScalar, and that sceCesEucJpToUtf16/sceCesEucJpToUtf32 still go through the unchanged decodeMbcs. No code changed.

[oneandonlydean]

@orezraey Full CI on main plus this PR (run 40021961421, head e04647b) fails guest_ces_japanese and guest_ces_japanese_module on both lanes: "Subprocess aborted" on Linux and exit 0xc0000409 on Windows. Both tests pass in other PRs' runs on the same main, so the abort comes from this change combined with current main. #9210 has also landed in libSceCesCs/Export.cpp since then. Please rebase onto main, keeping your decodeEucJpScalar decoder hunk alongside #9210's changes, and check these two tests on Linux and Windows.

[orezraey]

Merged main (with #9210) in 76945dd. The abort came from the strict JIS X 0208 EUC-JP profiles now on main: guest_ces_japanese sweeps them through sceCesEucJpToUcs2, and decodeEucJpScalar decoded them as CP51932, so their checks failed. Only the CP51932 profile, the one measured, now goes through decodeEucJpScalar; the strict profiles keep their decodeMbcs path from main, and the debt entry and description say so. All 14 guest_ces* tests, including guest_ces_japanese and guest_ces_japanese_module, pass three runs in a row on Windows (MinGW GCC 15.2) and Linux (WSL, GCC).

[rooklz]

@orezraey independent confirmation from a second console. I swept sceCesMbcToUtf32 on a PS5 with homebrew payloads over every one- and two-byte sequence, every 0x8F-led three-byte sequence and every 0x81 0x30-led sequence for all 15 MBCS profiles, and got the same EUC-JP rules you list: C1 bytes as themselves except 0x8E/0x8F, 0x0E refused, 0x8E E0-FE unassigned, no JIS X 0212 outside the X0208Ss2Ss3 profile (empty X 0212 rows 0x805C0020, the others 0x805C0014, length 3), and a NUL byte reporting its index. The length rule holds for every MBCS profile and for each encoding's To{Ucs2,Utf8,Utf16,Utf32}, which behave identically to sceCesMbcTo*.

To stay out of your way: I'm opening a PR for the lengths of rejected characters in the other encodings and in sceCesMbcTo*, and it leaves sceCesEucJpToUcs2 and sceCesEucJpToUtf8 to this PR. I also have the EUC-KR and EUC-CN single-byte rules (the same C1/0x0E pattern, with 0x8E/0x8F refused) and EUC-JP through decodeMbcs, which you left unchanged here. Would you rather extend this PR to decodeMbcs, or should I send that one after yours lands?

[orezraey]

@rooklz thanks for the second-console sweep; it matches what this PR measured. Please send the decodeMbcs change as its own PR: this one stays limited to the single-character EUC-JP conversions, and #10185 and #10186 already build on it. It can go up now with Depends on: #10184 (merge order only), or after this lands; I will keep decodeMbcs unchanged here.

[orezraey]

Everything in this PR landed with #10185, which contained this branch. I merged main (f4aabbb122) in, and the branch now has no diff against main, so nothing is left to review here.

[oneandonlydean]

Thanks! After the latest merge from main this PR has no changes left: the conversion work landed with #10185. Closing it as done.