feat(libSceCesCs): convert GBK and EUC-CN through the CP936 table by simonbronk · Pull Request #8976 · boykopovar/AnyPS5

simonbronk · 2026-10-21T08:11:47Z

What

Implements 34 libSceCesCs exports that were pending stubs:

sceCesUcsProfileInitGbkCp936, sceCesUcsProfileInitEucCnGb2312
sceCesGbTo{Ucs2,Utf8,Utf16,Utf16be,Utf16le,Utf32,Utf32be,Utf32le} and sceCesEucCnTo{…}
sceCes{Ucs2,Utf8,Utf16,Utf16be,Utf16le,Utf32,Utf32be,Utf32le}ToGb and …ToEucCn

The new Cp936.hpp holds the 21,791 double-byte cells of Microsoft's CP936.TXT, with single-byte 0x80 mapped to U+20AC as that file does. The GB 2312 mapping file is no longer published by unicode.org, so the EUC-CN profile uses the same table restricted to 0xA1-0xFE lead and trail bytes; the differences this implies (GBK cells inside that range, CP936 forms of 0xA1A4 and 0xA1AA) are recorded in docs/dev/debt/libSceCesCs.md. The MBCS string conversions (sceCesUtf8StrToMbcsStr and the others) accept both profiles. The scalar helpers from #8967 are renamed from *Japanese to *Profiled because they now serve both languages. Signatures follow the Orbital type database. Errors follow the existing converters and are not console measurements. User-defined areas, GB 18030 four-byte forms and sceCesGbUcsProfileSet* stay unimplemented. libSceCesCs goes from 86 to 120 of 284 in tools/progress.py.

This branch is stacked on #8972 (itself on #8967); only the top commit is new.

Tested

macOS 27 (Apple silicon), clang, C++20: libSceCesCs sources and tests compiled outside CMake, since the full build does not run on macOS. Passing:

new guest_ces_chinese: errors, byte orders, a GBK and EUC-CN round trip of every encodable BMP code point, and GBK strings through an MBCS context;
existing guest_ces, guest_ces_unicode, guest_ces_japanese and guest_ces_wide, with no regressions.

Cross-check: sceCesGbToUcs2 on all 32,766 lead/trail pairs matches Python's gbk codec (24,066 identical decodes; all other pairs fail in both).

The CMake/ctest build on Linux and Windows was not run locally. check_conventions.py is clean. pr_overlap.py lists only shared files with #8923, #7884 and #6044, none implementing these functions; #7884 edits the same MBCS dispatch in Export.cpp, so whichever lands second needs a rebase. No title tested.

Checklist

🤖 Generated with Claude Code

github-actions · 2026-10-21T08:12:26Z

Progress report (`b3e42bd` → `cd4bfc3`)

📈 System libraries: 46.74% (+0.48%, +56 implemented functions)

✅ 56 implemented

Library	Name
libSceCesCs	`sceCesEucCnToUcs2`
libSceCesCs	`sceCesEucCnToUtf16`
libSceCesCs	`sceCesEucCnToUtf16be`
libSceCesCs	`sceCesEucCnToUtf16le`
libSceCesCs	`sceCesEucCnToUtf32`
libSceCesCs	`sceCesEucCnToUtf32be`
libSceCesCs	`sceCesEucCnToUtf32le`
libSceCesCs	`sceCesEucCnToUtf8`
libSceCesCs	`sceCesGbToUcs2`
libSceCesCs	`sceCesGbToUtf16`
libSceCesCs	`sceCesGbToUtf16be`
libSceCesCs	`sceCesGbToUtf16le`
libSceCesCs	`sceCesGbToUtf32`
libSceCesCs	`sceCesGbToUtf32be`
libSceCesCs	`sceCesGbToUtf32le`
libSceCesCs	`sceCesGbToUtf8`
libSceCesCs	`sceCesMbcsStrGetUtf16Len`
libSceCesCs	`sceCesMbcsStrGetUtf32Len`
libSceCesCs	`sceCesMbcsStrToUtf16Str`
libSceCesCs	`sceCesMbcsStrToUtf32Str`
libSceCesCs	`sceCesSbcToUcs2`
libSceCesCs	`sceCesSbcToUtf16`
libSceCesCs	`sceCesSbcToUtf16be`
libSceCesCs	`sceCesSbcToUtf16le`
libSceCesCs	`sceCesSbcToUtf32`
libSceCesCs	`sceCesSbcToUtf32be`
libSceCesCs	`sceCesSbcToUtf32le`
libSceCesCs	`sceCesUcs2ToEucCn`
libSceCesCs	`sceCesUcs2ToGb`
libSceCesCs	`sceCesUcs2ToSbc`
libSceCesCs	`sceCesUcsProfileInitEucCnGb2312`
libSceCesCs	`sceCesUcsProfileInitGbkCp936`
libSceCesCs	`sceCesUtf16StrGetMbcsLen`
libSceCesCs	`sceCesUtf16StrToMbcsStr`
libSceCesCs	`sceCesUtf16ToEucCn`
libSceCesCs	`sceCesUtf16ToGb`
libSceCesCs	`sceCesUtf16ToSbc`
libSceCesCs	`sceCesUtf16beToEucCn`
libSceCesCs	`sceCesUtf16beToGb`
libSceCesCs	`sceCesUtf16beToSbc`
libSceCesCs	`sceCesUtf16leToEucCn`
libSceCesCs	`sceCesUtf16leToGb`
libSceCesCs	`sceCesUtf16leToSbc`
libSceCesCs	`sceCesUtf32StrGetMbcsLen`
libSceCesCs	`sceCesUtf32StrToMbcsStr`
libSceCesCs	`sceCesUtf32ToEucCn`
libSceCesCs	`sceCesUtf32ToGb`
libSceCesCs	`sceCesUtf32ToSbc`
libSceCesCs	`sceCesUtf32beToEucCn`
libSceCesCs	`sceCesUtf32beToGb`
libSceCesCs	`sceCesUtf32beToSbc`
libSceCesCs	`sceCesUtf32leToEucCn`
libSceCesCs	`sceCesUtf32leToGb`
libSceCesCs	`sceCesUtf32leToSbc`
libSceCesCs	`sceCesUtf8ToEucCn`
libSceCesCs	`sceCesUtf8ToGb`

github-actions · 2026-10-21T08:21:59Z

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on `main`)	Same exports	Same new files
PR 9107	`core/libs/prx/libSceCesCs/Export.cpp` L1229 `core/libs/prx/libSceCesCs/PendingExports.cpp` L5-79, L465-484, L490-499, L615-634, L640-664, L670-694, L715-734, L740-764, L770-794, L815-834 `docs/dev/debt/libSceCesCs.md` L17-18
PR 10056	`docs/dev/debt/libSceCesCs.md` L16-18

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 6044	`core/libs/prx/libSceCesCs/CMakeLists.txt`
PR 9874	`core/libs/prx/libSceCesCs/Export.cpp` `core/libs/prx/libSceCesCs/PendingExports.cpp` `docs/dev/debt/libSceCesCs.md`
PR 9912	`core/libs/prx/libSceCesCs/Export.cpp` `core/libs/prx/libSceCesCs/PendingExports.cpp` `docs/dev/debt/libSceCesCs.md`

ycbuzi · 2026-10-21T08:28:23Z

Windows run of a4049f3 merged cleanly onto main f894ba6, the CMake/ctest build the description lists as not run. The head carries the commits of #8967 and #8972 too. Windows 11 Home 10.0.26200, MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release, BUILD_TESTING=ON.

Built libSceCesCs, its module form and the ten test executables with no errors or warnings. ctest --repeat until-fail:2 on guest_ces, guest_ces_module, guest_ces_unicode, guest_ces_unicode_module, guest_ces_japanese, guest_ces_japanese_module, guest_ces_wide, guest_ces_wide_module, guest_ces_chinese and guest_ces_chinese_module: all ten pass on both repeats, the new guest_ces_chinese and its module form among them. Not run: Linux, a title.

clairtonmichel · 2026-10-21T08:43:43Z

Tested on AMD Ryzen 7 9800X3D and AMD Radeon RX 7900 XTX (Fedora Linux, Mesa 26.2.5):

Built cleanly without any errors.
Ran the localization and charset conversion test suite (including guest_ces_chinese, unicode, and module tests): 100% passed successfully (48/48 tests green).
GBK and EUC-CN CP936 conversion routines work flawlessly on host. Great contribution!

…ring conversions Implement sceCesSbcTo{Ucs2,Utf16,Utf32}* and their inverses, and the UTF-16 and UTF-32 MBCS string conversions and length queries, on the existing single-byte, CP932 and CP51932 profiles. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Add the GBK (CP936) and EUC-CN profiles, their scalar conversions to and from UTF-8, UTF-16, UTF-32 and UCS-2, and accept both profiles in the MBCS string conversions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

simonbronk · 2026-10-21T18:16:56Z

Linux run of f974e91 merged onto main 99844c7 (clean merge).

Environment: Ubuntu 24.04 x86-64 container under Rosetta on Apple Silicon (Colima), GCC 13, RelWithDebInfo, BUILD_TESTING=ON, ANYPS5_ENABLE_ASAN=ON. ASan builds dynamic executables, so this is not the CI configuration.

Results: the full build has no errors. guest_ces, guest_ces_chinese, guest_ces_dbcs, guest_ces_japanese, guest_ces_unicode, guest_ces_wide and their module forms: all pass, with no AddressSanitizer report.

Not run: the static CI configuration, native x86-64 hardware, GPU tests, a title.

AI-assisted (Claude Code).

ycbuzi · 2026-10-22T08:34:26Z

Windows run of f974e91, the update since my earlier run, merged onto main d6eafa5 (1d6ee41de0, clean). Windows 11 Home 10.0.26200, MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release, BUILD_TESTING=ON.

Built libSceCesCs, its module form and the ten test executables with no errors or warnings. ctest --repeat until-fail:2 on guest_ces, guest_ces_module, guest_ces_unicode, guest_ces_unicode_module, guest_ces_japanese, guest_ces_japanese_module, guest_ces_wide, guest_ces_wide_module, guest_ces_chinese and guest_ces_chinese_module: all ten pass on both repeats. Not run: a title.

simonbronk · 2026-10-23T09:35:01Z

Updated for #8972's main merge (4c33e22): merged it in as cd4bfc3, so this branch now carries #8972's exact head. The only conflict was docs/dev/debt/libSceCesCs.md: main's single-byte profile entry is kept next to this PR's GBK/EUC-CN entry, and the pending-export count goes from 168 to 134 (the 34 exports this PR implements), matching PendingExports.cpp.

Linux run of cd4bfc3, same container as on #8972 (GCC 13, RelWithDebInfo with ASan): no build errors in AnyPS5 code (only the known atrac9_gradient_tests link failure in this container), and the 12 guest_ces* tests pass with no ASan report, guest_ces_chinese and its module form included. check_conventions.py --base origin/main passes; it merges cleanly into main 8948f1c. pr_overlap.py lists #9175, #9204 and #9210, which are stacked on #8993, and #10056 for an adjacent edit in the debt file.

AI-assisted: yes

simonbronk mentioned this pull request Oct 21, 2026

feat(libSceCesCs): implement generic MBCS character, Shift-JIS code and JIS X 0213 helpers #8993

Merged

12 tasks done

github-actions Bot added build docs prx needs rebase Conflicts with main, rebase needed labels Oct 21, 2026

jacob6769 mentioned this pull request Oct 21, 2026

feat(libSceCesCs): implement the EUC-KR, UHC and Big5 scalar conversions #9107

Merged

12 tasks done

Simone and others added 2 commits October 21, 2026 15:23

simonbronk force-pushed the feat/libSceCesCs-gbk-euc-cn branch from a4049f3 to f974e91 Compare October 21, 2026 13:26

github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 21, 2026

rooklz mentioned this pull request Oct 21, 2026

feat(libSceCesCs): encode UCS-2, UTF-8, UTF-16 and UTF-32 characters into any MBCS profile #9175

Merged

12 tasks done

nikowolf123 mentioned this pull request Oct 21, 2026

feat(libSceCesCs): add 27 single-byte code page profiles #8923

Merged

12 tasks done

This was referenced Oct 21, 2026

feat(libSceCesCs): add the strict JIS X 0208 EUC-JP and Shift-JIS profiles #9204

Merged

feat(libSceCesCs): add the GB18030-2000 and GB18030-2005 profiles #9210

Merged

oneandonlydean self-assigned this Oct 22, 2026

github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 23, 2026

Merge main into feat/libSceCesCs-sbc-mbcs-utf16-utf32

4c33e22

atharvjairath mentioned this pull request Oct 23, 2026

feat(ces): add eight sparse single-byte profiles #10056

Open

12 tasks done

Merge boykopovar#8972 into feat/libSceCesCs-gbk-euc-cn

cd4bfc3

simonbronk mentioned this pull request Oct 23, 2026

feat(libSceCesCs): add UTF-16 and UTF-32 forms of the SBC and MBCS string conversions #8972

Merged

12 tasks done

simonbronk pushed a commit to simonbronk/AnyPS5 that referenced this pull request Oct 23, 2026

Merge boykopovar#8976 into feat/libSceCesCs-mbc-sjis-code

79abbd6

github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 23, 2026

oneandonlydean merged commit d1b6cdb into boykopovar:main Oct 23, 2026
10 checks passed

orezraey mentioned this pull request Oct 23, 2026

fix(libSceCesCs): terminate and count MBCS string conversions as measured on a PS5 #10185

Merged

12 tasks done

Navigation Menu

Repository navigation

feat(libSceCesCs): convert GBK and EUC-CN through the CP936 table - #8976

feat(libSceCesCs): convert GBK and EUC-CN through the CP936 table#8976
oneandonlydean merged 4 commits into
boykopovar:mainfrom
simonbronk:feat/libSceCesCs-gbk-euc-cn

simonbronk commented Oct 21, 2026

Uh oh!

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

ycbuzi commented Oct 21, 2026

Uh oh!

clairtonmichel commented Oct 21, 2026

Uh oh!

simonbronk commented Oct 21, 2026

Uh oh!

ycbuzi commented Oct 22, 2026

Uh oh!

simonbronk commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

simonbronk commented Oct 21, 2026

What

Tested

Checklist

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress report (b3e42bd → cd4bfc3)

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ycbuzi commented Oct 21, 2026

Uh oh!

clairtonmichel commented Oct 21, 2026

Uh oh!

simonbronk commented Oct 21, 2026

Uh oh!

ycbuzi commented Oct 22, 2026

Uh oh!

simonbronk commented Oct 23, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Progress report (`b3e42bd` → `cd4bfc3`)

github-actions Bot commented Oct 21, 2026 •

edited

Loading