feat(libSceCesCs): add 27 single-byte code page profiles by nikowolf123 · Pull Request #8923 · boykopovar/AnyPS5

nikowolf123 · 2026-10-21T06:15:35Z

What

sceCesRefersUcsProfile* threw not implemented for every code page except 1250, 1251 and 1252. 27 more now return a profile, like the existing three. sceCesSbcToUtf8, sceCesUtf8ToSbc and the MBCS context conversions (sceCesMbcsUcsContextInit with sceCesUtf8StrToMbcsStr, sceCesMbcsStrToUtf8Str, sceCesUcs2StrToMbcsStr and the others) accept these profiles:

Family	Profiles	Source of the high half (bytes 0x80-0xFF)
ISO 8859	`Iso8859_1`, `_2`, `_4`, `_5`, `_9`, `_10`, `_13`, `_14`, `_15`, `_16`	unicode.org ISO8859/8859-n.TXT
DOS code pages	`Cp437`, `Cp737`, `Cp775`, `Cp850`, `Cp852`, `Cp855`, `Cp860`, `Cp861`, `Cp862`, `Cp863`, `Cp865`, `Cp866`	Microsoft's PC tables, VENDORS/MICSFT/PC/CPnnn.TXT
KOI8	`Koi8R`, `Koi8U`	VENDORS/MISC/KOI8-R.TXT and KOI8-U.TXT
Windows	`Cp1254`, `Cp1256`, `Cp1258`	Microsoft's best-fit tables, bestfitnnnn.txt, byte table (`MBTABLE`)

Signatures: const struct* sceCesRefersUcsProfileXxx(void) for all of them, as in the Orbital type database that the library already cites for 1250-1252.
Same derivation as the existing tables: the high half of main's CP1250_HIGH is byte for byte the MBTABLE of bestfit1250.txt, and 1254, 1256 and 1258 are taken the same way. Their unassigned bytes map to the C1 control characters, as in the 1250 table.
Table properties: in every new table, bytes 0x00-0x7F map to themselves, and the 256 bytes map to 256 different BMP characters. So the reverse direction (sceCesUtf8ToSbc, MBCS output) is unambiguous, and a character outside the table returns UNASSIGNED_CODE (0x805C0020) through the existing code.
Code: the tables are in the new core/libs/prx/libSceCesCs/SingleByte.hpp, generated from those files. Export.cpp gets one lookup (sbcHighTable) that sbcToUnicode, unicodeToSbc and isSbc now use for 1250, 1251 and the new profiles. 1252 keeps its own path, and the three existing profiles keep their pointers and tables.
Not included, still throwing: the code pages whose public table leaves bytes undefined (ISO 8859-3, -6, -7, -8 and -11; 857, 864 and 869), the best-fit tables that map bytes to private-use characters (874, 1253, 1255 and 1257), and the profiles without a public table here (858, Ascii, AsciiWithKatakana, Bs4730, the five JisX0201*, the two KsX1003* and the two GbT1988*). For those, AnyPS5 would have to guess what the console does with the undefined bytes.
Unverified: the console's tables are not checked. This is recorded as a new "Single-byte profiles" entry in docs/dev/debt/libSceCesCs.md, and the pending-exports count there drops from 249 to 222.
Overlap: feat(libSceCesCs): convert EUC-KR, UHC and Big5 through the CP949 and CP950 tables #7884 (EUC-KR, UHC and Big5) is merged and this branch is rebased onto it; its isMbcs, decodeMbcs and encodeMbcs reach the new profiles through isSbc. Open feat(libSceCesCs): implement Shift-JIS and EUC-JP scalar conversions #8967, feat(libSceCesCs): add UTF-16 and UTF-32 forms of the SBC and MBCS string conversions #8972, feat(libSceCesCs): convert GBK and EUC-CN through the CP936 table #8976, feat(libSceCesCs): implement generic MBCS character, Shift-JIS code and JIS X 0213 helpers #8993, feat(libSceCesCs): implement the EUC-KR, UHC and Big5 scalar conversions #9107 and feat(libSceCesCs): encode UCS-2, UTF-8, UTF-16 and UTF-32 characters into any MBCS profile #9175 also change Export.cpp, PendingExports.cpp or the pending-exports count in the debt file, so whichever merges after this PR recounts that figure. feat(libSceCesCs): implement EUC-KR, UHC and Big5 to UTF-8 conversions #8804, which overlapped on the include block of Export.cpp and the helpers before main() in GuestCes.cpp, was closed in favour of feat(libSceCesCs): convert EUC-KR, UHC and Big5 through the CP949 and CP950 tables #7884. No open PR implements any of the 27.

Tested

Linux: WSL2 Ubuntu 26.04, GCC 15.2.0 (Ubuntu 15.2.0-16ubuntu1), Release. I built libSceCesCs, guest_ces_tests and guest_ces_module_tests and ran ctest -R '^guest_ces(_module)?$': guest_ces and guest_ces_module pass (2/2) at 0b3f339, on main ec7eacc. After the rebase onto main 2c4b116, at 27c3f6b, I built libs and the six guest_ces* test targets: ctest -R '^guest_ces' passes 6/6 (guest_ces, guest_ces_module, guest_ces_dbcs, guest_ces_dbcs_module, guest_ces_unicode and guest_ces_unicode_module). Export.cpp and GuestCes.cpp compile without warnings in both builds.

guest_ces (and guest_ces_module, the same test against the module build) now runs CheckSingleByteProfiles(). For each of the 27 profiles it checks:

that the profile is a non-null pointer, stable across calls, and different from the other 29 profiles;
that every byte 1-255 converts with sceCesSbcToUtf8, bytes below 0x80 to themselves, and converts back to the same byte with sceCesUtf8ToSbc;
that U+3042 returns UNASSIGNED_CODE;
four bytes per table (0x80, 0xA4, 0xC1 and 0xFF) against the code point in the source file.

It also checks a KOI8-R string through an MBCS context: "Привет" converts to F0 D2 C9 D7 C5 D4 and back. On main the first new call throws not implemented, so the test fails there.

tools/check_conventions.py --base origin/main passes. tools/progress.py --module libSceCesCs goes from 38 to 65 done functions out of 287 on main 2c4b116; the new ones are exactly these 27.

Windows: not built. No title was run.

Checklist

🤖 Generated with Claude Code

github-actions · 2026-10-21T06:16:25Z

Progress report (`316b742` → `b8bb173`)

📈 System libraries: 45.04% (+0.24%, +27 implemented functions)

✅ 27 implemented

Library	Name
libSceCesCs	`sceCesRefersUcsProfileCp1254`
libSceCesCs	`sceCesRefersUcsProfileCp1256`
libSceCesCs	`sceCesRefersUcsProfileCp1258`
libSceCesCs	`sceCesRefersUcsProfileCp437`
libSceCesCs	`sceCesRefersUcsProfileCp737`
libSceCesCs	`sceCesRefersUcsProfileCp775`
libSceCesCs	`sceCesRefersUcsProfileCp850`
libSceCesCs	`sceCesRefersUcsProfileCp852`
libSceCesCs	`sceCesRefersUcsProfileCp855`
libSceCesCs	`sceCesRefersUcsProfileCp860`
libSceCesCs	`sceCesRefersUcsProfileCp861`
libSceCesCs	`sceCesRefersUcsProfileCp862`
libSceCesCs	`sceCesRefersUcsProfileCp863`
libSceCesCs	`sceCesRefersUcsProfileCp865`
libSceCesCs	`sceCesRefersUcsProfileCp866`
libSceCesCs	`sceCesRefersUcsProfileIso8859_1`
libSceCesCs	`sceCesRefersUcsProfileIso8859_10`
libSceCesCs	`sceCesRefersUcsProfileIso8859_13`
libSceCesCs	`sceCesRefersUcsProfileIso8859_14`
libSceCesCs	`sceCesRefersUcsProfileIso8859_15`
libSceCesCs	`sceCesRefersUcsProfileIso8859_16`
libSceCesCs	`sceCesRefersUcsProfileIso8859_2`
libSceCesCs	`sceCesRefersUcsProfileIso8859_4`
libSceCesCs	`sceCesRefersUcsProfileIso8859_5`
libSceCesCs	`sceCesRefersUcsProfileIso8859_9`
libSceCesCs	`sceCesRefersUcsProfileKoi8R`
libSceCesCs	`sceCesRefersUcsProfileKoi8U`

MarcZX10

I reviewed the profile dispatch and generated tables. I independently compared every high-half entry in all 27 tables against the corresponding Python ISO-8859, CP, and KOI8 codecs (treating undefined CP125x bytes as C1 controls, as documented); all entries matched. The tables remain one-to-one with the ASCII low half. The PR’s focused Linux tests and convention check are author-reported; I did not rerun the guest tests. The current conventions/report checks pass, and I found no duplicate profile implementation.

github-actions · 2026-10-21T06:33:18Z

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on `main`)	Same exports	Same new files
PR 8967	`docs/dev/debt/libSceCesCs.md` L14
PR 8972	`docs/dev/debt/libSceCesCs.md` L14
PR 8976	`docs/dev/debt/libSceCesCs.md` L14
PR 8993	`docs/dev/debt/libSceCesCs.md` L14
PR 9107	`docs/dev/debt/libSceCesCs.md` L14-15
PR 9175	`docs/dev/debt/libSceCesCs.md` L14
PR 9204	`docs/dev/debt/libSceCesCs.md` L14
PR 9210	`docs/dev/debt/libSceCesCs.md` L14
PR 9970	`core/libs/prx/libSceCesCs/Export.cpp` L62-65, L70-71, L148, L536 `docs/dev/debt/libSceCesCs.md` L14

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 6044	`core/libs/tests/GuestCes.cpp`

ycbuzi · 2026-10-21T07:35:19Z

Windows run of 0b3f339 merged onto main b77bd58 (1e27b5f5bd), the build the description lists as not done. Windows 11 Home 10.0.26200, MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release, BUILD_TESTING=ON.

Built libSceCesCs, libSceCesCs-module and the four guest_ces* test executables with no errors or warnings; ctest --repeat until-fail:2 on guest_ces, guest_ces_module, guest_ces_unicode and guest_ces_unicode_module: all pass on both repeats. Not run: the negative control against main's library, a title.

rooklz · 2026-10-21T14:20:15Z

@nikowolf123 nice work on the single-byte profiles. I agree with leaving the pages with undefined bytes alone. Two groups you left out look fully defined to me, though: CP858 (no undefined bytes; ICU ibm-858_P100-1997.ucm), and the 7-bit ISO 646 / JIS X 0201 sets with published tables (Ascii, Bs4730, GbT1988 and its tilde variant, JisX0201, JisX0201Roman and their tilde variants, from unicode.org JIS0201.TXT and the glibc charmaps in icu-data). Are you planning those? If not, I would do them on top of this PR once it lands and leave KsX1003*, AsciiWithKatakana and JisX0201Katakana throwing, since their contents are not settled.

sceCesRefersUcsProfile threw for every code page except 1250, 1251 and 1252. Twenty-seven more now return a profile that sceCesSbcToUtf8, sceCesUtf8ToSbc and the MBCS context conversions accept: ISO 8859-1, -2, -4, -5, -9, -10, -13, -14, -15 and -16, code pages 437, 737, 775, 850, 852, 855, 860, 861, 862, 863, 865 and 866, KOI8-R, KOI8-U and Windows 1254, 1256 and 1258. SingleByte.hpp holds the high half of each table, taken from the public mapping files at unicode.org: ISO8859, Microsoft's PC tables, VENDORS/MISC for KOI8 and Microsoft's best-fit tables for 1254, 1256 and 1258, the same source and derivation as the existing Cp1250 table, whose high half equals bestfit1250's byte table. Bytes 0x00-0x7F are unchanged in all of them, and every table maps its 256 bytes to 256 distinct BMP characters. The three existing profiles keep their tables; one lookup now serves them all. Profiles whose public table leaves bytes undefined (ISO 8859-3, -6, -7, -8, -11, code pages 857, 864, 869), maps bytes to private use (874, 1253, 1255, 1257) or has no public table here still throw. guest_ces checks that each new profile is a distinct, stable pointer, four bytes per table against the source, a round trip of every byte through sceCesSbcToUtf8 and sceCesUtf8ToSbc, UNASSIGNED_CODE for a character outside the table, and a KOI8-R string through an MBCS context in both directions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

MarcZX10 approved these changes Oct 21, 2026

View reviewed changes

This was referenced Oct 21, 2026

feat(libSceCesCs): add UTF-16 and UTF-32 forms of the SBC and MBCS string conversions #8972

Merged

feat(libSceCesCs): convert GBK and EUC-CN through the CP936 table #8976

Merged

github-actions Bot added docs prx needs rebase Conflicts with main, rebase needed labels Oct 21, 2026

jacob6769 mentioned this pull request Oct 21, 2026

feat(libSceCesCs): implement the EUC-KR, UHC and Big5 scalar conversions #9107

Merged

12 tasks done

rooklz mentioned this pull request Oct 21, 2026

feat(libSceCesCs): encode UCS-2, UTF-8, UTF-16 and UTF-32 characters into any MBCS profile #9175

Merged

12 tasks done

nikowolf123 force-pushed the feat/cescs-sbc-profiles branch from 0b3f339 to 27c3f6b Compare October 21, 2026 15:10

rooklz mentioned this pull request Oct 21, 2026

feat(libSceCesCs): add the strict JIS X 0208 EUC-JP and Shift-JIS profiles #9204

Merged

12 tasks done

github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 21, 2026

rooklz mentioned this pull request Oct 21, 2026

feat(libSceCesCs): add the GB18030-2000 and GB18030-2005 profiles #9210

Merged

12 tasks done

oneandonlydean self-assigned this Oct 22, 2026

SkillzTech1 mentioned this pull request Oct 22, 2026

feat(libSceCesCs): add the Windows-1253, 1255 and 1257 profiles #9912

Open

12 tasks done

This was referenced Oct 23, 2026

feat(libs): implement 23 exports with focused behavioral tests #9963

Closed

feat(libressl3): report OpenSSL version strings #9970

Open

Merge main into feat/cescs-sbc-profiles

b8bb173

github-actions Bot added needs rebase Conflicts with main, rebase needed and removed needs rebase Conflicts with main, rebase needed labels Oct 23, 2026

oneandonlydean merged commit 4d57821 into boykopovar:main Oct 23, 2026
10 checks passed

rooklz mentioned this pull request Oct 24, 2026

feat(libSceCesCs): add the ISO 646 national variants and JIS X 0201 profiles #10430

Merged

12 tasks done

Navigation Menu

Repository navigation

feat(libSceCesCs): add 27 single-byte code page profiles - #8923

feat(libSceCesCs): add 27 single-byte code page profiles#8923
oneandonlydean merged 2 commits into
boykopovar:mainfrom
nikowolf123:feat/cescs-sbc-profiles

nikowolf123 commented Oct 21, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

MarcZX10 left a comment

Uh oh!

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Uh oh!

ycbuzi commented Oct 21, 2026

Uh oh!

rooklz commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

nikowolf123 commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Tested

Checklist

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress report (316b742 → b8bb173)

Uh oh!

MarcZX10 left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Oct 21, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ycbuzi commented Oct 21, 2026

Uh oh!

rooklz commented Oct 21, 2026

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

nikowolf123 commented Oct 21, 2026 •

edited

Loading

github-actions Bot commented Oct 21, 2026 •

edited

Loading

Progress report (`316b742` → `b8bb173`)

github-actions Bot commented Oct 21, 2026 •

edited

Loading