Skip to content

feat(libSceCesCs): add the ISO 646 national variants and JIS X 0201 profiles - #10430

Merged
oneandonlydean merged 2 commits into
boykopovar:mainfrom
rooklz:feat/cescs-iso646
Oct 28, 2026
Merged

oneandonlydean merged 2 commits into
boykopovar:mainfrom
rooklz:feat/cescs-iso646

Conversation

@rooklz

@rooklz rooklz commented Oct 24, 2026 •

Copy link
Copy Markdown
Contributor

What

Twelve single-byte profile getters that still threw now return profiles: sceCesRefersUcsProfileAscii, AsciiWithKatakana, Bs4730, GbT1988, GbT1988Tilde0x7e, JisX0201, JisX0201Katakana, JisX0201Roman, JisX0201RomanTilde0x7e, JisX0201Tilde0x7e, KsX1003 and KsX1003Tilde0x7e. They work with every existing single-byte scalar converter (sceCesSbcTo*, sceCes*ToSbc) and with the MBCS string and length functions through sceCesMbcsUcsContextInit, as the 27 profiles of #8923 do.

The tables are measured on a PS5. A homebrew payload that loads the console's libSceCesCs.sprx called sceCesSbcToUtf32 on every byte and sceCesUtf32ToSbc on every BMP scalar for these twelve profiles and the 30 already on main, and each of the twelve now hashes the same as the console's sweep (guest_ces_iso646 pins the hashes). On the console:

  • The 7-bit sets (ASCII, BS 4730, GB 1988-80 and its tilde variant, JIS X 0201 Roman and its tilde variant, JIS X 0201 Katakana, KS X 1003 and its tilde variant) answer CES_ERROR_INVALID_ENCODE for bytes 0x80-0xFF, not UNASSIGNED_CODE.
  • The changed positions: BS 4730 changes only 0x23 (£); 0x7E stays a tilde. GB 1988-80 puts ¥ at 0x24, JIS X 0201 Roman ¥ at 0x5C, KS X 1003 ₩ at 0x5C, and the base GB, JIS and KS sets put an overline at 0x7E, which the Tilde0x7e variants keep as ~. A reassigned ASCII character has no byte.
  • JIS X 0201 Katakana is a 7-bit set: 0x21-0x5F are U+FF61-U+FF9F and 0x60-0x7E are unassigned; the 8-bit profiles (JisX0201, JisX0201Tilde0x7e, AsciiWithKatakana) carry the katakana at 0xA1-0xDF.

SingleByte.hpp gains a two-entry low-half override list per table, an Unassigned marker (0xFFFF) for bytes a set does not define, and a 7-bit flag; sbcToUnicode reports an unassigned byte and the converters answer INVALID_ENCODE for a high byte of a 7-bit set.

Getter signatures follow the Orbital type database. The other single-byte forms (sceCesSbcToUtf8, ToUcs2, the string functions) share the tables and were not measured separately. The same sweep found the surrogate answer 0x805C0015 for every single-byte profile and the C1 holes of Cp1254 and Cp1258 on main; those are other changes (#10183 has the Cp125x ones).

Tested

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: n/a
  • AI-assisted: yes

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Progress report (b7105f7 → 0a1ee9f)

📈 System libraries: 49.45% (+0.1%, +12 implemented functions)

✅ 12 implemented
Library Name
libSceCesCs sceCesRefersUcsProfileAscii
libSceCesCs sceCesRefersUcsProfileAsciiWithKatakana
libSceCesCs sceCesRefersUcsProfileBs4730
libSceCesCs sceCesRefersUcsProfileGbT1988
libSceCesCs sceCesRefersUcsProfileGbT1988Tilde0x7e
libSceCesCs sceCesRefersUcsProfileJisX0201
libSceCesCs sceCesRefersUcsProfileJisX0201Katakana
libSceCesCs sceCesRefersUcsProfileJisX0201Roman
libSceCesCs sceCesRefersUcsProfileJisX0201RomanTilde0x7e
libSceCesCs sceCesRefersUcsProfileJisX0201Tilde0x7e
libSceCesCs sceCesRefersUcsProfileKsX1003
libSceCesCs sceCesRefersUcsProfileKsX1003Tilde0x7e

Changed libraries (implemented exports)

Library Before After Change
libSceCesCs 228/287 (79.44%) 240/287 (83.62%) +4.18 pp

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head cb3277e. The profile tables and converter paths consistently handle the low-byte overrides and unassigned high-byte sentinels, and the added tests exercise scalar and MBCS round trips across the profiles. I independently confirmed the JIS X 0201 Roman/katakana mappings against Unicode's JIS0201 mapping and ran C++20 syntax checks on Export.cpp and GuestCesIso646.cpp; both pass. The author reports the full focused Linux and Windows test runs in the PR description. The current hosted run passes conventions, eligibility, intake, labels and report; platform builds are skipped for these paths. I could not run the product tests in this environment because the SDL2 submodule and XRandR development dependency are unavailable. I found no blocking issue at this head.

@github-actions

github-actions Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 10056 core/libs/prx/libSceCesCs/Export.cpp L92
core/libs/prx/libSceCesCs/PendingExports.cpp L151-230
core/libs/prx/libSceCesCs/include/Api.hpp L240-255
docs/dev/debt/libSceCesCs.md L26-27
PR 11614 core/libs/prx/libSceCesCs/Export.cpp L95-97
core/libs/prx/libSceCesCs/SingleByte.hpp L36, L39, L320
docs/dev/debt/libSceCesCs.md L26-27

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 9912 core/libs/prx/libSceCesCs/Export.cpp
core/libs/prx/libSceCesCs/PendingExports.cpp
core/libs/prx/libSceCesCs/SingleByte.hpp
docs/dev/debt/libSceCesCs.md

@rooklz

rooklz commented Oct 24, 2026

Copy link
Copy Markdown
Contributor Author

Reworked against a PS5 sweep of every single-byte ISO 646 profile: high bytes are now rejected with 0x805C0014, BS 4730 only remaps 0x23, and JIS X0201 Katakana is GL 0x21-0x5F. The test checks the console sweep hashes for all 12 profiles.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 25, 2026
@rooklz
rooklz force-pushed the feat/cescs-iso646 branch from 209cbf7 to 5cb7146 Compare October 25, 2026 01:21
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 25, 2026
@oneandonlydean oneandonlydean self-assigned this Oct 25, 2026
@oneandonlydean

Copy link
Copy Markdown
Collaborator

@rooklz At 5cb7146, remove the test-local APS5_VABI declarations and include the canonical libSceCesCs public header. The exact-head production audit fails on duplicate declarations.

Please run ctest -R public_headers, including the applicable public_headers_compile_* tests, after the header changes; rerun the affected guest tests.

@rooklz

rooklz commented Oct 25, 2026

Copy link
Copy Markdown
Contributor Author

@oneandonlydean done at da91d0c0c3. The test-local APS5_VABI declarations are removed and the test includes prx/libSceCesCs/include/Api.hpp, which already declares every function it uses. In a Linux x86-64 container (Ubuntu 24.04, GCC 13.3, Release), ctest -R '^guest_ces|^public_headers' passes: public_headers, every public_headers_compile_* batch, and all guest_ces* tests.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 25, 2026
@rooklz
rooklz force-pushed the feat/cescs-iso646 branch from da91d0c to b4ba125 Compare October 25, 2026 12:58
@github-actions github-actions Bot added needs rebase Conflicts with main, rebase needed and removed needs rebase Conflicts with main, rebase needed labels Oct 25, 2026
@rooklz
rooklz force-pushed the feat/cescs-iso646 branch from b4ba125 to 6913ef4 Compare October 25, 2026 23:27
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 25, 2026
@rooklz
rooklz force-pushed the feat/cescs-iso646 branch from 6913ef4 to feed556 Compare October 26, 2026 09:10
@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 26, 2026
@rooklz
rooklz force-pushed the feat/cescs-iso646 branch from feed556 to 9e275b8 Compare October 26, 2026 16:32
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 26, 2026
@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 27, 2026
@rooklz
rooklz force-pushed the feat/cescs-iso646 branch from 9e275b8 to 001a986 Compare October 27, 2026 05:47
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 27, 2026
@rooklz

rooklz commented Oct 27, 2026

Copy link
Copy Markdown
Contributor Author

@MarcZX10 could you re-review when you have a moment? Your approval at cb3277e predates the later branch changes. The branch is now exactly two commits: the feature you reviewed (restacked as 350daeee41) and the canonical-header fix dean requested across the stack (0a1ee9fd20, test-only: GuestCesIso646.cpp now includes prx/libSceCesCs/include/Api.hpp and drops the local APS5_VABI declarations, +1/-26). Since then only rebases and the SPIRV-Headers/SPIRV-Tools gitlink rollback dean flagged on #10428; the gitlinks match main's current tip 77f68f314b. Hosted eligibility/conventions/labels/report are green on 0a1ee9fd20.

@oneandonlydean
oneandonlydean merged commit cb63e03 into boykopovar:main Oct 28, 2026
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants