Repository navigation
feat(libSceCesCs): add the ISO 646 national variants and JIS X 0201 profiles - #10430
Conversation
Progress report (b7105f7 → 0a1ee9f)📈 System libraries: 49.45% (+0.1%, +12 implemented functions) ✅ 12 implemented
Changed libraries (implemented exports)
|
MarcZX10
left a comment
There was a problem hiding this comment.
Reviewed head cb3277e. The profile tables and converter paths consistently handle the low-byte overrides and unassigned high-byte sentinels, and the added tests exercise scalar and MBCS round trips across the profiles. I independently confirmed the JIS X 0201 Roman/katakana mappings against Unicode's JIS0201 mapping and ran C++20 syntax checks on Export.cpp and GuestCesIso646.cpp; both pass. The author reports the full focused Linux and Windows test runs in the PR description. The current hosted run passes conventions, eligibility, intake, labels and report; platform builds are skipped for these paths. I could not run the product tests in this environment because the SDL2 submodule and XRandR development dependency are unavailable. I found no blocking issue at this head.
|
Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
cb3277e to
209cbf7
Compare
|
Reworked against a PS5 sweep of every single-byte ISO 646 profile: high bytes are now rejected with 0x805C0014, BS 4730 only remaps 0x23, and JIS X0201 Katakana is GL 0x21-0x5F. The test checks the console sweep hashes for all 12 profiles. |
209cbf7 to
5cb7146
Compare
|
@rooklz At 5cb7146, remove the test-local APS5_VABI declarations and include the canonical libSceCesCs public header. The exact-head production audit fails on duplicate declarations. Please run ctest -R public_headers, including the applicable public_headers_compile_* tests, after the header changes; rerun the affected guest tests. |
|
@oneandonlydean done at |
da91d0c to
b4ba125
Compare
b4ba125 to
6913ef4
Compare
6913ef4 to
feed556
Compare
feed556 to
9e275b8
Compare
9e275b8 to
001a986
Compare
001a986 to
0a1ee9f
Compare
|
@MarcZX10 could you re-review when you have a moment? Your approval at |
What
Twelve single-byte profile getters that still threw now return profiles:
sceCesRefersUcsProfileAscii,AsciiWithKatakana,Bs4730,GbT1988,GbT1988Tilde0x7e,JisX0201,JisX0201Katakana,JisX0201Roman,JisX0201RomanTilde0x7e,JisX0201Tilde0x7e,KsX1003andKsX1003Tilde0x7e. They work with every existing single-byte scalar converter (sceCesSbcTo*,sceCes*ToSbc) and with the MBCS string and length functions throughsceCesMbcsUcsContextInit, as the 27 profiles of #8923 do.The tables are measured on a PS5. A homebrew payload that loads the console's
libSceCesCs.sprxcalledsceCesSbcToUtf32on every byte andsceCesUtf32ToSbcon every BMP scalar for these twelve profiles and the 30 already onmain, and each of the twelve now hashes the same as the console's sweep (guest_ces_iso646pins the hashes). On the console:CES_ERROR_INVALID_ENCODEfor bytes0x80-0xFF, notUNASSIGNED_CODE.0x23(£);0x7Estays a tilde. GB 1988-80 puts ¥ at0x24, JIS X 0201 Roman ¥ at0x5C, KS X 1003 ₩ at0x5C, and the base GB, JIS and KS sets put an overline at0x7E, which theTilde0x7evariants keep as~. A reassigned ASCII character has no byte.0x21-0x5Fare U+FF61-U+FF9F and0x60-0x7Eare unassigned; the 8-bit profiles (JisX0201,JisX0201Tilde0x7e,AsciiWithKatakana) carry the katakana at0xA1-0xDF.SingleByte.hppgains a two-entry low-half override list per table, anUnassignedmarker (0xFFFF) for bytes a set does not define, and a 7-bit flag;sbcToUnicodereports an unassigned byte and the converters answerINVALID_ENCODEfor a high byte of a 7-bit set.Getter signatures follow the Orbital type database. The other single-byte forms (
sceCesSbcToUtf8,ToUcs2, the string functions) share the tables and were not measured separately. The same sweep found the surrogate answer0x805C0015for every single-byte profile and the C1 holes of Cp1254 and Cp1258 onmain; those are other changes (#10183 has the Cp125x ones).Tested
209cbf758f(verify run on my fork; Ubuntu 24.04, GCC, Release, lavapipe): everyguest_ces*test passes; its failures (agc_driver_sgpr_pattern_fill_keys) aremain's on that runner. macOS (Apple clang 21, x86-64): everyguest_ces*source passes against thisExport.cpp, and a harness built from it matches the console dump for every input above.guest_ces_iso646(new, in both link variants): every byte of every new profile decodes to the expected character or answersCES_ERROR_UNASSIGNED_CODEthroughsceCesSbcToUtf8,sceCesSbcToUcs2andsceCesSbcToUtf32, and encodes back throughsceCesUtf8ToSbc,sceCesUcs2ToSbcandsceCesUtf32ToSbc; the reassigned positions refuse their ASCII character (sceCesUtf32ToSbc(0x5C, JisX0201Roman)is unassigned,0x00A5gives0x5C); the 63 katakana round-trip toU+FF61-U+FF9F; theTilde0x7evariants keep~and refuseU+203E; the fourteen profiles are distinct pointers; and an MBCS context on JIS X 0201 converts5C B6 41 7Eto¥カA‾and back and reports0xE0as unassigned.guest_ces,guest_ces_unicodeandguest_ces_mbcstill pass, so the 27 earlier profiles and CP1252 are unchanged.guest_ces_iso646,guest_ces_iso646_module,guest_ces,guest_ces_module,guest_ces_mbcandguest_ces_unicodepass, each binary run directly with its exit status checked.guest_ces*tests, the two new ones included; the Linux full build (Ubuntu 24.04, GCC, Release, lavapipe) compiles everything and passes everyguest_ces*test, its two failures beingagc_driver_queued_index_read_orderandagc_driver_sgpr_pattern_fill_keys, whichmainfails on that runner. An earlier run on38445f98db, the same code without the WindowsPATHentry for the new test, passed the Linux full build with everyguest_ces*test green andagc_driver_sgpr_pattern_fill_keysas its only failure, whichmainfails on that runner; its Windows job could not loadlibSceCesCs.dllfor the new test, which is why the entry is there.check_conventions.pypasses.pr_overlap.pyreports feat(ces): add eight sparse single-byte profiles #10056, fix(libSceCesCs): convert single-byte characters as measured on a PS5 #10183, fix(libSceCesCs): terminate and count MBCS string conversions as measured on a PS5 #10185 and fix(libSceCesCs): encode EUC-JP and Shift-JIS as measured on a PS5 #10186 on thesbcToUnicodelines and feat(libSceCesCs): add the code page 858 profile #10214 and feat(libSceCesCs): add the Windows-1253, 1255 and 1257 profiles #9912 on theSingleByte.hpptable list; none implements these twelve profiles (feat(libSceCesCs): add the Windows-1253, 1255 and 1257 profiles #9912 and feat(libSceCesCs): add the code page 858 profile #10214 add CP1253/1255/1257 and CP858, feat(ces): add eight sparse single-byte profiles #10056 eight sparse ISO 8859 and PC code pages). fix(libSceCesCs): convert single-byte characters as measured on a PS5 #10183 measuredsceCesSbcToUtf8andsceCesUtf8ToSbcon a PS5 and found undefined bytes answering0x805C0020in both directions, which is what theUnassignedentries here answer; it rewritessbcToUnicodeto the sameboolshape, so whichever lands second resolves one hunk to the same code. feat(libSceCesCs): encode UCS-2, UTF-8, UTF-16 and UTF-32 characters into any MBCS profile #9175 adds a debt entry two lines above this one's.Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)