Repository navigation
feat(libSceCesCs): implement the GB four-byte range and user-defined area setters - #13573
KernelPanicNT wants to merge 4 commits into
Conversation
…area setters sceCesGbUcsProfileSet4ByteCharRange accepts ranges 0, 1, 2, 3, 5 and 7 on the GB 18030 profiles and sceCesGbUcsProfileSetUdaMapping accepts 0 and 1 on the GB 18030 and GBK profiles, as measured as a payload process on a device. Other values return 0x805C0001; a null, GBK (range) or EUC-CN profile returns 0x805C0004. The setting is kept in the sheet and read by the GB conversions: ranges 0 and 2 drop the four-byte BMP forms, 5 adds U+10000 and 7 adds U+10FFFF; mapping 0 makes the GB 18030 user-defined cells unassigned and mapping 1 lets GBK decode the F8A1-FEFE and A140-A7A0 areas. Supplementary characters the measurement did not cover throw for a non-default range.
|
Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.
|
Progress report (0263899 → 5818cdd)📈 System libraries: 26.7% (+0.01%, +2 implemented functions) ✅ 2 implemented
Changed libraries (implemented exports)
|
|
@KernelPanicNT Reviewed at |
|
Verified Build
Tests (all at the PR head)
Code reading
Open
|
adriadam10
left a comment
adriadam10
left a comment
There was a problem hiding this comment.
Reviewed at f785292f5d: the setters, the range table, the sheet bytes 8/9, the decoder/encoder changes, the throwing paths and the debt entries match the description, and the test pins the measured hashes (it aborts with the setters as stubs).
One thing left, the one @oneandonlydean asked for above and not yet done:
- State the firmware version of the payload measurement in the PR body and in the
sceCesGbUcsProfileSet4ByteCharRange and sceCesGbUcsProfileSetUdaMappingentry ofdocs/dev/debt/libSceCesCs.md(line 18), asdocs/dev/REVERSE_ENGINEERING.md("Sources, strongest first") requires for a console measurement. Other entries in that file say it as "(firmware 13.60)".
With that added this is good to merge.
…-setters # Conflicts: # docs/dev/debt/libSceCesCs.md
The sceCesGbUcsProfileSet4ByteCharRange and sceCesGbUcsProfileSetUdaMapping entry now states that the ranges, mappings and conversion results come from a payload process on a device (firmware 13.00).
|
Thanks @oneandonlydean for pointing to "Sources, strongest first" and asking for the firmware, and @adriadam10 for the careful review and for spelling out exactly where it was missing. Done in 365d547: the The conflict with main is resolved: only the pending-export count in the debt file clashed, and it now reads 15 (main's 17 minus these two). The setter tests and the rest of Thanks also @blawesom for the independent run and code read. |
What
Source of the behaviour
Measured with a homebrew probe running as a payload process on a device (firmware 13.00). The probe loaded the system
libSceCesCs.sprx, built the four GB profiles in its own guarded static sheets, called each setter on each profile with 17 values (0-8, 15, 16, 0x100, 0x10000, INT_MAX, -1, -2, INT_MIN) after restoring the sheet byte for byte, and recorded the result and every changed sheet byte. For each accepted value 0-8 that left a distinct sheet, it then ransceCesGbToUtf32on 28 fixed byte sequences andsceCesUtf32ToGbon 26 fixed code points and logged result, size and output. No guard, sheet or buffer outside the expected one changed and no call faulted.Set4ByteCharRange, GB 18030-2000 and -2005 sheets, values 0, 1, 2, 3, 5, 70x805C0001, sheet unchangedSet4ByteCharRangeon GBK and EUC-CN sheets (all 17 values), null profile (0 and 1)0x805C0004, sheet unchangedSetUdaMapping, GB 18030 and GBK sheets, 0 and 10x805C0001, sheet unchangedSetUdaMappingon EUC-CN (all 17 values), null profile (0 and 1)0x805C0004, sheet unchanged81308130,8135F437,8431A4390x805C0020, size 40x805C0020, size 090308130and U+10000 convert (withgbMax2 or 3,0x805C0031and size 4);E3329A35and U+10FFFF0x805C0020E3329A35and U+10FFFF convert8431A530,E3329A36,FE39FE390x805C0020, size 4, with every range81,8130,813081,81308130withgbMax2,817F,80,FF), two-byte cells outside the user-defined areasAAA1,AFFE,F8A1,FEFE,A140,A7A0; U+E000, U+E233, U+E234, U+E4C5, U+E4C6, U+E5E50x805C0020(size 2 decoding, 0 encoding);A6D9U+E78D andFE51U+E816 still convertA140U+E4C6,A7A0U+E765,F8A1U+E234,FEFEU+E4C5;AAA1andAFFEstay0x805C0020; the six code points above still encode to0x805C0020A140-A7A0andF8A1-FEFEareas; encoding unchangedPendingExports.cpp#L21,#L31; declarationsApi.hpp#L214,#L216.decodeGb(GBK),decodeGb18030,encodeGb18030; the default supplementary mapping kept as is:mapsGb18030Supplementary; the GBK user-defined values come fromgb18030Cell; the inits that now also write the two setting bytes:Export.cpp#L2440-L2456.libSceCesCs.md#L16,#L17,#L29.int sceCesGbUcsProfileSet4ByteCharRange(union*p, int gb4byteRange),int sceCesGbUcsProfileSetUdaMapping(union*p, int udaMapping)); the probe's conversions are#L206and#L452.REVERSE_ENGINEERING.md#L9.Implements
sceCesGbUcsProfileSet4ByteCharRange(void* profile, int range)andsceCesGbUcsProfileSetUdaMapping(void* profile, int udaMapping). Both calledNotImplemented_nid_no_patch.The inits now write the edition's default range and mapping into sheet bytes 8 and 9, and the setters validate and store a new value there. The accepted ranges and what each does to the measured samples are one table in
Export.cpp; the GB 18030 decoder and encoder consult it for four-byte forms, and both GB 18030 directions and the GBK decoder consult the mapping for the user-defined areas. With the default range the supplementary mapping is unchanged (none for 2000, plane 2 for 2005). With any other range, U+10000 and U+10FFFF follow the measurement and every other supplementary character throwsstd::runtime_error, because the probe did not reach plane 2 or the middle of the range (ranges 0 and 2 answer alike on every sample, so what 2 adds is not known). What is generalised from the samples is listed in a newdocs/dev/debt/libSceCesCs.mdentry: the whole four-byte BMP range, every cell of an area, values outside the 17 tried, the other conversion forms (ToUcs2,ToUtf8,ToUtf16, the string conversions), which share the decoder and encoder, and the two setters combined. The GBK entry now names onlysceCesGbUcsProfileSetSbcsas unhandled, the GB 18030 entry drops the sentence about these two throwing, and the pending-export count goes from 17 to 15.Open questions and overlap
0x80-0xA0as single bytes (80U+0080,81308130U+0081 with size 1,817FU+0081); here those leads answer0x805C0014. The setters reject EUC-CN, so this PR does not change it.SetSbcsselection in the tag's eighth byte; it leaves these two exports as stubs. This PR stores its settings after the tag (bytes 8 and 9), so the two designs fit together, but the init functions,PendingExports.cpp,Api.hppand the debt file conflict textually.sceCesUcsProfileInitGb18030_2022next to these stubs (still declaration-only); #12946 edits the same CMake variant list; #9912, #11614 and #15409 remove other pending stubs and change the same pending-count line, which #14085 and #14212 also rewrite.tools/pr_overlap.py --branch feat/cescs-gb-profile-setters: no open PR implements these exports. #9912, #11614, #12946, #13486, #14212 and #15409 conflict withmain, so they were compared by hand; none adds these two functions.Tested
Linux (GCC). The new test runs as
guest_ces_gbsettersandguest_ces_gbsetters_module(linked againstlibSceCesCsandlibSceCesCs-module). It calls both setters with the 17 values on the four profiles and a null profile and checks every result, and that the sheet changes exactly when the device's did. For the 16 measured sheet states it runs the 28 decode and 26 encode samples and compares an FNV-1a hash of (result, size, output) with the hash of the device results; it also checks samples from the table and that the unmeasured supplementary characters and a Shift-JIS profile throw. All 24guest_cestests pass onmain(b11c7bf84f) with this branch applied. The new tests fail with the two setters reduced toNotImplemented_nid_no_patchstubs (both abort, checked at this head); atf785292f5dthey also failed withExport.cpp,PendingExports.cppandApi.hppfrommain(they do not build) and with setters that store the value while the conversions ignore it (both abort, whileguest_ces_chineseandguest_ces_mbcstill pass).tools/progress.py: +2 implemented functions (libSceCesCs 270/287 -> 272/287).tools/check_conventions.pypasses.Checklist
main; no other open PR implements the same functionsNotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file.mdfiles or images added to the repository (attach them to this PR)