[atharvjairath]

What

The eight profile getters for ISO-8859-3/6/7/8/11 and CP857/869/874 currently throw. They now return profiles usable by the existing scalar UTF-8/UCS-2/UTF-16/UTF-32 converters and MBCS string/length APIs. Undefined bytes return UNASSIGNED_CODE. UTF-8 scalar output/length handling and string termination/partial counts use the console-measured shared paths now merged into main. Reverse lookup excludes the undefined-entry marker, so U+FFFF cannot encode to a hole.

Mappings are regenerated from the Unicode-hosted ISO tables, Microsoft PC tables and CP874.TXT. Getter signatures follow the Orbital type database at b321fcd. CP874 uses the strict mapping, without Windows best-fit substitutions. Opaque profile layout, unmeasured error precedence and physical-console verification remain open in the library debt entry. The added mappings have the attributed firmware-module checks below; those checks do not validate string paths or every error contract.

tools/progress.py reports 8 additional completed exports (CES: 228 to 236 of 287). The scanner on the tested branch is unchanged, and the newer main scanner gives the same CES counts; this measures declared export coverage, not game compatibility. The final source has 51 pending exports, recomputed after main's JIS2004 initializers and measured GB18030 ranges landed. Upstream profile and encoder improvements are separate from this PR's eight profile getters.

The shared lookup rejects both the measured dependency's zero marker and the sparse profiles' U+FFFF marker. Tests cover all eight profiles across wide scalar and string APIs, including optional lengths, zero capacity, prefix preservation and U+FFFF reverse rejection.

Integration with current main: Rebased onto main 8f255109caeee67251202cb98598a517b712acf5 after #10739 (measured GB18030 ranges) merged. Only the Chinese fixture conflicted. Main's GB18030-2000 BMP-only range, GB18030-2005 plane-2 supplementary range, 24 preserved PUA mappings and measured A8BC/8135F437 swap are retained. The Chinese fixture now uses supported inputs while preserving partial counts, terminators, short-pair capacity, canaries and UCS-2 supplementary rejection. Its 2000 BMP case succeeds where the 2005 supplementary case rejects UCS-2. Our production/header patch payload and all other fixture payloads are unchanged. SingleByte.hpp, Cp936.hpp and the sparse fixture are byte-identical to the prior head; main's Gb18030.hpp, JIS X 0213 table and JIS2004 fixture are byte-identical to main. All earlier CP858, JIS2004, Big5, rejected-decoder, strict JIS, EUC-KR/UHC and generic encoder work is preserved. This refresh adds no exports beyond our original eight.

The sparse fixture now follows the merged rejected-decoder contract: an unassigned byte reports one consumed byte and zero output length, clears a UTF-32 scalar destination, and preserves adjacent canaries and UTF-8 buffer bytes. It also verifies the scalar result when the consumed-length pointer is omitted. These checks cover every undefined byte across all eight profiles. Those expectations were corrected during the earlier decoder integration and are unchanged in this refresh. The sparse fixture and production payload are unchanged; the Chinese integration assertions are adapted to the supported inputs above. No assertion or production check was disabled.

The sparse fixture now also checks the merged generic UTF-32 and UTF-8 encoders for every assigned byte in all eight profiles. It verifies the encoded byte, reported lengths, adjacent canaries, zero-capacity behavior and U+FFFF rejection. The existing coverage is retained, with rejected-decoder expectations updated as described above. Main's full generic-encoder sweeps and measured Chinese/DBCS/Japanese expectations are retained.

The public-header review remains addressed using canonical CES declarations and profile types. The eight getter placeholders have their actual const uint8_t* return type; the sparse fixture includes the umbrella header without local export declarations. Main's GB18030 tables, four-byte UTF-8 encoder, UTF-16 surrogate-pair adapters, UCS-2 supplementary rejection, isGb selector, measured decoder/string loop, strict versus measured JIS behavior and Korean/Big5 checks are preserved. The Chinese fixture retains the partial-count, termination, short-pair capacity and canary checks.

Landing order: No open prerequisites remain. The measured stack, #9175, #10718, #10722, #10738, #10214, #10426 and #10739 are integrated main history. CP858 is preserved as upstream work; it no longer needs to wait for this PR.

Tested

Exact tested head: 1ae7a83f3433710606d09c1c45419def687ebfd6; parent and main base: 8f255109caeee67251202cb98598a517b712acf5; tree: 4e9942e02c11d6045e78d4855aa51a611649ef83. Recovery ref: backup/pr-10056-before-main-refresh-20261027-1354 preserves prior remote head e635486ca84a28c046d7f7e9cc142ef06c19dcd0. Author and committer: atharvjairath <54663702+atharvjairath@users.noreply.github.com>.

• Linux x86-64, Ubuntu 24.04/GCC 13.3 Release under native ARM QEMU 11.0.3: 18/18 passed, 12.85 s total.
• Windows x64 PE, WinLibs GCC 15.2.0 UCRT r7 Release through Wine 9/QEMU 11.0.3: 18/18 passed, 66.28 s total.
• The complete real libc and both CES variants were rebuilt. Coverage includes sparse profiles, measured conversions, Japanese strict-profile round trips, GB18030/Chinese and Korean/Big5 conversions, Unicode, wide APIs, generic MBCS encoders/helpers and both new JIS2004 test variants. No assertion or production check was disabled and no new runtime workaround was used.
• Conventions after committing, whitespace checks and the full repository public-header audit pass (368 libraries, 12,810 exports). The CES umbrella compiles standalone, included twice, with Linux C++20 and -Werror; implementation and fixtures compile against canonical declarations on both tested platforms. Upstream test(libkernel): use the libkernel public header in the macOS unwind module info test #11718 is now included, so the earlier duplicate macOS unwind declaration failure is resolved on this tested head. No audit check was disabled.
• The exact overlap CLI scan and manual shared-file review found no independent duplicate of these eight getters. feat(libSceCesCs): add the Windows-1253, 1255 and 1257 profiles #9912 adds other single-byte profiles; feat(libSceCesCs): add the ISO 646 national variants and JIS X 0201 profiles #10430/feat(libSceCesCs): select the single-byte set of an MBCS profile and a replacement character for a context #10431 share profile/helper/API lines and add ISO646 profiles and setters. Other peers add ISO2022, ISO646, CP864, JIS-level helpers or measured mapping/error changes. Main includes fix(libSceCesCs): decode KS X 1001 cell 0xA2E8 as U+327E as the console does #10737, fix(libSceCesCs): map the strict JIS X 0208 profiles' unsettled cells as the console does #10718, fix(libSceCesCs): report the length of a rejected MBCS character as the console does #10722, fix(libSceCesCs): encode the Big5 box-drawing duplicates into their lower cells as the console does #10738, feat(libSceCesCs): add the code page 858 profile #10214, feat(libSceCesCs): add the EUC-JIS-2004 and Shift_JIS-2004 profiles #10426 and fix(libSceCesCs): map the GB 18030 profiles as the console does #10739; their measured mapping, decoder, Big5 encoding, JIS2004 and fixture changes are preserved. Current shared-file payloads were reviewed again. Comparisons against this PR's previous remote revision are not competing work. These unmerged peer trees were not imported or validated together with this branch.
• The merged fix(libSceCesCs): map the GB 18030 profiles as the console does #10739 cases are exercised by both Chinese fixture variants, including measured boundaries and the retained capacity/count/canary assertions described above.
• Earlier sparse-sentinel negative controls and the original GBK-selector diagnostic are historical evidence and were not rerun for this refresh.

Both complete matrices passed in this refresh. The interrupted Docker run from the prior JIS2004 refresh remains historical evidence in its original receipts.

Later main audit: Both remaining owned branches merge cleanly with 5d52b06ff3de0b2ce640dd296db465295c80e7fa. Later libc stream read-to-write handling and mutex-name wrapper declarations/fixtures, kernel memory/time changes and workflow test-path handling were inspected where relevant. No owned module or fixture changed after the exact runtime-tested base above. This scoped source/merge audit is not runtime validation of newer main or the whole project.

Focused builds use the complete real libraries and libc with repository-pinned dependencies. Native hardware, native Windows desktop, whole-project, title and PS5 console validation were not performed. The attributed firmware-module report below covers selected scalar paths; independent physical-console verification and unmeasured string/error-precedence paths remain open. Full upstream product CI requires maintainer dispatch.

Attributed firmware-module evidence: BunnyFl reports three identical runs of firmware 13.42 libSceCesCs.sprx (SHA-256 00c01529bc13223be8afabbde566fcbdc027a0366c6d6224effc86ba6cba3173) loaded natively on an x86-64 PC. Their SBC-to-UTF32/UTF8 sweep over all 256 bytes and UTF32-to-SBC sweep over U+0000–U+10FFFF matched all eight tables. Undefined cells returned 0x805C0020, had no reverse encoding, and U+FFFF was rejected. Reported hole counts were 7/45/3/36/8 for ISO-8859-3/6/7/8/11 and 3/9/31 for CP857/869/874. This is attributed firmware-module execution on a PC, not independently reproduced here or physical-console validation. It did not cover MBCS strings/lengths or general error precedence and is not an approval.

Focused build reproduction

Save the following as CMakeLists.txt in a separate harness directory. Initialize the checkout's 3rdparty/ZArchive and 3rdparty/zstd, and place copies or symlinks to those directories under the harness's 3rdparty/. They match the base pins 965b66c8d67b6b7e30fd63b3b75aa91a99ff303b and f8745da6ff1ad1e7bab384bd1f9d742439278e99.

cmake_minimum_required(VERSION 3.20)
project(CesValidation LANGUAGES CXX)
enable_testing()
set(CMAKE_CXX_STANDARD 20)
set(BUILD_TESTING ON)
set(ANYPS5_SOURCE_ROOT /src CACHE PATH "AnyPS5 source checkout")
set(LIBS_INCLUDE_DIR ${ANYPS5_SOURCE_ROOT}/core/libs)
set(ANYPS5_TEST_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/tests)
set(ANYPS5_WINDOWS_TEST_EXIT ${ANYPS5_SOURCE_ROOT}/cmake/WindowsTestExit.cpp)
add_compile_options(-ffunction-sections -fdata-sections)
add_link_options(-Wl,--gc-sections)
function(add_test_executable target)
    add_executable(${target} ${ARGN})
    set_target_properties(${target} PROPERTIES RUNTIME_OUTPUT_DIRECTORY "$<1:${ANYPS5_TEST_OUTPUT_DIRECTORY}>")
    if(MINGW)
        target_sources(${target} PRIVATE "${ANYPS5_WINDOWS_TEST_EXIT}")
        target_link_options(${target} PRIVATE LINKER:--wrap=main)
    endif()
endfunction()
function(add_asan_test_options test options)
    if(ANYPS5_ENABLE_ASAN)
        set_property(TEST ${test} APPEND PROPERTY ENVIRONMENT_MODIFICATION "ASAN_OPTIONS=string_append::${options}")
    endif()
endfunction()
function(configure_windows_unwind target)
    if(MINGW)
        if(NOT EXISTS "${CMAKE_OBJCOPY}")
            message(FATAL_ERROR "Windows DWARF unwinding requires objcopy")
        endif()
        add_custom_command(TARGET ${target} POST_BUILD
                COMMAND "${CMAKE_OBJCOPY}" --rename-section .eh_frame=.ehfram --rename-section .eh_fram=.ehfram "$<TARGET_FILE:${target}>"
                VERBATIM
        )
    endif()
endfunction()
if(MINGW)
    add_compile_options(-fno-asynchronous-unwind-tables)
endif()
include(${ANYPS5_SOURCE_ROOT}/cmake/ZArchive.cmake)
set(BUILD_TESTING OFF)
add_library(libc SHARED)
add_subdirectory(${ANYPS5_SOURCE_ROOT}/core/libs/prx/libc libc)
configure_windows_unwind(libc)
if(WIN32)
    target_compile_options(libc PRIVATE "SHELL:-include ${LIBS_INCLUDE_DIR}/prx/libc/include/HostStandardStreams.hpp")
endif()
if(CMAKE_SYSTEM_NAME STREQUAL "Linux")
    target_link_options(libc PRIVATE LINKER:-z,now)
    target_link_options(libc INTERFACE LINKER:--no-as-needed)
endif()
set(BUILD_TESTING ON)
add_subdirectory(${ANYPS5_SOURCE_ROOT}/core/libs/prx/libSceCesCs-module ces-module)
add_subdirectory(${ANYPS5_SOURCE_ROOT}/core/libs/prx/libSceCesCs ces)
foreach(target IN ITEMS libSceCesCs libSceCesCs-module)
    target_link_libraries(${target} PUBLIC libc)
    if(WIN32)
        target_compile_options(${target} PRIVATE "SHELL:-include ${LIBS_INCLUDE_DIR}/prx/libc/include/HostStandardStreams.hpp")
    endif()
endforeach()

On an x86-64 Linux host:

cmake -S /path/to/harness -B /path/to/ces-build -G Ninja -DANYPS5_SOURCE_ROOT=/path/to/checkout -DCMAKE_BUILD_TYPE=Release
cmake --build /path/to/ces-build --parallel 4
ctest --test-dir /path/to/ces-build -R '^guest_ces' --output-on-failure --timeout 30

The reported ARM-host Linux run additionally sets CMAKE_CROSSCOMPILING_EMULATOR to QEMU 11.0.3's native ARM qemu-x86_64. The Windows build uses the same harness with WinLibs cmake.exe, ninja.exe, gcc.exe and g++.exe; all eighteen generated test programs are run through QEMU/Wine. WINELOADERNOEXEC=1 preserves QEMU execution, and WINEPATH includes the libc, CES, CES-module and compiler-runtime DLL directories. A host-side CTest manifest runs each executable with the same 30-second timeout.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[github-actions]

Progress report (f8e3c7f → 1ae7a83)

📈 System libraries: 48.89% (+0.06%, +8 implemented functions)

✅ 8 implemented

Library	Name
libSceCesCs	sceCesRefersUcsProfileCp857
libSceCesCs	sceCesRefersUcsProfileCp869
libSceCesCs	sceCesRefersUcsProfileCp874
libSceCesCs	sceCesRefersUcsProfileIso8859_11
libSceCesCs	sceCesRefersUcsProfileIso8859_3
libSceCesCs	sceCesRefersUcsProfileIso8859_6
libSceCesCs	sceCesRefersUcsProfileIso8859_7
libSceCesCs	sceCesRefersUcsProfileIso8859_8

Changed libraries (implemented exports)

Library	Before	After	Change
libSceCesCs	228/287 (79.44%)	236/287 (82.23%)	+2.79 pp

[github-actions]

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request	Conflicting files (lines on main)	Same exports	Same new files
PR 10428	docs/dev/debt/libSceCesCs.md L23-24, L26
PR 10430	core/libs/prx/libSceCesCs/Export.cpp L92 core/libs/prx/libSceCesCs/PendingExports.cpp L151-230 core/libs/prx/libSceCesCs/include/Api.hpp L240-255 docs/dev/debt/libSceCesCs.md L26-27
PR 10431	core/libs/prx/libSceCesCs/Export.cpp L92 core/libs/prx/libSceCesCs/PendingExports.cpp L151-235 core/libs/prx/libSceCesCs/include/Api.hpp L240-256 docs/dev/debt/libSceCesCs.md L26-27
PR 10751	docs/dev/debt/libSceCesCs.md L23-24, L26
PR 11614	core/libs/prx/libSceCesCs/PendingExports.cpp L141-160 core/libs/prx/libSceCesCs/SingleByte.hpp L39 core/libs/prx/libSceCesCs/include/Api.hpp L238-241 docs/dev/debt/libSceCesCs.md L26

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 9912	core/libs/prx/libSceCesCs/Export.cpp core/libs/prx/libSceCesCs/PendingExports.cpp core/libs/prx/libSceCesCs/SingleByte.hpp docs/dev/debt/libSceCesCs.md

[fogydes]

OK. Independent checks, including a byte-level comparison against the upstream mapping tables and a recompute of the test's own constants:

1. Upstream parity: I fetched unicode.org's CP857.TXT and parsed the branch's SingleByte.hpp HIGH array directly (35 tables × 128 entries). Table index 5 (Cp857 per the enum order) matches the upstream mapping 128/128 entries, byte-for-byte, with undefined bytes exactly where upstream has none (0xD5, 0xE7, 0xF2). [verified]
2. Test constants are not invented: recomputing the FNV-1a checksum over the profile's promised behavior (identity low half, table-driven high half, length+UTF-8 bytes hashed, undefined bytes skipped) reproduces the Cp857 fixture's checksum 0x7025402330A56CAD and its undefined mask 0x0004008000200000 exactly from the branch's table — so the test would catch a wrong table, not just a wrong constant. [verified]
3. Registration: guest_ces_sparse_profiles and guest_ces_sparse_profiles_module are added in libSceCesCs/CMakeLists.txt in the same per-variant pattern as the existing tests. [verified]
4. Error-path surface is consistent with the PR's claims: reverse lookup skips the 0xFFFF marker, decodeMbcs returns 0x805C0020 (unassigned) without consuming, and sceSbcToUtf8 returns the same error before touching the caller's output buffer (the test asserts 0xA5 sentinels survive). [verified from Export.cpp]
5. The eight getter signatures were cross-checked against the cited Orbital export list — all present with matching return/specifier shape. [verified against the cited source]
6. Debt entry exists with the required unverified-behavior caveats, and the still-throwing profiles (Cp864 + private-use) remain on NotImplemented_nid_no_patch rather than gaining made-up data. [verified]
7. The Windows/Linux ctest pass counts and the Wine/QEMU PE runs are [author-reported only] — not reproducible here; the defining evidence is the table parity and the checksums in (1)/(2), which are local.

Envelope: current main is 664648356c; base is ef0d8ae3 — the commit the PR description itself cites (1 ahead, 7 behind); no main commits since that base touch libSceCesCs, the test file, or the debt entry; merge-tree against origin/main is clean.

Disclosure: Windows host; upstream file downloaded from unicode.org during review; checksum/mask recomputation scripted against the branch tree. The console-side mapping truth (as opposed to the Microsoft CP857 source of truth) remains unverified, which the debt entry already states.

[MarcZX10]

Reviewed exact head 93050a9a0c1bea1dd2d1db32c8b7af717592ce9e29.

[MarcZX10]

Correction and complete review on exact head 93050a9a0c1bea1dd2d1db32c7e3eda86a9be8: my immediately preceding review had a mistyped head hash and did not state the finding; this review replaces it. The sparse-profile implementation has broad coverage and the description reports exact-head Linux 16/16 and Windows 16/16 focused runs, plus baseline regression controls. Hosted eligibility, conventions, labels and report checks pass; product jobs are path-skipped. I have not independently run the CES test suite.

A contribution-rule sequencing issue remains. Direct merge previews of this PR with #10183, #10185 and #10186 produce content conflicts in core/libs/prx/libSceCesCs/Export.cpp; the #10056/#10184 preview is clean. These are real shared-helper conflicts, not duplicate implementations of the profile getters. Please coordinate with the affected author and declare the agreed landing order with Depends on, then ensure the rebased second change preserves both the sparse-profile semantics and the console-measured conversion behavior. #10185 already depends on #10183 and #10184, and #10186 depends on #10185, so please keep that stack coherent. This is a keep-both integration request; no PR should be closed because of queue size.

[atharvjairath]

@orezraey, I propose landing #10183/#10184 → #10185 → #10186 before #10056. I’m preparing the sparse-profile follow-up on that stack and preserving the measured conversion behavior. Does that order work for you?

[orezraey]

@atharvjairath Yes, that order works: #10183/#10184, then #10185, then #10186, then #10056. I'll keep the stack's Depends on lines as they are and note the agreement on #10183.

[MarcZX10]

Approved exact head b4670f6e8a915b3f5715e0885102109e83ecc5d4. The author-to-author landing order is now explicit: #10183/#10184, then #10185, then #10186, then this PR; the checklist declares Depends on: #10186, and this branch contains the exact current #10186 head. The sparse follow-up rejects undefined-byte sentinels in both decode and reverse lookup, keeps the shared measured conversion behavior, and adds broad profile/conversion/canary coverage. I independently compiled and ran the exact-head Export.cpp plus GuestCesSparseProfiles.cpp fixture with GCC C++20 and section garbage collection; it passed. Exact-head conventions and git diff --check pass, and current upstream main is an ancestor of the reviewed head. The author separately reports 16/16 Linux and 16/16 Windows focused CES runs with negative controls; hosted eligibility/conventions/labels/report checks pass and product build jobs are skipped by path policy. The full project build and console validation were not run here; the debt entry accurately leaves sparse mappings unverified on console.

[oneandonlydean]

@atharvjairath This PR now conflicts with main in core/libs/tests/GuestCes.cpp, core/libs/tests/GuestCesChinese.cpp. Main’s corrected Ces profile types and public-header migration overlap this PR’s sparse-profile fixtures, whose helpers retain the older profile pointer type. Main now requires each exported function to be declared once, in the library's public header (checked by the public_headers tests). Could you reconcile the profile helpers and test loops with the current public interfaces?

[BunnyFl]

I checked the eight profiles against firmware 13.42's own libSceCesCs.sprx (sha256 00c01529bc13223be8afabbde566fcbdc027a0366c6d6224effc86ba6cba3173). The module was loaded natively on x86-64 and called directly, with the profile each 13.42 getter returns: sceCesSbcToUtf32 and sceCesSbcToUtf8 on every byte 0x00-0xFF, and sceCesUtf32ToSbc on every scalar 0-0x10FFFF.

All eight match this head's SingleByte.hpp byte for byte (256 of 256 each), including every Unassigned entry: those bytes answer 0x805C0020 on 13.42, and no scalar encodes to them. Unassigned bytes per profile: ISO-8859-3 7, -6 45, -7 3, -8 36, -11 8, CP857 3, CP869 9, CP874 31. Each profile encodes exactly the scalars it decodes, back to the same byte; U+FFFF answers 0x805C0020 with all eight. The UTF-8 and UTF-32 decoders give the same result on every byte.

Three runs, identical output. Not checked: the MBCS string and length functions, and which error wins when several apply.