[kanza-cm]

What

Two gaps in the guest wide scanf on Linux, both listed in the technical debt:

• %l[...] read its scanset byte by byte, so a member outside ASCII threw instead of matching. The scanset is now decoded as UTF-8 and compared as code points, which also turns a range such as à-ê into a real code point range.
• A code point outside the BMP threw. It is now stored as a UTF-16 surrogate pair: widths count characters, _s capacities count units, and a character that does not fit the destination leaves an empty result, as the capacity path already did.

The remaining throwing cases are unchanged: input that is not UTF-8, positional arguments next to a wide conversion, and wide conversions in fscanf. The debt entry no longer lists the two cases this fixes.

Tested

• Linux (Ubuntu 24.04 under WSL2, GCC 15.2.0, Release, -DBUILD_TESTING=ON): guest_formatting with the new cases, guest_wide_formatting, guest_wide_stream_scan and guest_scan_count pass.
• New cases in core/libs/tests/Formatting.cpp: %l[é], %l[à-ê] with a multibyte range, %l[^é], a mixed ASCII/non-ASCII member, %ls of an emoji, %2ls stopping after one character, and %ls into a 4-unit _s buffer where one emoji fits and two do not.
• Windows (WinLibs GCC 15.2.0, Release): libc still builds. This block is #ifndef _WIN32, so Windows CI does not exercise it; the Windows path hands %ls, %lc and %l[ to the host CRT.
• tools/check_conventions.py --base origin/main passes.
• Not tested: a title that scans a non-ASCII scanset or a supplementary character; none is known to.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Fixes: none
• Unblocks: none
• GPUs tested: n/a
• AI-assisted: yes

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 3965	docs/dev/debt/libc.md
PR 5021	core/libs/tests/Formatting.cpp docs/dev/debt/libc.md
PR 5303	docs/dev/debt/libc.md
PR 6819	docs/dev/debt/libc.md
PR 7081	docs/dev/debt/libc.md
PR 7386	core/libs/prx/libc/src/Formatting.cpp core/libs/tests/Formatting.cpp docs/dev/debt/libc.md
PR 7447	docs/dev/debt/libc.md
PR 7544	docs/dev/debt/libc.md
PR 7648	docs/dev/debt/libc.md
PR 7715	docs/dev/debt/libc.md
PR 7851	docs/dev/debt/libc.md
PR 8150	docs/dev/debt/libc.md
PR 8417	docs/dev/debt/libc.md
PR 8965	docs/dev/debt/libc.md
PR 9371	docs/dev/debt/libc.md
PR 9447	docs/dev/debt/libc.md
PR 9466	docs/dev/debt/libc.md
PR 9526	docs/dev/debt/libc.md
PR 9677	docs/dev/debt/libc.md
PR 9761	docs/dev/debt/libc.md
PR 9863	docs/dev/debt/libc.md
PR 9873	docs/dev/debt/libc.md
PR 9929	docs/dev/debt/libc.md
PR 9935	docs/dev/debt/libc.md
PR 9964	docs/dev/debt/libc.md
PR 10002	docs/dev/debt/libc.md
PR 10007	docs/dev/debt/libc.md
PR 10113	docs/dev/debt/libc.md
PR 10158	docs/dev/debt/libc.md
PR 10165	docs/dev/debt/libc.md
PR 10200	docs/dev/debt/libc.md
PR 10226	docs/dev/debt/libc.md
PR 10234	docs/dev/debt/libc.md
PR 10245	docs/dev/debt/libc.md
PR 10250	docs/dev/debt/libc.md
PR 10253	docs/dev/debt/libc.md
PR 10265	docs/dev/debt/libc.md
PR 10270	docs/dev/debt/libc.md
PR 10274	docs/dev/debt/libc.md
PR 10315	docs/dev/debt/libc.md
PR 10321	docs/dev/debt/libc.md
PR 10325	docs/dev/debt/libc.md
PR 10328	docs/dev/debt/libc.md
PR 10345	docs/dev/debt/libc.md
PR 10349	docs/dev/debt/libc.md
PR 10397	docs/dev/debt/libc.md
PR 10405	docs/dev/debt/libc.md
PR 10506	docs/dev/debt/libc.md
PR 10510	docs/dev/debt/libc.md
PR 10512	docs/dev/debt/libc.md
PR 10565	docs/dev/debt/libc.md
PR 10569	docs/dev/debt/libc.md
PR 10749	docs/dev/debt/libc.md
PR 10780	docs/dev/debt/libc.md
PR 10987	docs/dev/debt/libc.md
PR 11109	docs/dev/debt/libc.md
PR 11172	docs/dev/debt/libc.md
PR 11260	docs/dev/debt/libc.md
PR 11262	docs/dev/debt/libc.md
PR 11266	docs/dev/debt/libc.md
PR 11621	docs/dev/debt/libc.md
PR 11625	docs/dev/debt/libc.md
PR 11632	docs/dev/debt/libc.md
PR 11634	docs/dev/debt/libc.md
PR 11661	docs/dev/debt/libc.md
PR 11664	docs/dev/debt/libc.md
PR 11689	docs/dev/debt/libc.md
PR 11698	docs/dev/debt/libc.md
PR 11793	docs/dev/debt/libc.md
PR 11831	docs/dev/debt/libc.md
PR 11848	docs/dev/debt/libc.md
PR 11870	docs/dev/debt/libc.md
PR 11895	docs/dev/debt/libc.md
PR 11907	core/libs/prx/libc/src/Formatting.cpp core/libs/tests/Formatting.cpp docs/dev/debt/libc.md
PR 11947	docs/dev/debt/libc.md
PR 12053	docs/dev/debt/libc.md
PR 12057	docs/dev/debt/libc.md
PR 12122	docs/dev/debt/libc.md
PR 12149	docs/dev/debt/libc.md
PR 12184	docs/dev/debt/libc.md
PR 12271	docs/dev/debt/libc.md
PR 12292	docs/dev/debt/libc.md
PR 12397	core/libs/prx/libc/src/Formatting.cpp core/libs/tests/Formatting.cpp docs/dev/debt/libc.md
PR 12507	core/libs/prx/libc/src/Formatting.cpp core/libs/tests/Formatting.cpp
PR 12510	docs/dev/debt/libc.md
PR 12511	docs/dev/debt/libc.md
PR 12518	docs/dev/debt/libc.md
PR 12530	core/libs/prx/libc/src/Formatting.cpp core/libs/tests/Formatting.cpp docs/dev/debt/libc.md
PR 12532	core/libs/prx/libc/src/Formatting.cpp core/libs/tests/Formatting.cpp docs/dev/debt/libc.md
PR 12560	docs/dev/debt/libc.md
PR 12574	docs/dev/debt/libc.md
PR 12592	docs/dev/debt/libc.md
PR 12593	docs/dev/debt/libc.md
PR 12594	docs/dev/debt/libc.md
PR 12595	docs/dev/debt/libc.md
PR 12598	docs/dev/debt/libc.md
PR 12605	docs/dev/debt/libc.md
PR 12627	docs/dev/debt/libc.md
PR 12682	docs/dev/debt/libc.md
PR 12688	docs/dev/debt/libc.md
PR 12700	docs/dev/debt/libc.md
PR 12704	docs/dev/debt/libc.md
PR 12841	docs/dev/debt/libc.md
PR 12849	docs/dev/debt/libc.md
PR 12920	docs/dev/debt/libc.md
PR 12947	docs/dev/debt/libc.md
PR 12965	docs/dev/debt/libc.md
PR 13119	core/libs/prx/libc/src/Formatting.cpp core/libs/tests/Formatting.cpp docs/dev/debt/libc.md
PR 13186	docs/dev/debt/libc.md
PR 13390	docs/dev/debt/libc.md
PR 13449	docs/dev/debt/libc.md
PR 13784	docs/dev/debt/libc.md
PR 13793	docs/dev/debt/libc.md
PR 13794	docs/dev/debt/libc.md
PR 13804	docs/dev/debt/libc.md
PR 13809	docs/dev/debt/libc.md
PR 13828	docs/dev/debt/libc.md
PR 14020	docs/dev/debt/libc.md
PR 14085	docs/dev/debt/libc.md
PR 14212	docs/dev/debt/libc.md
PR 14235	docs/dev/debt/libc.md
PR 14298	docs/dev/debt/libc.md
PR 14326	docs/dev/debt/libc.md
PR 14585	docs/dev/debt/libc.md
PR 14588	docs/dev/debt/libc.md
PR 14595	docs/dev/debt/libc.md
PR 14610	docs/dev/debt/libc.md
PR 14619	docs/dev/debt/libc.md
PR 14620	docs/dev/debt/libc.md
PR 14623	docs/dev/debt/libc.md
PR 14828	docs/dev/debt/libc.md
PR 14864	docs/dev/debt/libc.md
PR 14878	docs/dev/debt/libc.md
PR 14915	docs/dev/debt/libc.md
PR 14955	docs/dev/debt/libc.md
PR 15004	docs/dev/debt/libc.md
PR 15095	docs/dev/debt/libc.md
PR 15098	docs/dev/debt/libc.md
PR 15099	docs/dev/debt/libc.md

[ycbuzi]

Windows run of dea162e merged onto main a6dcea9 (e774d96f1a, clean), the platform the description lists as not tested. Windows 11 Home 10.0.26200, MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release, BUILD_TESTING=ON.

Built libc and guest_formatting_tests with no errors or warnings. ctest --repeat until-fail:2 on guest_formatting: passes on both repeats. Not run: a title.

[wachi35676]

Linux run with a wider set of inputs. This is the only host where the change runs: the code is #ifndef _WIN32, so the Windows run on this PR builds it but doesn't execute it.

Head dea162e177 merged onto main a6dcea927 (clean). WSL2 Ubuntu 24.04, GCC 13.3.0, Release.

• guest_formatting, guest_wide_formatting, guest_wide_stream_scan, guest_scan_count: 4/4, 3 runs.
• Negative control: with ScanWideField.cpp, ScanWideField.hpp and Formatting.cpp from main, guest_formatting aborts (scanf 16-bit wide scanset with a character outside ASCII not implemented).
• check_conventions.py --base a6dcea927: passes.

Probe: sscanf/sscanf_s into a char16_t[8] pre-filled with 0xa5a5, showing the code units written. All of these threw on main:

• %l[ê-à] (reversed range) on é: 0, nothing written. %l[]é] on ]éx: 005d 00e9 0000. %l[^]é] on abé: 0061 0062 0000. %l[a-] on -a: 002d 0061 0000.
• %l[😀] on two emoji and x: d83d de00 d83d de00 0000. %l[à-😀] on U+FFFF 😀: ffff d83d de00 0000.
• %1ls and %1l[^x] on an emoji: d83d de00 0000 (width counts characters).
• sscanf_s %ls, capacity 2 on an emoji: 0 with 0000; capacity 3: d83d de00 0000; capacity 3 on a😀: 0 with 0000.
• %ls on U+10FFFF: dbff dfff 0000. UTF-8-encoded surrogate ED A0 80: still throws (not UTF-8).

%lc writes two units for one character. %lc on an emoji writes d83d de00, and %2lc on two emoji writes d83d de00 d83d de00. So a guest char16_t c; sscanf(s, "%lc", &c) gets one unit written past c, and a char16_t c[2] with %2lc gets two. sscanf_s is bounded (capacity 1 returns 0 and writes 0000). The plain forms have no capacity, so this is a write past the guest's object whenever the input is outside the BMP. Throwing for a non-BMP %c field in the non-_s forms, as before, would avoid that until the console's behaviour is known.

AI-assisted: yes (Claude Code); results above were run on this machine.

[kanza-cm]

Thanks for the probe, it found a real defect: an unsized %lc field has no capacity to bound, so a supplementary character wrote its second unit past the caller's object. Fixed in 9143eee: the unsized field throws for a character outside the BMP again, while the _s forms keep writing both units because their capacity is already checked in units. Two cases were added to the refusal test (%lc and %2lc on an emoji), the debt entry now records the unsized field, and guest_formatting was re-run on Windows and on Ubuntu 24.04 under WSL2 after the change. All of your other probe cases (%l[😀], reversed and negated ranges, %1ls, %ls on U+10FFFF, sscanf_s capacities) keep their results.

[kanza-cm]

Ready for the full run, whenever a collaborator has a moment for the manual dispatch:

gh workflow run build.yml --repo boykopovar/AnyPS5 --ref main --field pr=9340

State: MERGEABLE, conventions/intake/labels green, three commits on feat/libc-wide-scanset-utf8.

• Windows (GCC 15.2.0, BUILD_TESTING=ON): libc and guest_formatting_tests build clean; guest_formatting passes. Reported by @ycbuzi.
• Ubuntu 24.04 under WSL2 (GCC 13.3.0): guest_formatting, guest_wide_formatting, guest_wide_stream_scan and guest_scan_count pass, with a negative control on main. Reported by @wachi35676, whose probe also found the unsized %lc overflow now fixed in 9143eee65.
• tools/check_conventions.py --base origin/main passes.

Note for sequencing: the overlap comment lists #7386 and #9087 as touching core/libs/prx/libc/src/Formatting.cpp. This change does not build on either; if they land first, I will rebase.

[ycbuzi]

Windows run of 9143eee merged onto main 8b3a6ba (9c2962a5b9, clean). Windows 11 Home 10.0.26200, MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release, BUILD_TESTING=ON.

Built libc and the four test executables with no errors or warnings. ctest --repeat until-fail:2 on guest_formatting, guest_wide_formatting, guest_wide_stream_scan and guest_scan_count: all four pass on both repeats.

On the Windows line in the summary above: as @wachi35676 noted, the wide-scan path is inside #ifndef _WIN32 in Formatting.cpp, so on Windows this change is compiled out. A negative control confirms it here: with Formatting.cpp, ScanWideField.cpp and ScanWideField.hpp taken from main, the same four tests still pass on Windows. So my Windows runs show that this PR builds and keeps these tests passing on Windows, not that its change runs there; Linux is the host that exercises it. Not run: a title.

[kanza-cm]

Rebased onto main 11f976b27; the branch head is now 5c6e9e685. The rebase was clean, the Formatting.cpp changes that landed since were in the fopen and MSYS2 areas.

Re-run after the rebase: libc and the four executables build on Windows (GCC 15.2.0) and on Ubuntu 24.04 under WSL2; guest_formatting, guest_wide_formatting, guest_wide_stream_scan and guest_scan_count pass on Linux, and guest_formatting passes on Windows.

On @ycbuzi's negative control: thanks, that is the right framing and the description already says it — this block is compiled out on Windows, so the Windows run shows the build and the absence of a regression, not that the change runs there. Linux is the host that exercises it, and @wachi35676's run is the one that covers execution.

[oneandonlydean]

@Canzi18 Please remove the new scan tests requiring unsupported non-BMP unsized %lc/%2lc and positional forms to throw: the tests should not pin unimplemented behaviour. Retain the production safety guards, malformed-input validation, positive surrogate/scanset cases and explicit encoding limitations, then rerun the applicable formatting/scanning tests.

[harisibrar1]

Checked against main 39a36bb and FreeBSD 11.4 lib/libc/stdio/vfscanf.c / vfwscanf.c:

• It still merges cleanly onto main, and main still throws for both cases, so the change is still relevant.
• Width-limited string fields in the non-_s forms have the same problem the unsized %lc one had. wanted counts characters (Formatting.cpp:196/205) and limit is ULONG_MAX outside _s, but each non-BMP character appends 2 units. Reading the code, char16_t buf[3]; sscanf("😀😀", "%2ls", buf) writes d83d de00 d83d de00 0000, which is 5 units for a buffer sized width+1 (wachi35676's probe shows the same for %1ls: 3 units). On main this throws. FreeBSD's convert_wstring/convert_wccl store one wchar_t per character, so width+1 is always enough there; with a 16-bit wchar_t I don't know what the console does. Until that's known, would it work to throw for a non-BMP character in a width-limited non-_s %ls/%l[ field, as the %lc path already does? The same applies to %2l[...].
• Code-point scansets and ranges like à-ê aren't what either reference does. FreeBSD 11.4 vfscanf.c tests %l[ through a 256-entry byte table (__sccl, then ccltab[wctob(wi)]), and vfwscanf.c's inccl is a plain wmemchr with no ranges. Could the debt entry say that scanset members and ranges are compared as code points, unverified on the console?
• Minor: a reversed range (%l[ê-à], and z-a already on main) matches nothing, while FreeBSD's __sccl treats it as the literal members ê, -, à. Also, a suppressed %*lc throws on a non-BMP character even though nothing is written. Neither blocks the PR.

Reviewed with AI assistance (Claude Code); the checks above were run locally on my machine.

[oneandonlydean]

@Canzi18 I agree with the width-limited non-_s safety finding. In addition to my existing test-contract ask, please guard non-BMP input in width-limited %ls/%l[ until a safe measured 16-bit contract is established: %2ls currently accumulates four UTF16 units and writes a fifth terminator into storage callers size as width+1. Keep safe positive/scanset tests and sentinel checks without pinning an unsupported throw. Please also state that code-point scanset members/ranges are unverified and differ from the cited FreeBSD paths.

[wachi35676]

Linux re-run of head 3f2fa3b280 merged onto main b90d009d9f (clean), WSL Ubuntu 24.04, GCC 13.3, Release.

• guest_formatting, guest_wide_formatting, guest_wide_stream_scan and guest_scan_count pass 4/4, 3 runs. With the scan sources from my earlier head dea162e177, guest_formatting aborts, and with main's it aborts too.

• The unsized %lc overflow is fixed. %lc and %2lc on an emoji now throw before writing anything (the probe buffer stays a5a5), where dea162e177 wrote d83d de00 and d83d de00 d83d de00. %lc on é still gives 00e9, and the sscanf_s capacities behave as before.

• The width-limited non-_s forms still write past width + 1 units, as @harisibrar1 pointed out. Running them confirms it:

  • %1ls on 😀 writes d83d de00 0000, three units into storage sized for two.
  • %2ls on a😀b writes four units, and on 😀😀 five (d83d de00 d83d de00 0000). Both are sized for three.
  • %2l[^x] on 😀😀x writes five units.

  The new Require(... "%2ls" ... == u"a\U0001F600") pins that four-unit write. Guarding a non-BMP character in width-limited %ls/%l[ in the non-_s forms, as @oneandonlydean asked, would remove it.

AI-assisted: yes (Claude Code); results above were run on this machine.

[cocoa55]

I opened #11724, which lets fscanf/vfscanf/scanf (and the _s stream forms) read %ls/%lc/%l[ on Linux through the guest scanner. It changes the same wide-field loop in Formatting.cpp, and the same scanf entry in docs/dev/debt/libc.md, as this PR, so the two conflict. #9087 also touches that loop.

CONTRIBUTING asks us to agree on an order. I suggest #9087, then this PR, then #11724. I've marked #11724 as Depends on: #9087, #9340 and will rebase it after both merge, so you don't need to change anything. #11724 decodes each character before deciding whether the field takes it, so your surrogate pairs and non-ASCII scanset members will also work on streams after the rebase. I'll add a stream test for them when I rebase.

If you'd rather go the other way, tell me and I'll swap the order.

[JJenkx]

@Canzi18 measurements that bear on this PR's direction: titles' own libc.prx doesn't decode UTF-8 in the wide scanf conversions. I ran the sscanf of PPSA07923, PPSA05916 and PPSA13934 natively on Windows 11 (a scratch loader calls it by NID). All three agree.

Each byte goes to mbrtowc on its own (the C locale's MB_CUR_MAX is 1), so any byte ≥ 0x80 ends a %ls, %lc or %l[ conversion. That includes valid UTF-8 and characters outside the BMP:

• c3 a9 ("é") with %ls: 0, nothing stored, errno 86;
• f0 9f 98 80 with %ls: -1;
• 61 c3 a9 with %l[^ ]: 0, 0061 stored without a terminator;
• c3 61 with %l[a-z]: 0. Membership is tested on the raw byte, and c3 isn't in the set.

So decoding non-ASCII scanset members and non-BMP input as UTF-8 would give titles results their own libc.prx never produces. A non-ASCII scanset member (%l[à-ê]) I haven't measured. Its bytes are ≥ 0x80, and since every such input byte fails the conversion first, a member like that can probably never match.

The full rules and a comparison table are on #11724, where I proposed making the wide branch follow them on both platforms, stacked on that PR. I'd suggest this PR wait for that discussion.

AI-assisted: yes (Claude Code).

[blawesom]

Static review plus an independent Linux run (CachyOS, kernel 7.2.10, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, Ryzen 7 9800X3D). No findings.

What I checked (static)

• The scan-set classification moves from byte-level to codepoint-level: InWideScanset decodes each scanset member via the existing DecodeUtf8, so multi-byte members and ranges (%l[\xc3\xa0-\xc3\xaa]) work on code points rather than raw bytes. Negation is detected after decoding (^ as a first character), which matches how the byte-level version read it. Non-UTF-8 scanset text still throws via the existing NotImplemented_nid_no_patch path — consistent with the file's unverified-locale convention (called out in the debt note as a known assumption).
• EncodeWideScanCharacter replaces the old "one char16_t per code point" assumption with proper UTF-16: code points ≤ 0xFFFF write one unit, above becomes a surrogate pair. The old code had NotImplemented…("character outside the BMP") for > 0xFFFF input in ReadWideScanCharacter — moved/removed correctly since encoding is now the writer's concern, not the reader's.
• Width semantics are separated as C/FreeBSD intend: widths count characters (characters < wanted), _s buffer sizes count units (text.size() + count > limit overflow check) — the debt note records this explicitly ("widths count characters and _s sizes count units"), matching convert_wchar's FreeBSD behavior it cites.
• The unsized %lc non-BMP case throws NotImplemented — the caller's typed object holds one 16-bit unit and cannot represent a surrogate pair, matching what the note documents ("the caller's object holds one unit") rather than silently dropping/corrupting. The old width-LIMITED truncation semantics (wanted > limit → empty result) is replaced with an explicit overflow check that also writes u'\0' when target != nullptr, matching the C standard's requirement for _s.

Focused test run

tree	result
this head 3f2fa3b280	1/1 guest_formatting Passed (0.01s)

Test additions pin exactly the changed paths: non-ASCII scanset members (é, ranges, negation with non-ASCII), emoji as a surrogate pair in %ls (with and without width cap 2ls), correct overflow return 0 when two non-BMP characters exceed a 4-unit _s buffer, and the two expected-throw cases (%lc/%2lc for emoji) asserting the destination is left untouched.

Not an AGC/shader/VideoOut change — no paired Vulkan-validation run required.