feat(libc): scan 16-bit wide conversions in fscanf on Linux by KernelPanicNT · Pull Request #6018 · boykopovar/AnyPS5

KernelPanicNT · 2026-10-14T19:21:39Z

What

Review note. This branch is stacked on #6017, so the Files changed tab also shows that PR's commit. This PR is only the second commit, e32dea4; please merge #6017 first, after which this diff shrinks to that commit. The progress report's +2 counts vsscanf from #6017 and fscanf from this PR.

Depends on #6017, which adds 16-bit wide scanning to sscanf, vsscanf, sscanf_s and vsscanf_s on Linux. This PR is the single commit on top of it. On Linux, fscanf still throws for %ls, %lc, %l[, %S and %C (RejectGuestWideScan, called from fscanf), because glibc would store a 32-bit wchar_t per character into the guest's 16-bit buffer. This PR scans them from the stream on glibc hosts.

Source of the behaviour. No console measurement is involved. The model is the same as in the PR this depends on, now read from a stream:

FreeBSD 11.4 (release/11.4.0, commit 8c63994) vfscanf.c L161-L308 and L683-L728: with l, %c, %[ and %s read the stream one character at a time through the locale's mbrtowc. %S/%C are %s/%c with l (L594-L612). FreeBSD's vsscanf runs the same __svfscanf on a string FILE (vsscanf.c L61-L71), so fscanf and sscanf share these semantics.
none.c L99-L113: the C-locale mbrtowc maps one byte to its value, as this library's mbrtowc does.
The stream side uses the m assignment-allocation character of POSIX.1-2008 fscanf (added in Issue 7, SD5-XSH-ERN-132; accepted by glibc's __isoc99_vfscanf). The number of bytes a stream directive reads is not known in advance, and m makes glibc allocate exactly that many. This is host plumbing only; the guest sees no allocation. Apple's libc has no m (vfscanf.c handles j, l, q, t, z, L and h only), so the stream path is built only where __GLIBC__ is defined. The string functions do not use m.

Changes:

The body of ScanGuest moves into ScanGuestSource, a template over an input source. ScanGuest now wraps it with a StringScanSource, which holds the existing string cursor and, as in the PR this depends on, scans a wide directive into a buffer bounded by the remaining input. Its callers (Linux sscanf/vsscanf for wide formats, sscanf_s/vsscanf_s on both platforms) keep their behaviour on every host.
- StreamScanSource (glibc hosts only) reads a host FILE* and holds its lock (flockfile, released by the destructor, also when the scanner throws) for the whole fscanf call, so another guest thread cannot read the same stream between the per-directive calls. Literals and whitespace are matched with one getc/ungetc pair per peeked byte. Each conversion other than %n is one glibc fscanf call with a trailing %n, which counts the bytes the call consumed. A guest %n is filled from the scanner's own count, without a glibc call.
On glibc hosts, fscanf sends a format with a 16-bit wide conversion to ScanGuestSource over the stream. Formats without one still go to glibc's vfscanf unchanged, and SyncStatus still runs after either path. On a non-Windows host without glibc (macOS), a format with a wide conversion throws std::runtime_error before reading the stream; other formats go to the host vfscanf as before.
On the stream, a wide directive becomes %<width>m<narrow conversion>. glibc allocates the bytes, which are widened through mbrtowc as in the string case and then freed. Every byte the conversion stored is widened, counted by the trailing %n, so a NUL byte that %ls or %l[ reads from the stream becomes a 0 unit and the bytes after it are kept, as FreeBSD's convert_wstring and convert_wccl store L'\0' and continue. %ls and %l[ are terminated, %lc is not. The q handling and the narrow s, c and [ of the PR this depends on apply to the stream too: %qS is wide and %qs narrow.
RejectGuestWideScan had no other caller and is removed.

Still throwing, and why:

A wide conversion with a positional argument (%1$ls) or the ' flag throws std::runtime_error before reading the stream, as in the PR this depends on.
An invalid length on %n (%ls%Ln, std::invalid_argument) throws only when the scanner reaches that directive. By then fscanf has consumed the stream up to it, and the throw skips va_end and SyncStatus. This is recorded in docs/dev/debt/libc.md.
On a non-Windows host without glibc, any 16-bit wide conversion in fscanf throws std::runtime_error before reading the stream, because the host has no m. The previous throw was NotImplemented_nid_no_patch on every non-Windows host.
Windows is unchanged: fscanf translates the format for the host, whose wchar_t is 16 bits. %S and %C still throw in that translation.

Tests: guest_standard_streams gains CheckWideScan on glibc hosts, run on tmpfile streams. Each case also checks what is left in the stream through fgetc:

%ls%n %3lc%d on ab cdef 12: 2 assignments, %n is 2, %d fails on f, and f is left.
%C+%lc on a-b: 1 assignment, and - is left.
%l[a-z] on 12: returns 0, the buffer is untouched, and 1 is left.
%S on bytes \xe9t\xff : U+00E9, t, U+00FF, and the space is left.
A 300-byte word: read in full, terminated, and the following space is left.
An empty stream: EOF, with the guest EOF flag set, and fgetc returns EOF.
%*ls%d on ab: returns 0, and fgetc returns EOF.
%1$ls: throws with the buffer and the stream untouched, and a is left.
%l[^\n] on the bytes a, NUL, b, newline: 1 assignment, the units a, 0, b, 0, the guard units after them untouched, and the newline is left.
%qS on xy z: 1 assignment, x, y, 0, and the space is left.

On other non-Windows hosts, CheckWideScan checks that %ls on ab throws std::runtime_error with the buffer untouched and a left.

Progress (tools/progress.py, base e27ff7c plus the PR this depends on): system libraries 3270/3679 → 3271/3679 (+fscanf_nid_postfix). The tool counted fscanf as unimplemented because its single definition reached NotImplemented_nid_no_patch on Linux.

Tested

Head e32dea4c5c95614b5320a54613bbd1413089195e, base f4fa4a38fbffa8f19a3cdb6ec4b593c3da919e10.

Linux (Ubuntu 24.04, GCC 14.2.0, Ninja, Release, BUILD_TESTING=ON, AMD EPYC 7763 (GitHub Codespaces), tests run natively): guest_standard_stream_tests and guest_formatting_tests each pass in 5 of 5 runs, the test executable being run directly five times.
Negative control: keep the tests and replace only core/libs/prx/libc/src/Formatting.cpp with the base version: guest_standard_stream_tests fails with fscanf 16-bit wide conversion on Linux not implemented; guest_formatting_tests still passes there, as expected for regression coverage. Restoring the PR source and rebuilding, guest_standard_stream_tests and guest_formatting_tests each pass in 5 of 5 runs again.
Windows, full Build workflow on my fork (run 38736530145, attempt 2; attempt 1 stopped at configure when the runner could not download FFmpeg), this head built together with the updated heads of feat(libc): scan 16-bit wide conversions in sscanf and vsscanf on Linux #6017, feat(libc): scan 16-bit wide conversions in fscanf on Linux #6018 and feat(libSceAmpr): implement __cxa_call_unexpected #6024 and merged into the fork's main at b08db6a093f440af697d71ea907cec5f675fcdc5, WinLibs MinGW-w64 GCC 15.2.0, Release, BUILD_TESTING=ON: 1278/1279 tests pass, including guest_standard_streams and guest_formatting. The one failure, prx_thread_local_destructors, reports a thread_local destructor in libSceAgcDriver (traceDrawKeyLookup), which main gained in de2ddda the same day; this PR does not touch that library. The relinker (Linux, Windows, macOS), register-context and write-watch jobs of the same run pass.
Conventions checker and git diff --check pass. The Linux job of the Build workflow now targets the project's self-hosted runner, so it was not run on the fork; no title was run.

Duplicate and overlap check

Earlier PRs for the same change, closed without merge. feat(libc): scan 16-bit wide conversions in fscanf on Linux #5291 by @Nixitov scanned wide fscanf conversions on Linux by decoding the stream's UTF-8; it was stacked on feat(libc): scan 16-bit wide conversions on Linux #5285, also by @Nixitov, which did the same for sscanf/vsscanf (the PR this depends on covers that part). Both were closed on 2026-10-14 by @oneandonlydean with the same comment: "Closing: no title is known to reach this. With the queue at 700+ open PRs, please hold off on new PRs that don't name a title needing the change until it's down. If a title stops on this on current main, reopen with the title ID and the log line." This PR does not name a title either (Unblocks: none). It covers the same functions as feat(libc): scan 16-bit wide conversions in fscanf on Linux #5291 with a different method: each byte is widened through this library's single-byte mbrtowc, following FreeBSD's C locale, instead of decoding UTF-8. feat(libc): scan 16-bit wide conversions in fscanf on Linux #5291 already had the stream design this PR uses for everything but the wide bytes: it read literals, whitespace and %n counts one character at a time with a getc/ungetc peek so that the stream position and the counts match, scanned each other conversion of such a format with one host fscanf call and a trailing %n through a variadic HostFscanf wrapper over HostVfscanf, and dispatched in fscanf_nid_postfix on HasGuestWideScan. This PR follows that design. fix(libc): throw for 16-bit wide scanf conversions on Linux #3984 introduced the throw this PR replaces.
Open PRs naming the function. I searched open, merged and closed PRs for fscanf, its NID npLpPTaSuHg and "wide scanf". No open PR implements wide scanning in fscanf. feat(libc): implement wide and unlocked stream I/O #5816 adds wide stream input (fgetwc, fgetws, getwc, getwchar) in FormattingWide.cpp/StandardStreams.cpp, not scanf conversions.
Open PRs touching the same files. I ran a three-way merge (git merge-file) of each PR head against this branch, per file, from the PR's merge base.
- Merge cleanly:
  - fix(libc): fail stream calls on a stream that a failed freopen closed #5632, whose one-line closed-stream check at the top of fscanf_nid_postfix sits above the changed lines.
  - fix(libc): check allocation result in vasprintf and guard null pointer in strndup #5709 in Formatting.cpp.
  - feat(libc): implement wide and unlocked stream I/O #5816 in core/libs/tests/GuestStandardStreams.cpp.
  - fix(libc): ignore the space and + flags for unsigned printf conversions #5774 and fix(libc): match title printf behavior on Windows #3987 in core/libs/tests/Formatting.cpp.
  - feat(libc): implement wide and unlocked stream I/O #5816, fix(libc): follow measured wide printf behavior #5303, perf(agc): import resident direct memory through a section alias on Windows #5851, feat: read title files from a read-only app0.zar (ZArchive) layer #5835, fix(libc): start the guest environment empty instead of copying the host's #5772, feat(libc): implement x87 long-double finite classification #5599, fix(libc): match title printf behavior on Windows #3987 and fix(windows): enter guest startup on an arena-backed main stack #3965 in docs/dev/debt/libc.md.
- Textual conflicts, each resolved by keeping both sides:
  - fix(libc): fail stream calls on a stream that a failed freopen closed #5632 and fix(libc): keep the host standard stream when a guest freopen of it fails #5668 add a test function in place of int main() { right after CheckDescriptorAccessModes in core/libs/tests/GuestStandardStreams.cpp, where this PR inserts CheckWideScan and its call.
  - Carried over from the PR this depends on: fix(libc): build the host-facing libc code on macOS #4627 and Macos m4 #6222 (includes in core/libs/tests/Formatting.cpp; Macos m4 #6222 was opened after this PR, re-checked on 2026-10-15) and feat(libc): implement collate<wchar_t>::_Getcat #5662 (the collate<char> line next to the scanf line in docs/dev/debt/libc.md).
- feat(libc): provide the runtime scripting allocator hooks #5092 already conflicts with main in docs/dev/debt/libc.md.
Base. This branch is one commit on top of feat(libc): scan 16-bit wide conversions in sscanf and vsscanf on Linux #6017, which is based on e27ff7c.

Checklist

🤖 Generated with Claude Code

github-actions · 2026-10-14T19:22:31Z

Progress report (`2e5dd5a` → `e32dea4`)

📈 System libraries: 88.75% (+0.05%, +2 implemented functions)

✅ 2 implemented

Library	Name
libc	`fscanf_nid_postfix`
libc	`vsscanf_nid_postfix`

github-actions · 2026-10-14T19:26:15Z

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 3891	`core/libs/tests/Formatting.cpp`
PR 3965	`docs/dev/debt/libc.md`
PR 3987	`core/libs/tests/Formatting.cpp` `docs/dev/debt/libc.md`
PR 4156	`core/libs/tests/Formatting.cpp`
PR 4396	`core/libs/tests/Formatting.cpp`
PR 4397	`core/libs/tests/Formatting.cpp`
PR 4422	`core/libs/tests/Formatting.cpp`
PR 4423	`core/libs/tests/Formatting.cpp`
PR 4424	`core/libs/tests/Formatting.cpp`
PR 4520	`core/libs/tests/Formatting.cpp`
PR 4521	`core/libs/tests/Formatting.cpp`
PR 4522	`core/libs/tests/Formatting.cpp`
PR 4523	`core/libs/tests/Formatting.cpp`
PR 4566	`core/libs/tests/Formatting.cpp`
PR 4599	`core/libs/tests/Formatting.cpp`
PR 4627	`core/libs/tests/Formatting.cpp`
PR 4718	`core/libs/tests/Formatting.cpp`
PR 4740	`core/libs/tests/Formatting.cpp`
PR 4749	`core/libs/tests/Formatting.cpp`
PR 4760	`core/libs/tests/Formatting.cpp`
PR 4813	`core/libs/tests/Formatting.cpp`
PR 5021	`core/libs/tests/Formatting.cpp`
PR 5092	`docs/dev/debt/libc.md`
PR 5303	`docs/dev/debt/libc.md`
PR 5668	`core/libs/tests/GuestStandardStreams.cpp`
PR 5772	`docs/dev/debt/libc.md`
PR 5774	`core/libs/tests/Formatting.cpp`
PR 5835	`docs/dev/debt/libc.md`
PR 6044	`core/libs/tests/Formatting.cpp` `core/libs/tests/GuestStandardStreams.cpp`
PR 6061	`docs/dev/debt/libc.md`
PR 6107	`docs/dev/debt/libc.md`
PR 6124	`docs/dev/debt/libc.md`
PR 6133	`docs/dev/debt/libc.md`
PR 6144	`docs/dev/debt/libc.md`
PR 6217	`docs/dev/debt/libc.md`
PR 6222	`core/libs/tests/Formatting.cpp` `docs/dev/debt/libc.md`
PR 6226	`docs/dev/debt/libc.md`
PR 6315	`core/libs/prx/libc/src/Formatting.cpp` `core/libs/tests/Formatting.cpp` `docs/dev/debt/libc.md`
PR 6318	`docs/dev/debt/libc.md`
PR 6319	`docs/dev/debt/libc.md`
PR 6322	`docs/dev/debt/libc.md`
PR 6330	`docs/dev/debt/libc.md`
PR 6331	`docs/dev/debt/libc.md`
PR 6334	`docs/dev/debt/libc.md`
PR 6339	`docs/dev/debt/libc.md`
PR 6345	`docs/dev/debt/libc.md`

On Linux, sscanf, vsscanf, sscanf_s and vsscanf_s threw for %ls, %lc, %l[, %S and %C because glibc stores a 32-bit wchar_t per character into the guest's 16-bit buffer. A format with one of these conversions is now scanned directive by directive in ScanGuest: each wide directive reads its bytes as the narrow directive does and widens every byte through this library's mbrtowc, which is what FreeBSD 11.4's vfscanf.c reads in the single-byte C locale of none.c. The byte buffer for a wide directive is bounded by the remaining input, not only by the guest's width. As in FreeBSD, %S and %C are wide whatever length modifier precedes them. The format pre-scan and ScanGuest recognise the q length modifier as FreeBSD does, so %lS, %hC and %qS are found too instead of reaching glibc, and %qd or %qn in a format scanned by ScanGuest stores a long long. On Windows, ScanGuest hands q to the host as ll for integer conversions. A narrow s, c or [ is handed to the host without any length modifier other than l, so %lls, %qs or %Lc in a format scanned by ScanGuest are narrow, as in FreeBSD, instead of reaching glibc as wide. A wide conversion combined with a positional argument or the ' flag still throws. fscanf keeps throwing for wide conversions. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

fscanf threw for %ls, %lc, %l[, %S and %C on Linux. The body of ScanGuest moves into ScanGuestSource, which reads from a source: a string, as before through ScanGuest, or a stream, where each conversion other than %n is one glibc fscanf call and the scanner peeks with getc/ungetc under the stream's lock. fscanf uses the stream source for formats with a wide conversion, so a wide directive reads its bytes as the narrow one and widens them through this library's mbrtowc, as sscanf already does. On the stream, wide directives take their bytes with the POSIX.1-2008 m assignment-allocation modifier, since a stream's string length is not known before it is read. The string source keeps its buffer bounded by the remaining input. Every byte the stream conversion stored is widened, so a NUL byte read by %ls or %l[ is kept as a 0 unit, as FreeBSD's convert_wstring and convert_wccl store it. m needs glibc, so on other non-Windows hosts fscanf throws std::runtime_error for a wide conversion. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

github-actions Bot added libc docs labels Oct 14, 2026

github-actions Bot mentioned this pull request Oct 15, 2026

Macos m4 #6222

Closed

11 tasks

KernelPanicNT and others added 2 commits October 15, 2026 11:41

KernelPanicNT force-pushed the feat/libc-fscanf-guest-wide-conversions branch from cdf9322 to e32dea4 Compare October 15, 2026 13:16

This was referenced Oct 15, 2026

feat(libc): scan 16-bit wide conversions in sscanf and vsscanf on Linux #6017

Closed

feat(libSceAmpr): implement __cxa_call_unexpected #6024

Merged

This was referenced Oct 15, 2026

fix(windows): enter guest startup on an arena-backed main stack #3965

Open

fix(libc): open closed standard descriptors on /dev/null when libc loads #6107

Merged

github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 15, 2026

KernelPanicNT closed this Oct 15, 2026

KernelPanicNT mentioned this pull request Oct 16, 2026

feat(libc): scan 16-bit wide conversions in fscanf on Linux #6732

Closed

11 tasks done

Navigation Menu

Repository navigation

feat(libc): scan 16-bit wide conversions in fscanf on Linux - #6018

feat(libc): scan 16-bit wide conversions in fscanf on Linux#6018
KernelPanicNT wants to merge 2 commits into
boykopovar:mainfrom
KernelPanicNT:feat/libc-fscanf-guest-wide-conversions

KernelPanicNT commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

KernelPanicNT commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Tested

Duplicate and overlap check

Checklist

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Progress report (2e5dd5a → e32dea4)

Uh oh!

github-actions Bot commented Oct 14, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

KernelPanicNT commented Oct 14, 2026 •

edited

Loading

github-actions Bot commented Oct 14, 2026 •

edited

Loading

Progress report (`2e5dd5a` → `e32dea4`)

github-actions Bot commented Oct 14, 2026 •

edited

Loading