Skip to content

fix(libc): read swscanf scanset ranges and string lengths as titles do - #12420

Merged
adriadam10 merged 4 commits into
boykopovar:mainfrom
JJenkx:windows/libc-swscanf-ranges
Oct 30, 2026
Merged

adriadam10 merged 4 commits into
boykopovar:mainfrom
JJenkx:windows/libc-swscanf-ranges

Conversation

@JJenkx

@JJenkx JJenkx commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What

swscanf and swscanf_s threw for three things that titles' own libc.prx (PPSA07923 and PPSA05916 agree on every row) reads without trouble. A title that hit any of them stopped.

Scanset ranges. Members are read left to right. X-Y is a range when a - follows a member and isn't the last character before ]:

input format titles
bca- %[a-c] 1, bca
za %[z-a] 0: a reversed range matches nothing, its ends included
-ab / a-b %[-a] / %[a-] 1, -a / a-: a first or last - is a member
b / a-c %[a-a-c] 0 / 1, a-c: after a range, - is a member again
+ %[*--] 1: the range * to -
b] %[]-c] 1: a ] that opens the set can start a range
éa %[a-é] 1: ranges compare 16-bit units
dcb / -]a %[^a-c] / %[^]-] 1, d / 0

Lengths on %s, %c and %[. h, hh, ll, j, z and L read narrow, like no length. Only l reads wide. Here every length but l threw.

Unknown conversions. %y, %5y and %hls (h followed by the conversion l) are matching failures. Here they threw.

Unknown integer and floating lengths, %n with suppression or width, and narrow input that wcrtomb can't convert still throw; those weren't measured.

Tested

Windows 11 Enterprise 10.0.22621, Ryzen 9 5950X, WinLibs GCC 15.2.0 UCRT r7, Ninja, Release, BUILD_TESTING=ON, on main 750cf6cf48.

  • Against titles' own libc.prx: I loaded PPSA07923's and PPSA05916's libc.prx natively and called their swscanf by NID (HNnWdT43ues) side by side with this build. Over 66 input/format pairs, comparing the return value, errno, the bytes or units stored and %n:
    • main throws on the first range, %hs or unknown conversion; the other rows already matched;
    • this branch: 0 differences.
  • Tests: guest_wide_conversion (21 new rows; the two "scanset range throws" checks now expect the measured result), guest_wide_stream_scan and guest_stdio_variadic pass in each of 3 runs.
  • Negative control: with main's ScanningWide.cpp, guest_wide_conversion fails.

Not tested on Linux or macOS. The scanner is the same code on every platform.

Overlap: tools/pr_overlap.py reports #9087, #9863 and #10729, all editing the same long swscanf debt entry (libc.md line 201). This PR changes only its "It throws for ..." sentence. #10324 touches ScanningWide.cpp in unrelated stream-orientation lines. I've asked on #9087, #9863 and #10729 to let theirs land first; I'll rebase after each.

Empty-input provenance clarification — 2026-10-29

Added the requested provenance caveat in 3961a1f069963e5cf67d70124ae8c9c0576b51d1: unknown conversions at end of input (for example %y on empty input) were not established by the published measurements, and their matching-failure result is explicitly unverified. This does not treat the separately measured empty unterminated-scanset rows as evidence for %y.

Only one debt sentence changes. Production, tests, public headers and submodule pointers are unchanged. Conventions against current main and whitespace pass (the existing long-entry warning remains), and the current-main merge preview is clean; no new runtime, Windows or SDK measurement is claimed. The range-rule change in the narrow scanner’s InWideScanset is separately implemented by #12468, rather than claimed as part of this PR. Pending external merge-order confirmations remain pending.

Patch and source-preservation receipt, SHA-256 c08b29d044ad929bf7d681059a6a330959ab81edf8784502dcc369f3148b49f9.

AI-assisted: yes (Codex).

Checklist

🤖 Generated with Claude Code

@MarcZX10

Copy link
Copy Markdown
Contributor

I independently built the focused scanner targets from head 96363b0687d6bdb4969f72761cc08c0e8cf8e382 in the configured Linux build tree:

  • guest_wide_conversion_tests
  • guest_wide_stream_scan_tests
  • guest_stdio_variadic_tests

All three passed under qemu-x86_64 -cpu max (x86-64 emulation). The variadic test printed its expected vprintf_s/wprintf_s/vwprintf_s output. These are emulated focused tests, not native console or title comparisons. git diff --check is clean. tools/check_conventions.py reports zero errors and the existing long-debt-entry warning for docs/dev/debt/libc.md:201 (3,265 characters). Current hosted conventions, eligibility, intake, labels and report checks are still pending.

The scanset loop consumes X-Y as a range only when the hyphen has a following endpoint; that preserves initial/final hyphens and lets a closing ] be the first range member, as covered by the new tests. I did not find a concrete blocker in the code delta. Please keep the PR's Tested section clear that this local run used QEMU; the Windows title-libc measurements in the description are separate evidence and are not reproduced here.

One coordination point: the overlap report flags the shared swscanf debt line with #9087, #9863 and #10729. The description says those may land first and that this branch will rebase, while the checklist says Depends on: none. Since the overlap is only in debt text and not implementation, this need not block the scanner change, but please make the intended landing order explicit and keep the debt edit rebased against whichever PRs land first.

@JJenkx

JJenkx commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

@MarcZX10 I made Depends on match the author’s existing proposal in Overlap: #9087, #9863 and #10729 first, then this debt-entry edit rebased after them. Those authors have already been asked; their agreement is still pending, so this is a proposed order rather than settled coordination. The shared line is documentation only; the scanner remains ready for review.

Your focused Linux results remain explicitly QEMU x86-64 execution, separate from the author’s Windows-host title-library measurements. No native Linux, console or title comparison is inferred from your run.

AI-assisted: yes.

@rooklz rooklz left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed head 96363b0687 on macOS, since the description says Linux and macOS weren't tested. Setup: macOS 26 (Apple silicon), Apple clang 21.0.0, libc++, C++20, built for x86_64 and run under Rosetta. arm64 can't build ScanningWide.cpp at all because of the sizeof(long double) == 16 static asserts.

Setup. guest_wide_conversion isn't in the relinker-only lane and the full libc target needs submodules, so I compiled ScanningWide.cpp alone (head, then main 750cf6cf48's copy) against a small harness. The harness stubs only the 7 imports the object needs: UTF-8 wcrtomb, ASCII iswspace, and stream functions that abort if called. It runs the 21 rows from the new test table plus 7 more: %js, %zs, %Ls, %ts, %[]] on ab, %[]-] on ]-a, and %y on empty input. Each row checks the return value and the stored bytes against an untouched ! buffer.

  • Head, ASan+UBSan: all 21 PR rows and the 6 extra length/scanset rows pass. No sanitizer reports.

  • main's ScanningWide.cpp: 23 of the 28 rows fail. 12 throw scanset range is implementation-defined, 7 throw invalid string length modifier (h, hh, ll, j, z, L, t) and 4 throw unsupported conversion (%y ×3 and %hls, reported as conversion 108 = l). The 5 rows with no range (%[-a], %[a-], %[^]-], %[]], %[]-]) already pass. So the change is exactly the three throws the description lists.

  • Mutants on InSet / Conversion, all killed by the PR's own 21 rows:

    • member + 2 < → member + 1 <, so a trailing - starts a range: only the %[a-] row fails (my extra %[]-] row fails too).
    • member += 3 → member += 1, so ends are re-read and - stops being a member again after a range: 5 rows fail.
    • range test && → ||: 7 rows fail.
    • no range at the first member: 9 rows fail.
    • reversed range swapped to match [min, max]: %[z-a] and %[a--] fail.
    • unknown conversion Matching → Input: the 3 unknown-conversion rows fail.

    So the table pins the range parsing tightly.

  • git merge-tree against upstream/main 750cf6cf48 is clean. check_conventions --base upstream/main with this title and body reports 0 errors. Its one warning, debt-length on libc.md:201 at 3265 characters, was already over the limit on main (3083).

One non-blocking question. With no input left, an unknown conversion returns 0 on this branch: swscanf(u"", u"%y", …) gives 0, and so does u" " with u" %y". %d and %s on the same empty input return EOF (-1). That matches what %S/%C already do on main, and the debt entry says for those that "whether input ending before them is an input failure instead is unverified". If your 66 pairs didn't include an empty-input unknown conversion, it may be worth adding "unknown conversions" to that unverified clause, since the new sentence says they "follow the libc.prx of PPSA07923 and PPSA05916" without that caveat. If they did include it, a row for it in the table would lock it in.

AI-assisted: yes (Claude Code); everything above was run on this machine.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 5021 docs/dev/debt/libc.md
PR 5303 docs/dev/debt/libc.md
PR 7447 docs/dev/debt/libc.md
PR 7715 docs/dev/debt/libc.md
PR 8150 docs/dev/debt/libc.md
PR 8965 docs/dev/debt/libc.md
PR 9371 docs/dev/debt/libc.md
PR 9447 docs/dev/debt/libc.md
PR 9526 docs/dev/debt/libc.md
PR 10324 core/libs/prx/libc/src/ScanningWide.cpp
docs/dev/debt/libc.md
PR 10405 docs/dev/debt/libc.md
PR 10620 docs/dev/debt/libc.md
PR 10989 docs/dev/debt/libc.md
PR 11332 docs/dev/debt/libc.md
PR 11661 docs/dev/debt/libc.md
PR 11689 docs/dev/debt/libc.md
PR 11698 docs/dev/debt/libc.md
PR 11831 docs/dev/debt/libc.md
PR 12057 docs/dev/debt/libc.md
PR 12292 docs/dev/debt/libc.md
PR 12560 docs/dev/debt/libc.md
PR 12586 docs/dev/debt/libc.md
PR 12605 docs/dev/debt/libc.md
PR 12920 docs/dev/debt/libc.md

@ebosido444-ai

Copy link
Copy Markdown
Contributor

The range handling holds up against the cases added — a-, -a, ] first and ^ exclusion all behave as the table asserts. Three things worth tightening:

  1. core/libs/prx/libc/src/ScanningWide.cpp:319 — default: return Outcome::Matching; makes every unknown conversion a silent end-of-scan. Combined with the removed length-modifier Fail in Text(), the scanner now has three different answers to a format it does not understand: %Ld and %lp still throw (pinned at core/libs/tests/GuestWideConversion.cpp:146), %y and %hls return 0 as a matching failure, and a bad scanset no longer throws at all. The table pins {u"ab", u"%y", 0, ""} and {u"ab", u"%hls", 0, ""}, so this looks deliberate — but the debt entry should name the libc.prx that shows an unknown conversion ending the scan silently rather than throwing, because 0 is also what a genuine mismatch returns and a caller cannot tell the two apart. %lp keeping its throw is the case that reads inconsistent next to %y.

  2. :398 — member + 2 < directive.setEnd forms a pointer two elements past setEnd when the set is shorter than two characters (%[], %[a]). member + 1 < directive.setEnd is the only comparison valid for an arbitrary set; the length test is clearer as (directive.setEnd - member) >= 3.

  3. The %[a-a-c] and %[c-a-z] rows read an overlapping or reversed range as a single-character range followed by literal members. That is a defensible reading, but it is the one place the implementation cannot agree with glibc, so it needs the same module citation as the other ranges rather than standing on these tests alone.

@ebosido444-ai

Copy link
Copy Markdown
Contributor

A second copy of the range rule was not updated with the one in InSet:

  1. core/libs/prx/libc/src/ScanWideField.cpp:51 — InWideScanset still tests specifier[index] == '-' && index != first && index + 1 < end, with no literal - at the end of the set and no reversed-range rejection. The two scanners therefore disagree on the same format text: for %[a-a-c] the wide path here reads - as a range from a to a and then matches literal c, where InSet matches the single character a followed by literal - and c; for %[z-a] this path leaves z as a member and InSet does not. That is reachable as sscanf("%l[...]"), so the behaviour a title sees depends on which of the two scanners parsed its format string. Pinning the same ranges[] table against InWideScanset would fail today.

  2. ScanningWide.cpp:289 — Fail("unterminated scanset") is still there, whereas the sibling row {"ab", "%[^]", 0, ""} in core/libs/tests/Formatting.cpp measures 0 for an unterminated set through the narrow path. If the titles' libc.prx ends the scan in the wide path too, this one needs the same treatment for the scanners to agree.

@JJenkx

JJenkx commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

@ebosido444-ai thanks, both comments were useful. I measured each point on titles' own libc.prx first (PPSA07923 and PPSA05916, swscanf by NID HNnWdT43ues); both agree. 1bb1f9728c follows them.

First comment

  1. Unknown conversions vs %Ld/%lp. You were right that it read inconsistently: the titles accept the ones that threw here. %Ld and %Lx read an int ("5" gives 05000000), %lp and %hp read a pointer, and %hhd an 8-bit int. %y, %hls and %qd are matching failures. 1bb1f9728c now does the same: IntegerSize takes L as no length, and Pointer ignores its length. The two "throws" checks now pin the measured results, and the debt entry cites the module. Other unknown integer and floating lengths still throw, since they weren't measured.
  2. member + 2 < setEnd. Fixed as directive.setEnd - member >= 3.
  3. The a-a-c / c-a-z reading. That comes from the module, not from the tests alone. The ranges table in the PR body was measured on both titles: %[a-a-c] rejects b and takes a-c, and %[c-a-z] takes - but not b or x. The debt entry names the module.

Second comment

  1. InWideScanset is the narrow scanner's %l[ path, used on Linux in-process only. On Windows, %l[ goes to the host via TitleScanset from fix(libc): match bundled narrow scansets and scan directive parsing #12428. I'll bring it to the same rule in my stacked follow-up on feat(libc): read 16-bit wide fields in fscanf on Linux #11724 (the C-locale wide fields), which rewrites InWideScanset, runs that path on Windows too, and can test it there. Changing it here couldn't be run on this machine.
  2. Unterminated scanset. The titles end the scan as a matching failure in the wide scanner too: %[^] and %[ab on ab give 0, and %[^] on empty input also gives 0. 1bb1f9728c returns a matching failure instead of throwing; new checks cover all three.

Checks:

  • Against the titles: the 76 swscanf pairs (the earlier 66 plus these 10) show 0 differences on both titles, except %Lf. There the stored low 8 bytes match, and the difference is past them in the 16-byte long double, which this PR doesn't touch; SteelCrab's build(libs): support Emscripten memory64 compilation #11341 rewrites that store.
  • Tests: guest_wide_conversion, guest_wide_stream_scan and guest_stdio_variadic pass in each of 3 runs (Windows 11 Enterprise 10.0.22621, WinLibs GCC 15.2.0 UCRT r7, Release).

AI-assisted: yes (Claude Code).

JJenkx and others added 3 commits October 28, 2026 18:58
The swscanf of titles' libc.prx (PPSA07923 and PPSA05916 agree) accepts
what this scanner threw for:

- a scanset range X-Y matches the units from X to Y; a reversed range
  matches nothing, its ends included; a dash first, last or right after
  a range is a member, and a ] that opens the set can start a range;
- %s, %c and %[ with any length but l (h, hh, ll, j, z, L) read narrow;
- an unknown conversion such as %y is a matching failure.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The swscanf of titles' libc.prx (PPSA07923 and PPSA05916 agree) reads
%Ld and %Lx as an int, ignores any length on %p (%lp, %hp), and ends
the scan at an unterminated scanset as a matching failure, even at the
end of the input. Here all three threw. The range test in InSet also no
longer forms a pointer past the end of a set shorter than two members.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@adriadam10
adriadam10 force-pushed the windows/libc-swscanf-ranges branch from 1bb1f97 to a51d64d Compare October 28, 2026 18:04
@JJenkx

JJenkx commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Linux run at this PR's exact head a51d64dbb7 (a51d64dbb74f749fcd422aedbc73800c4f84c644), not merged onto anything else; its base on main is e4da19aff0.

Arch Linux, kernel 7.2.9, Ryzen 9 7950X, g++ (GCC) 16.2.1 20260810, glibc 2.44, x86-64. Release, BUILD_TESTING=ON, Ninja -j4, private worktree.

  • ctest -R '^guest_wide_conversion$': passes in each of 3 runs (3 passes).

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review — changes requested for overlap coordination

I reviewed the scanner changes on head a51d64dbb74f749fcd422aedbc73800c4f84c644. The scanset walker handles the reported left-to-right range/member cases, and the text/unknown-conversion paths match the Windows title-library measurements described in the PR. I found no new code defect in the changed logic. The author reports 66 differential cases with zero mismatches, three passing runs of the focused Windows suites, and a negative control. I did not run the harness locally.

I cannot clear the overlap/readiness blocker yet. The proposed order (#9087, #9863 and #10729 first, then this PR) is recorded as pending, and the #9863 author has not confirmed the #9863-first sequence. Please obtain confirmation from the overlapping authors and keep the Depends on list aligned with those agreements. Current main also changed the shared docs/dev/debt/libc.md after this branch’s base; the merge preview is clean, but the debt hunk needs reconciliation against that current version.

After the order and debt entry are settled, the existing scanner test results remain relevant if production/test files are unchanged; rerun conventions/overlap checks on the integrated head. Hosted policy and build checks currently pass, with Linux product CI skipped.

@JJenkx

JJenkx commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Added the requested provenance caveat in 3961a1f069963e5cf67d70124ae8c9c0576b51d1: unknown conversions at end of input (for example %y on empty input) were not established by the published measurements, and their matching-failure result is explicitly unverified. This does not treat the separately measured empty unterminated-scanset rows as evidence for %y.

Only one debt sentence changes. Production, tests, public headers and submodule pointers are unchanged. Conventions against current main and whitespace pass (the existing long-entry warning remains), and the current-main merge preview is clean; no new runtime, Windows or SDK measurement is claimed. The range-rule change in the narrow scanner’s InWideScanset is separately implemented by #12468, rather than claimed as part of this PR. Pending external merge-order confirmations remain pending.

Patch and source-preservation receipt, SHA-256 c08b29d044ad929bf7d681059a6a330959ab81edf8784502dcc369f3148b49f9.

AI-assisted: yes (Codex).

@JJenkx

JJenkx commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Linux run at this PR's exact head 3961a1f069 (3961a1f069963e5cf67d70124ae8c9c0576b51d1), not merged onto anything else; its base on main is e4da19aff0.

Arch Linux, kernel 7.2.9, Ryzen 9 7950X, g++ (GCC) 16.2.1 20260810, glibc 2.44, x86-64. Release, BUILD_TESTING=ON, Ninja -j4, private worktree.

  • ctest -R '^guest_wide_conversion$': passes in each of 3 runs (3 passes).

This head adds only a debt-file sentence; this is a runtime run at the head itself.

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked — new head, changes requested

The exact-head Linux test report is useful, but the PR still records the overlapping authors’ landing-order agreement as pending.

What I ran

  • Exact head: 3961a1f069963e5cf67d70124ae8c9c0576b51d1; PR base recorded as e4da19aff06e25ae16b90f460dff22e8ac1eb786; current main: fd9b3f62d43d6753d28a31a7c285e1f51e9c1b42.
  • Environment: Linux x86_64; GitHub CLI/API inspection only. I did not build or run the tests locally.
  • Commands:
gh api repos/boykopovar/AnyPS5/pulls/12420 --jq '{head:.head.sha,base:.base.sha,mergeable:.mergeable,state:.state}'
gh api 'repos/boykopovar/AnyPS5/compare/e4da19aff06e25ae16b90f460dff22e8ac1eb786...fd9b3f62d43d6753d28a31a7c285e1f51e9c1b42' --jq '[.files[] | select(.filename=="core/libs/prx/libc/src/ScanningWide.cpp" or .filename=="core/libs/tests/GuestWideConversion.cpp" or .filename=="docs/dev/debt/libc.md") | {filename,status,additions,deletions}]'
gh pr checks 12420 --repo boykopovar/AnyPS5

Results

Check Result
Current-main file comparison No changes to the three PR paths between the recorded base and current main; GitHub reports the PR mergeable.
Latest code change Author says the new commit changes only one debt sentence; production and test files are unchanged.
Focused test Author reports exact-head Linux guest_wide_conversion passed 3/3. I did not run it.
Hosted checks Eligibility, conventions, labels, report, Windows, relinker, ASan, and register-context checks pass; Linux product tests are skipped.
Landing-order agreement The PR body still says confirmations from the overlapping authors are pending.

Negative control

Assertion Change to code under test Did the new test fail?
The scanner tests reject incorrect parsing behavior No mutation run in this recheck; the latest change is debt text only. Not exercised by me.

Findings

Blocking

The checklist says Depends on: #9087, #9863, #10729 but also says the authors’ agreement is pending. The current contribution rule says to agree with authors of overlapping PRs which lands first and declare that order with Depends on. Please obtain the confirmations and update the checklist to the agreed order before requesting approval.

Non-blocking

The author’s exact-head Linux test report is useful, and the latest change is documentation-only. I found no new scanner-code issue in this head. I did not independently run the test or the convention checks.

Sources

Not verified

No local build, focused test, convention check, or negative-control mutation. Linux product CI is skipped by the workflow.

To get this to a good state

Obtain explicit author agreement on the order for the overlapping PRs and update the checklist accordingly. I will recheck the same merge, overlap, convention, and focused-test state on the updated head.

@JJenkx

JJenkx commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

@MarcZX10 I checked the author threads again. #9087’s author explicitly agreed that it lands first and this PR follows. The requests to #9863 and #10729 remain unanswered. The checklist now separates the confirmed order from those pending confirmations; no agreement is inferred.

@UnhealthyKraken @dwk001 could you confirm the proposed order, your debt-entry edit first and #12420 rebased afterward? The proposal preserves both sentences in the shared scanner debt entry and adds no implementation dependency. The scanner code and prior measured results are unchanged by this metadata clarification.

AI-assisted: yes (Codex).

@adriadam10
adriadam10 merged commit 1764ec5 into boykopovar:main Oct 30, 2026
20 of 21 checks passed
@adriadam10

Copy link
Copy Markdown
Collaborator

Re-checked: the only change since the approved head is the author's docs commit, which adds one docs/dev/debt/libc.md sentence saying unknown conversions at end of input are unmeasured. Code and tests are unchanged; the merge preview is clean against current main and CI is green; merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants