Skip to content

fix(libc): read wide scanf fields byte by byte, as titles' libc.prx does - #12468

Merged
oneandonlydean merged 8 commits into
boykopovar:mainfrom
JJenkx:windows/libc-wide-scan-c-locale
Nov 2, 2026
Merged

oneandonlydean merged 8 commits into
boykopovar:mainfrom
JJenkx:windows/libc-wide-scan-c-locale

Conversation

@JJenkx

@JJenkx JJenkx commented Oct 28, 2026 •

Copy link
Copy Markdown
Contributor

What

Titles' own libc.prx (PPSA07923, PPSA05916 and PPSA13934 agree on every row) doesn't decode UTF-8 in the narrow scanf family's 16-bit wide conversions. Each byte goes to mbrtowc, whose state is kept for the field, and in the C locale MB_CUR_MAX is 1, so any byte from 0x80 up ends the conversion. The full measurements are on #11724 (comment). In short:

  • A UTF-8 lead byte (0xC0–0xEF, where mbrtowc returns -2) is held.
    • The next byte the field takes then fails it with EILSEQ, as a matching failure.
    • If the field ends instead (width used up, a space for %ls, a byte outside the set for %l[, or end of input), it's an input failure.
  • Any other refused byte (0x80–0xBF, 0xF0–0xFF) is an input failure as the field's first byte and a matching failure after it.
  • Stored units: those read before a failure stay stored without a terminator.
  • Widths count bytes.
  • Scanset members and X-Y ranges are bytes. A reversed range matches nothing, and a - first, last or right after a range is a member, as in swscanf (fix(libc): read swscanf scanset ranges and string lengths as titles do #12420) and Windows narrow %[ (fix(libc): match bundled narrow scansets and scan directive parsing #12428).
  • %C isn't a conversion and ends the scan; %S, %hS and %lS read like %ls.

Before this PR:

  • the guest scanner (Linux) decoded UTF-8, and threw for invalid input, characters outside the BMP and non-ASCII scanset members;
  • Windows handed these conversions to the host, which stored each byte as one unit and threw for %S and %C.

The change, on top of #11724:

  • The wide branch of ScanSource runs on every platform. Windows sscanf, vsscanf, fscanf and vfscanf send formats with a wide conversion to it, as Linux does.
  • The per-character step calls this library's mbrtowc (which matches the titles' on every byte value) with the state kept for the field, so the lead-byte and invalid-byte cases fall out of the same call.
  • PeekWide, the UTF-8 decoder and its throws are gone. InWideScanset compares bytes with the range rule above.
  • Wide detection uses the scanner's own IsWideScan, so %hS and %lS count too.
  • The wide tests in Formatting.cpp and GuestWideStreamScan.cpp now run on Windows as well. Their UTF-8 cases are replaced by measured title results.

Unchanged and not measured: sscanf_s sizes (that's #9087's), positional arguments next to a wide conversion (still thrown; #11726).

Tested

Windows 11 Enterprise 10.0.22621, Ryzen 9 5950X, WinLibs GCC 15.2.0 UCRT r7, Ninja, Release, BUILD_TESTING=ON, on #11724's head 6a95925815.

  • Against titles' own libc.prx: I loaded PPSA07923's, PPSA05916's and PPSA13934's libc.prx natively and called their sscanf by NID (1Pk0qZQGeWo) side by side with this build. I compared the return value, errno and every stored unit.

    • 318 hand-picked pairs (the feat(libc): read 16-bit wide fields in fscanf on Linux #11724 tables, %S/%C with lengths and widths, and 16 %l[ range cases), excluding sscanf_s sizes: 0 differences. main differs on most of them and throws on %S/%C.

    • 200,000 random pairs each, built from a b c, a space, 5 and the bytes 0x7F, 0x80, 0xA9, 0xBF, 0xC0, 0xC3, 0xE2, 0xEF, 0xF0 and 0xFF, over 17 wide formats:

      PPSA07923 PPSA05916 PPSA13934
      main (without %S/%C, which abort it) 113,215 differences
      this branch 0 0 0

      All three were run on the final head d17f1fbbae.

  • Tests: guest_formatting, guest_wide_stream_scan, guest_stdio_variadic, guest_scan_count and guest_wide_formatting pass in each of 3 runs.

  • Negative controls:

  • Full CTest (2,612 tests, GPU drivers hidden; on 8c4c3f6376, before the range commit): the same failures as main, plus guest_raise_exception, which fails some runs on main too and passes 4 of 5 here.

The initial results above are Windows measurements; later Linux qualification is recorded below. macOS runtime remains untested.

Overlap: tools/pr_overlap.py reports:

I offered on #11724 to leave this to cocoa55 if they'd rather fold it in, and the offer stands. I'll rebase after #11724 and whichever of the others lands first.

Current Linux qualification — 2026-10-29

Restored StringSource::Remaining() in a84100962ceb61a1646c04aa62bdcc254c8cec01, a one-line fast-forward on d9cf30a7a26ebd847a1aefb24d33037d3decd706. The existing prefix and range code from #12530 can now retain its two calls; no scanner behavior or feature is widened.

Linux Release: five focused formatting/scanning suites pass three repetitions on this public one-line fix (15 passes, zero skips). A private source tree exactly matching the merge of this fix with #12530 3eca2be06731a86e6dc6a7cf07259dc1378eb149 builds and passes the same15checks. Removing only the restored accessor from that combined production makes libc fail to compile at the Remaining calls. Restoring it passes all15checks again and reproduces the initial combined libc SHA256. Conventions against current main157b and whitespace pass. The private combined tree was not published; the public accessor fix was followed by the current-main integration described below.

No new Windows/macOS, title/reference-library or GPU result is claimed. The #11724 author confirmed the separate stacked follow-up; current-main integration is described below. AI-assisted: yes.

Current-main integration follow-up: 068725777be07527eda51e086d8850500b7b8707 includes main 157b02c4f18, preserving the actual #11724 ancestry and restored accessor. The resolution retains the measured cross-platform wide routing and main’s quad-length guard before Linux string-scan dispatch, without restoring the obsolete wide-scan refusal. All five suites pass three times on this integrated source (15 further passes); both formatting translation units pass MinGW cross syntax checks. Conventions and whitespace pass. The private #12530 experiment remains unpublished.

For my own PRs, the landing order is #12507 → #12530 → #12468. The retained accessor also permits the two later #12530 calls to compile if branches are combined in the other order. The #11724 author subsequently confirmed this separate stacked follow-up.

Evidence: immutable Linux qualification bundle. Its SHA-256 is ca6b35926bcc6d3f21a9650517894c86b65594e2e2b653c5c32cd1c1de89c835. No fresh Windows runtime execution is claimed; older measurements retain their recorded source pins.

AI-assisted: yes (Codex).

Current qualification — 2026-10-30

Integrated upstream main 08a10d225306ad102b57408e2ef7566bcfb829b2 at abe5a9d8036e7174a7c7664cde8e4c2f5a5ac3c4, retaining the actual previous head and main as parents. Manual conflict resolution changes only debt documentation; production and test files equal the automatic merge output. Both parents' added test lines and main gitlinks are preserved.

Linux CPU: 15 passes (guest_formatting, guest_scan_count, guest_wide_stream_scan, guest_stdio_variadic, guest_crt); public_headers 1 passes; no skips. MinGW source syntax and conventions pass.

The reviewed StringSource::Remaining() accessor and actual parent history are retained. Sibling scanner ordering remains a separate integration limit.

No new Windows/macOS runtime, native SDK, game, GPU or performance claim. The Windows user pause remains in force.

Evidence: immutable source and qualification receipts, SHA-256 61f5500e7526ea4e107159927de78c37db7e0ac1cb7ebd8625b22ab0ad5d9bd7 (download verified).

Linux stream-failure correction — 2026-10-30, head 3a27a5b8159c4899f5a7cf9f097ae0b880eb1771

The previously unmeasured stream-position behavior is now measured against two bundled SDK libc modules (FAR and Tetris). For input C3 A9, %ls leaves both bytes unread after the encoding failure; the previous head consumed C3. The correction restores the held lead byte and follows the SDK's signed-byte pushback behavior, including FF being consumed as the EOF sentinel while any held lead byte is restored. Width/end-of-input stops keep their measured consumption. Return values, stored units and %n are preserved.

  • All 256 bytes after empty, ASCII and UTF-8-lead prefixes, followed by z, across 14 formats: 10,752 ordinary stream rows and 10,752 secure stream rows. Both references agree on every field, including complete remaining input. Previous head abe5a9d8036e7174a7c7664cde8e4c2f5a5ac3c4 differs in 3,056 rows per mode; the correction has zero differences.
  • Another 1,288 string/stream rows (ordinary and secure) have zero differences after correction. Rows overlap; these are not unique-behavior counts. Secure capacities fit the fields; broader size constraints remain fix(libc): follow snprintf_s and sscanf_s constraints as measured on a PS5 #9087's.
  • 96 direct/va-list ordinary/secure fixture cases check guarded output, guest errno, %n, logical stream position and all remaining bytes. The final fixture fails the previous library's stream-position assertion and passes the correction. The older test expecting only A9 unread now checks C3 A9.
  • Four focused suites pass, zero skips: guest_wide_stream_scan, guest_formatting, guest_scan_count, guest_stdio_variadic. MinGW production/fixture syntax, conventions and whitespace pass.

CPU-only Linux qualification; no new Windows/macOS runtime, console hardware, GPU, title or FPS result. Source parent and dependency order are unchanged.

Synthetic probes, exact patch, numeric reference outputs and qualification logs, SHA-256 36a482b7b69ade13d5b9d091196c47338956e7f8000dba7c708c85e86d954f75, uploaded/downloaded and verified. No SDK binaries or game assets included. AI-assisted: yes.

Independent Windows validation: bunbeans2-arch reports Windows 11 Pro 22631, i9-9900K, MSYS2 UCRT64 GCC16.1, Release, main f7f03daf52 plus this exact 3a27a5b815 diff. Both guest_wide_stream_scan and guest_formatting pass5/5; keeping the tests while restoring main production makes both fail5/5; plain-main tests pass3/3. This is the contributor's reported run, not a Windows run performed in this Linux session. The exact head has a subsequent approved review with no blocking code issue.

Current-main qualification — 2026-10-30, repeat 3

Rebased to 8a0ba602095aa5a280dc363263ae8f914e3b53be on tested main d2863131efb0e41c116e4197841b6a3086ce087a, using an exact lease from 3a27a5b8159c4899f5a7cf9f097ae0b880eb1771. Seven remaining topic commits retain their authorship and order. The source union preserves main's measured q-directive cutoff before ordinary sscanf/vsscanf dispatch, together with this topic's bytewise 16-bit wide-field path on every host. Stream and secure-form routing is retained. The published ScanWideField.cpp is byte-identical, including range handling; Remaining and invalid-character pushback remain. Main's locking, EOF/read-error errno fixtures, portable temporary-file helper and adjacent debt notes are preserved.

Fresh Linux Release libc and test builds pass guest_formatting, guest_wide_formatting, guest_wide_stream_scan, guest_scan_count, guest_stdio_variadic, guest_crt and file_streams three times each: 21 passes, plus public_headers once, zero skips. Four MinGW production/fixture syntax checks pass. Conventions, whitespace, authored-series and submodule-pointer checks pass.

Later main 6b48aee5d2ea9fce6155163d78f8ce8dc84f7827 leaves every topic file unchanged and has a clean merge preview; it was not executed. Earlier Windows, title-library/reference and negative-control measurements keep their original pins. This refresh adds Linux execution and MinGW syntax qualification; it adds no new Windows/macOS runtime, GPU, title or performance claim. AI-assisted: yes (Codex).

Immutable source, build and test receipts, SHA-256 784c7afd95828e9d3ec65614c80683f1ed09ce1874c0fa685b3f96c48b25dba0 (download verified).

The separate inherited macOS write-watch configuration failure is tracked by pending #13220. A maintainer closed #13293 as a duplicate of that earlier identical fix. This scanner refresh does not claim a macOS runtime pass.

Checklist

🤖 Generated with Claude Code

Dependency coordination confirmed: cocoa55 explicitly agreed that #11724 retains the stream-wide conversion work and #12468 changes strings and streams to the measured byte-by-byte rule on top (author reply). Required predecessors remain #11724 and #12530; the owned-branch order remains #12507 → #12530 → #12468. These dependencies are not claimed merged.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Progress report (ba62780 → dd88656)

📈 System libraries: 26.25% (+0.01%, +3 implemented functions)

✅ 3 implemented
Library Name
libc fscanf_nid_postfix
libc vfscanf_nid_postfix
libc vfscanf_s_nid_postfix

Changed libraries (implemented exports)

Library Before After Change
libc 1489/1505 (98.94%) 1492/1505 (99.14%) +0.20%

@MarcZX10

Copy link
Copy Markdown
Contributor

I syntax-compiled the exact-head Formatting.cpp, ScanWideField.cpp, Formatting.cpp test file and GuestWideStreamScan.cpp with GCC 12/C++20 on Linux; all four compilation checks passed. I did not run the runtime matrix in this environment, so the title-library comparisons and behavioral test runs remain the author-reported evidence in the PR. Hosted policy/build checks are still queued.

@github-actions

github-actions Bot commented Oct 28, 2026 •

Copy link
Copy Markdown

Other open pull requests touch the same code. Merging one will break the other, or both implement the same thing. If this one builds on them, list them in Depends on.

Pull request Conflicting files (lines on main) Same exports Same new files
PR 9863 core/libs/tests/GuestWideStreamScan.cpp L164-165

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 5021 core/libs/tests/Formatting.cpp
docs/dev/debt/libc.md
PR 5303 docs/dev/debt/libc.md
PR 7447 docs/dev/debt/libc.md
PR 7715 docs/dev/debt/libc.md
PR 8150 docs/dev/debt/libc.md
PR 8417 docs/dev/debt/libc.md
PR 8965 docs/dev/debt/libc.md
PR 9371 docs/dev/debt/libc.md
PR 9447 docs/dev/debt/libc.md
PR 9526 docs/dev/debt/libc.md
PR 9929 docs/dev/debt/libc.md
PR 10405 docs/dev/debt/libc.md
PR 10987 docs/dev/debt/libc.md
PR 11172 docs/dev/debt/libc.md
PR 11260 docs/dev/debt/libc.md
PR 11262 docs/dev/debt/libc.md
PR 11266 docs/dev/debt/libc.md
PR 11632 docs/dev/debt/libc.md
PR 11634 docs/dev/debt/libc.md
PR 11831 docs/dev/debt/libc.md
PR 12560 docs/dev/debt/libc.md
PR 12682 docs/dev/debt/libc.md
PR 12688 docs/dev/debt/libc.md
PR 12700 docs/dev/debt/libc.md
PR 12965 docs/dev/debt/libc.md
PR 13390 docs/dev/debt/libc.md
PR 13804 docs/dev/debt/libc.md
PR 14085 docs/dev/debt/libc.md
PR 14212 docs/dev/debt/libc.md

@oneandonlydean oneandonlydean self-assigned this Oct 28, 2026

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found one concrete regression in the exact head d17f1fbbae26 while tracing the new narrow-format wide conversion path. In ScanSource, IsWideScan includes uppercase %C, but the new branch immediately executes if (conversion == 'C') return finish(assigned);. That returns before reading input or consuming the destination argument. A format containing %C followed by another conversion therefore stops early and leaves the %C input unread; a standalone %C reports zero assignments. The previous scanner explicitly mapped %C to the wide-character (c) path, and the PR's updated debt text removes %C from the unsupported list. Please implement the %C conversion (including its correct field width and secure-scan behavior) or explicitly reject it consistently, and add a regression covering input consumption and the following directive. I did not independently run the focused test binary in this session; this finding follows directly from the current control flow and the prior implementation.

@JJenkx

JJenkx commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

@MarcZX10 thanks for tracing it. The early return is intentional: it's what the titles' libc.prx does. To be sure, I re-measured it natively: a scratch loader calls the sscanf (1Pk0qZQGeWo) and sscanf_s (24m4Z4bUaoY) of PPSA07923 and PPSA05916 by NID. Both titles and both functions give identical results:

input format result what happens
"xy" %C, %2C, %lC, %hC 0 destination untouched
"" %C 0, not EOF
"x5" %C%d 0 the %d destination is untouched
"x5" %*C%d 0 the %d destination is untouched
"xy" %C%n 0 %n is not written
"5x" %d%C 1
"5 x" %d %C%d 1 the second %d is untouched

So in the narrow scanners %C isn't a conversion: it ends the scan with the count so far, reads no input, and takes no argument. Any width or length is ignored, and sscanf_s behaves the same. main's old mapping of %C to the wide %lc path ("%S %C" returning 2) doesn't match the titles. The PR's debt entry for the scanf family ends with "%C ends the scan" for that reason.

Pushed d9cf30a7a2 (test-only, fast-forward on d17f1fbbae). It adds the regressions you asked for, using exactly the cases above: input consumption through %n, the following directive, empty input, %*C, %lC/%hC, and the same cases through vsscanf_s.

Tested on Windows 11, WinLibs GCC 15.2.0 UCRT r7, Release, this exact head: guest_formatting, guest_stdio_variadic, guest_scan_count, guest_wide_stream_scan, guest_wide_formatting, file_streams and guest_standard_streams* pass 3/3, and conventions pass. With the scanner sources of #11724's head 6a95925815 and this test file, guest_formatting fails. That file also holds this PR's earlier wide-field tests, though, so the control isn't specific to the new %C lines.

AI-assisted: yes (Claude Code).

@JJenkx

JJenkx commented Oct 28, 2026

Copy link
Copy Markdown
Contributor Author

Linux run at this PR's exact head d9cf30a7a2 (d9cf30a7a26ebd847a1aefb24d33037d3decd706), not merged onto anything else; its base on main is d093ea9ced.

Arch Linux, kernel 7.2.9, Ryzen 9 7950X, g++ (GCC) 16.2.1 20260810, glibc 2.44, x86-64. Release, BUILD_TESTING=ON, Ninja -j4, private worktree.

  • ctest -R '^guest_(formatting|wide_stream_scan)$': passes in each of 3 runs (6 passes).

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked — approved

The %C concern from my earlier review is resolved. The title-library measurements you added show that narrow %C deliberately ends the scan without reading input or consuming an argument; the new cases cover the following conversion, %n, empty input, suppression, width/length modifiers, and vsscanf_s. Thanks for adding those exact-head regressions.

I also checked the follow-up Linux report: guest_formatting and guest_wide_stream_scan passed three runs each on the exact PR head. The PR checks report the Windows build, relinker on Ubuntu/Windows/macOS (including x86_64), ASan, register-context, write-watch, eligibility, conventions, labels, and report jobs passing; Linux hosted CI was skipped. The full library test matrix remains supported by the author-reported Windows/Linux test runs in the PR description and comments.

The PR is mergeable against current main, and the checklist records its dependency on #11724. I did not run local tests in this review.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 29, 2026
@JJenkx

JJenkx commented Oct 29, 2026

Copy link
Copy Markdown
Contributor Author

Restored StringSource::Remaining() in a84100962ceb61a1646c04aa62bdcc254c8cec01, a one-line fast-forward on d9cf30a7a26ebd847a1aefb24d33037d3decd706. The existing prefix and range code from #12530 can now retain its two calls; no scanner behavior or feature is widened.

Linux Release: five focused formatting/scanning suites pass three repetitions on this public one-line fix (15 passes, zero skips). A private source tree exactly matching the merge of this fix with #12530 3eca2be06731a86e6dc6a7cf07259dc1378eb149 builds and passes the same15checks. Removing only the restored accessor from that combined production makes libc fail to compile at the Remaining calls. Restoring it passes all15checks again and reproduces the initial combined libc SHA256. Conventions against current main157b and whitespace pass. The private combined tree was not published; the public accessor fix was followed by the current-main integration described below.

No new Windows/macOS, title/reference-library or GPU result is claimed. The external #11724 author coordination remains pending; current-main integration is described below. AI-assisted: yes.

Current-main integration follow-up: 068725777be07527eda51e086d8850500b7b8707 includes main 157b02c4f18, preserving the actual #11724 ancestry and restored accessor. The resolution retains the measured cross-platform wide routing and main’s quad-length guard before Linux string-scan dispatch, without restoring the obsolete wide-scan refusal. All five suites pass three times on this integrated source (15 further passes); both formatting translation units pass MinGW cross syntax checks. Conventions and whitespace pass. The private #12530 experiment remains unpublished.

For my own PRs, the landing order is #12507 → #12530 → #12468. The retained accessor also permits the two later #12530 calls to compile if branches are combined in the other order. This does not claim external #11724 author agreement.

Evidence: immutable Linux qualification bundle. Its SHA-256 is ca6b35926bcc6d3f21a9650517894c86b65594e2e2b653c5c32cd1c1de89c835. No fresh Windows runtime execution is claimed; older measurements retain their recorded source pins.

@github-actions github-actions Bot added needs rebase Conflicts with main, rebase needed and removed needs rebase Conflicts with main, rebase needed labels Oct 29, 2026
JJenkx added a commit to JJenkx/AnyPS5 that referenced this pull request Oct 30, 2026
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 30, 2026
@JJenkx

JJenkx commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Integrated upstream main 08a10d225306ad102b57408e2ef7566bcfb829b2 at abe5a9d8036e7174a7c7664cde8e4c2f5a5ac3c4, retaining the actual previous head and main as parents. Manual conflict resolution changes only debt documentation; production and test files equal the automatic merge output. Both parents' added test lines and main gitlinks are preserved.

Linux CPU: 15 passes (guest_formatting, guest_scan_count, guest_wide_stream_scan, guest_stdio_variadic, guest_crt); public_headers 1 passes; no skips. MinGW source syntax and conventions pass.

The reviewed StringSource::Remaining() accessor and actual parent history are retained. Sibling scanner ordering remains a separate integration limit.

No new Windows/macOS runtime, native SDK, game, GPU or performance claim. The Windows user pause remains in force.

Evidence: immutable source and qualification receipts, SHA-256 61f5500e7526ea4e107159927de78c37db7e0ac1cb7ebd8625b22ab0ad5d9bd7 (download verified).

AI-assisted: yes.

@MarcZX10

Copy link
Copy Markdown
Contributor

Re-checked — new head, dependency order pending

The current-main integration is clean and the author reports focused checks passing, but the PR’s declared dependencies and landing order are still unresolved.

What I ran

Checked head abe5a9d8036e7174a7c7664cde8e4c2f5a5ac3c4 against current main 5ecef095b61e7a406b279c97896f3a8a6fbd6353; the merge preview is clean (tree 34e6189111fe03ff9f47a2d4d629f530f7faca68).

git merge-tree --write-tree refs/remotes/upstream/main refs/remotes/upstream/pr-12468
git diff --check 08a10d225306ad102b57408e2ef7566bcfb829b2...refs/remotes/upstream/pr-12468

Results

Check Result
Merge preview Clean
Diff check Passed; no output
Focused Linux and MinGW checks Reported by the author on this head; not independently run
Hosted checks Completed Linux relinker, Windows relinker, ASan, register-context, Windows, and conventions checks passed; macOS jobs are queued; product Linux job skipped

Not verified

I did not rerun the reported Linux tests or MinGW syntax check.

To get this to a good state

The PR lists #11724 and #12530 as dependencies and gives the landing order #12507 → #12530 → #12468; its description says coordination with #11724’s author is still pending. Please update the dependency/order status once confirmed. I’ll re-check this head and the same commands after that update.

@JJenkx

JJenkx commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

@MarcZX10 The external author agreement was already present but missing from our current description: cocoa55 explicitly confirmed keeping #11724 and #12468 separate, with this byte-by-byte follow-up stacked on #11724. I have corrected the stale “coordination pending” wording and linked that confirmation.

The required predecessors remain #11724 and #12530, with the owned order #12507 → #12530 → #12468. This resolves the missing author-coordination statement; it does not claim those dependencies have landed. No source or branch head changed, and no new test result is claimed.

AI-assisted: yes.

@JJenkx

JJenkx commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

@MarcZX10 Corrected the previously unmeasured stream-failure consumption in 3a27a5b8159c4899f5a7cf9f097ae0b880eb1771. Both bundled SDK references leave the complete invalid multibyte character unread; the old branch consumed its lead byte. The correction matches both references in 21,504 exhaustive ordinary/secure stream rows (including all remaining input), plus 1,288 additional string/stream rows. Old head differs in 3,056 exhaustive rows per stream mode; the correction has zero differences.

96 direct/va-list ordinary/secure fixture cases fail the previous library and pass the change. Four focused CPU suites pass, zero skips; MinGW production/fixture syntax, conventions and whitespace pass. No new Windows runtime or title/FPS claim. The author-agreed dependency order from the prior reply is unchanged.

Numeric/synthetic evidence, SHA-256 36a482b7b69ade13d5b9d091196c47338956e7f8000dba7c708c85e86d954f75, uploaded/downloaded and verified. The description includes the measurements and removes the stale unmeasured-stream-position statement. AI-assisted: yes.

@bunbeans2-arch

Copy link
Copy Markdown
Contributor

A Windows run with GCC 16: Windows 11 Pro 10.0.22631, i9-9900K, MSYS2 UCRT64 GCC 16.1.0, Release, main f7f03da with this PR's diff (3a27a5b815) applied.

  • With this PR: guest_wide_stream_scan and guest_formatting passed 5 of 5.
  • Same tree with the non-test files put back to main (this PR's tests kept, ScanWideField.cpp removed): both failed 5 of 5. guest_wide_stream_scan failed at the new check wide field failure keeps measured result, count and stored units, and guest_formatting hit vsscanf format conversion not implemented from main's library.
  • main f7f03da alone (its own tests and source): both passed 3 of 3.

AI-assisted: yes (Claude Code)

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-checked — new head, looks good

I reviewed exact head 3a27a5b8159c4899f5a7cf9f097ae0b880eb1771; the author reports the focused Windows tests passing 5/5 on current main f7f03daf52f11191174b1be089a56e268390a707 with this diff, and both red controls failing as expected. Hosted Windows, conventions, eligibility, labels, report, Ubuntu relinker, relinker ASan, and Ubuntu register-context checks pass; Linux product tests are skipped, and the macOS jobs remain pending. I did not rerun the C++ tests locally or verify console behavior. No blocking issue remains in the reviewed change.

@JJenkx

JJenkx commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Both failed macOS write-watch jobs in run 41016170432 stop at CMake configure, before building or executing scanner/write-watch fixtures. The tested integration has two identical fault-mode registration loops; current main 51c31 reproduces the duplicate-name diagnostics, while this PR public 3a27a5 has only one loop. A separate main-only fix removes the redundant loop, preserving all 18 unique tests and both fault modes. Actual before/after configure/generate proof includes Darwin cross-configuration on Linux with APPLE=1. No macOS runtime pass or scanner branch fix is claimed. Fix PR: #13293.

Evidence: immutable qualification receipts, SHA-256 8cece16d735c276255db5e3c6617a50957dc86e743d12ccf894c579fb52dacd8 (download verified).

AI-assisted: yes (Codex).

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 30, 2026
cocoa55 and others added 7 commits October 30, 2026 18:50
fscanf, vfscanf, scanf, vscanf and the _s stream forms threw on %ls,
%lc, %l[, %S and %C on Linux. Route those formats through the guest
scanner sscanf already uses, reading the stream through StreamSource.

The wide field reader now works on any source through Peek and Skip
instead of a contiguous buffer: it decides whether to take a character
from its lead byte, then consumes the whole UTF-8 sequence, so a
rejected character stays in the stream without multi-byte pushback.

AI-assisted: yes
Deciding from the lead byte only holds while scanset members are ASCII.
Each source now decodes its next character with PeekWide without
consuming it, and the field reader skips its bytes only once it takes
the character. StreamSource peeks by reading the UTF-8 sequence and
pushing it back with ungetc, as the fwscanf stream reader in
ScanningWide.cpp already does. Keep TSource::Contiguous.

AI-assisted: yes
Titles' libc.prx (PPSA07923, PPSA05916, PPSA13934) reads %ls, %lc, %l[
and %S by passing each byte to mbrtowc with the state kept for the
field, so in the C locale any byte from 0x80 up ends the conversion:

- a UTF-8 lead byte is held, and the next byte the field takes fails it
  with EILSEQ as a matching failure;
- a field that ends while a byte is held is an input failure;
- any other byte mbrtowc refuses is an input failure as the field's
  first byte and a matching failure after it;
- the units read before a failure stay stored without a terminator.

Widths count bytes, scanset members are compared as bytes, and %C is
not a conversion there, so it ends the scan.

The guest scanner decoded UTF-8 instead and threw for anything else, and
Windows handed these conversions to the host, which stored each byte as
one unit and threw for %S and %C. The scanner now does the above on every
platform, and Windows sends formats with a wide conversion to it, as
Linux does. Wide detection now follows the scanner's own rule, so %hS and
%lS count. The wide tests run on Windows too and expect the titles'
results; the UTF-8 cases they had are replaced by measured ones.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The narrow sscanf of titles' libc.prx (PPSA07923 and PPSA05916 agree)
reads a %l[ set left to right: X-Y is a byte range when a - follows a
member and is not last, a reversed range matches nothing, and a - first,
last or after a range is a member. InWideScanset made a reversed range's
first end a member and chained a following -, so %l[z-a] took z and
%l[a-a-c] took b. It now follows the rule swscanf's InSet uses.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ater directives

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@JJenkx
JJenkx force-pushed the windows/libc-wide-scan-c-locale branch from 3a27a5b to 8a0ba60 Compare October 30, 2026 22:59
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 30, 2026
@JJenkx

JJenkx commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Current-main qualification — 2026-10-30, repeat 3

Rebased to 8a0ba602095aa5a280dc363263ae8f914e3b53be on tested main d2863131efb0e41c116e4197841b6a3086ce087a, using an exact lease from 3a27a5b8159c4899f5a7cf9f097ae0b880eb1771. Seven remaining topic commits retain their authorship and order. The source union preserves main's measured q-directive cutoff before ordinary sscanf/vsscanf dispatch, together with this topic's bytewise 16-bit wide-field path on every host. Stream and secure-form routing is retained. The published ScanWideField.cpp is byte-identical, including range handling; Remaining and invalid-character pushback remain. Main's locking, EOF/read-error errno fixtures, portable temporary-file helper and adjacent debt notes are preserved.

Fresh Linux Release libc and test builds pass guest_formatting, guest_wide_formatting, guest_wide_stream_scan, guest_scan_count, guest_stdio_variadic, guest_crt and file_streams three times each: 21 passes, plus public_headers once, zero skips. Four MinGW production/fixture syntax checks pass. Conventions, whitespace, authored-series and submodule-pointer checks pass.

Later main 6b48aee5d2ea9fce6155163d78f8ce8dc84f7827 leaves every topic file unchanged and has a clean merge preview; it was not executed. Earlier Windows, title-library/reference and negative-control measurements keep their original pins. This refresh adds Linux execution and MinGW syntax qualification; it adds no new Windows/macOS runtime, GPU, title or performance claim. AI-assisted: yes (Codex).

Immutable source, build and test receipts, SHA-256 784c7afd95828e9d3ec65614c80683f1ed09ce1874c0fa685b3f96c48b25dba0 (download verified).

The separate inherited macOS write-watch configuration failure is tracked by pending #13220. A maintainer closed #13293 as a duplicate of that earlier identical fix. This scanner refresh does not claim a macOS runtime pass.

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Nov 1, 2026
@JJenkx

JJenkx commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Linux run at exact head dd886561500787ed26783721924ec950fc1fbeda, today's merge of main, with nothing else merged in. Arch Linux, kernel 7.2.11, Ryzen 9 7950X, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON. The hosted linux job was skipped for this head.

  • Build: full build passes. None of the build warnings is in this PR's files.
  • Tests: ctest -R "guest_wide_stream_scan|guest_formatting|guest_scan_count|guest_stdio_variadic|scanf|wide|stdio|libc_|guest_string" -j8 gives 26/26 passing, with no skips. That includes the four focused suites named in the description: guest_wide_stream_scan, guest_formatting, guest_scan_count and guest_stdio_variadic.

CPU-only. No title, GPU or macOS run.

@JJenkx

JJenkx commented Nov 1, 2026

Copy link
Copy Markdown
Contributor Author

Windows exact-head check of dd88656150 for the scheduled cross-platform review.

  • Setup: Windows 11 Enterprise 10.0.22621, Ryzen 9 5950X, WinLibs GCC 15.2.0 UCRT r7, Ninja, Release, full build with BUILD_TESTING=ON. Built from this exact head; libc.prx was rebuilt for the run.
  • ctest -R "scan|wide|wchar|stdio|locale" -j8 --repeat until-fail:3: 23/23 pass.

🤖 Generated with Claude Code

@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Nov 1, 2026
@oneandonlydean
oneandonlydean merged commit 646d099 into boykopovar:main Nov 2, 2026
18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants