Skip to content

fix(tests): decode export-cache Git paths as UTF-8 - #13269

Merged
oneandonlydean merged 1 commit into
boykopovar:mainfrom
JJenkx:fix/tests-export-cache-utf8
Oct 31, 2026
Merged

oneandonlydean merged 1 commit into
boykopovar:mainfrom
JJenkx:fix/tests-export-cache-utf8

Conversation

@JJenkx

@JJenkx JJenkx commented Oct 30, 2026 •

Copy link
Copy Markdown
Contributor

What

The export-cache test helper decodes Git's UTF-8 output with the parent's locale. Under CP1252, a temporary directory containing é or Ж is misdecoded by git rev-parse --absolute-git-dir, so self.cache points at the wrong path and two tests raise FileNotFoundError.

Request UTF-8 explicitly in this one test helper, matching production pr_overlap.git. This is the additional Python fixture issue reported alongside #13083; C++ scratch-file handling is tracked separately in #13104.

Tested

Linux, Python 3.14.7, pinned main a1782bdc02c68599830868a7005fd5bced4b7894 against this one-line change. The existing seven-test suite uses synthetic local Git repositories.

Parent decoding / temporary root Main Change
Native default / ASCII 7 pass 7 pass
CP1252 model / é 2 errors 7 pass
CP1252 model / Ж 2 errors 7 pass

No skips. CP1252 is modeled at the parent subprocess decoder on Linux; native Windows execution remains untested here. Git fixtures use empty templates with hooks disabled. Conventions and whitespace checks pass.

Unmodified upstream overlap checker: all 1,470 open PRs checked against main 2f3fa0f7dc3ce81028c89c7ec17a3010f37c9eb0; no shared files or exports.

Linux qualification logs, source pins and overlap report (SHA-256 368a5cdbe83686c1401f9c46ba3576a04c60d88c624b147e12e809a93cd94bf9; upload/download verified).

Qualification update — 2026-10-30, repeat 2

Actual Windows check at exact published 35f00deb91f5749b297ab3ec39b35dba19f8385f: Windows 11 Enterprise 10.0.22621, Ryzen 9 5950X, Python 3.15 with -X utf8=0 / PYTHONUTF8=0. The unchanged seven export-cache fixture tests pass at each ASCII, é and Ж TEMP path: 21/21. Restoring only the old locale-decoding helper (text=True in place of explicit UTF-8) keeps ASCII green but reproduces two FileNotFoundError errors in each non-ASCII path. Restoring the exact head gives another 21/21 passes. This is actual Windows execution, complementing the earlier Linux CP1252 model. Exact archived tools source, raw logs, script and FIFO receipt are attached; no production behavior, SDK/game/GPU or performance claim.

Evidence: https://github.com/JJenkx/AnyPS5/releases/download/linux-validation-20261023/owned-current-windows-20261030-repeat2.tar.gz, SHA256 e423c97373758bf59239e02c138b74f35f1a45488793bc92878a4c836f94a584. AI-assisted: yes (Codex).

AI-assisted: yes (Codex).

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: n/a
  • AI-assisted: yes (Codex)

@MarcZX10 MarcZX10 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests run and reviewed — looks good

The explicit UTF-8 decoder fixes the path regression: under a CP1252-decoder emulation and a Unicode temporary root, the PR head passes while the base fails two tests.

What I ran

Reviewed head 35f00deb91f5749b297ab3ec39b35dba19f8385f against PR base/current main 3d19f3fed31be38d7196ea1bd74e29ab6cc3c9a4; a local merge preview was clean. Environment: Linux x86_64, Python 3.11.2, Git 2.39.5. I ran the changed suite in isolated worktrees pinned to the PR head and base. For the locale regression check, a temporary sitecustomize.py shim forced CP1252 only for captured text subprocesses that did not specify an encoding; LC_ALL=C.UTF-8 kept filesystem paths UTF-8-capable, and TMPDIR pointed to /tmp/AnyPS5-测试.

python3 tools/pr_precheck.py --repo boykopovar/AnyPS5 --pr 13269
python3 tools/check_conventions.py --base 3d19f3fed31be38d7196ea1bd74e29ab6cc3c9a4 --head 35f00deb91f5749b297ab3ec39b35dba19f8385f
python3 tools/check_test_registration.py --scripts
python3 -m unittest tools.tests.test_pr_overlap_export_cache

env LC_ALL=C.UTF-8 TMPDIR=/tmp/AnyPS5-测试 PYTHONPATH=/tmp/anyps5-cp1252-emulation python3 -m unittest tools.tests.test_pr_overlap_export_cache

The final two commands were run separately in worktrees at the PR head and base. In the temporary shim, subprocess.run adds encoding="cp1252" only when text=True, capture_output=True, and no encoding was passed.

Results

Check Result
Changed Python suite Passed: 7 tests at the PR head, including with a Unicode temp root.
CP1252 regression control Discriminated: base failed 2 tests with corrupted paths; PR head passed all 7.
Conventions and PR template Passed in the no-build precheck.
Test registration Passed: all 222 test scripts are registered.
Patch whitespace Passed.
Current-main merge preview Passed cleanly; tree 01a977976aca5fd25418a932953de02a67ebfcfd.
No-build precheck Core applicable checks passed. Its optional test-quality helper was absent from the isolated PR checkout, and its default-locale negative control did not emulate CP1252; I separately ran the targeted control above.
Hosted CI Eligibility, conventions, intake, and labels were still queued at the final snapshot; no hosted result is claimed.
Unedited local test output

PR head, with the CP1252 decoder emulation:

.......
----------------------------------------------------------------------
Ran 7 tests in 0.297s

OK

Base, with the same emulation:

E...E..
======================================================================
ERROR: test_a_cache_from_other_parsers_is_ignored (tools.tests.test_pr_overlap_export_cache.ExportCacheTests.test_a_cache_from_other_parsers_is_ignored)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/AnyPS5-pr13269-base/tools/tests/test_pr_overlap_export_cache.py", line 92, in test_a_cache_from_other_parsers_is_ignored
    cache = json.loads(self.cache.read_text(encoding="utf-8"))
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 1059, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 1045, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/AnyPS5-测试/tmplha18832/.git/pr_overlap-exports.json'

======================================================================
ERROR: test_an_unreadable_cache_is_rebuilt (tools.tests.test_pr_overlap_export_cache.ExportCacheTests.test_an_unreadable_cache_is_rebuilt)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/tmp/AnyPS5-pr13269-base/tools/tests/test_pr_overlap_export_cache.py", line 100, in test_an_unreadable_cache_is_rebuilt
    self.cache.write_text("{not json", encoding="utf-8")
  File "/usr/lib/python3.11/pathlib.py", line 1079, in write_text
    with self.open(mode='w', encoding=encoding, errors=errors, newline=newline) as f:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/pathlib.py", line 1045, in open
    return io.open(self, mode, buffering, encoding, errors, newline)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/AnyPS5-测试/tmpbvbtag1w/.git/pr_overlap-exports.json'

----------------------------------------------------------------------
Ran 7 tests in 0.280s

FAILED (errors=2)

Negative control

Assertion Change to code under test Did the new head fail?
The cache path is derived from Git output without mojibake when the parent locale decoder is CP1252 and the temp root contains non-ASCII characters. Ran the base version without explicit UTF-8, then the PR version with encoding="utf-8". Base: yes, 2 FileNotFoundErrors. PR head: no, all 7 tests pass.

Findings

Blocking

  • None found in this focused test-only change.

Non-blocking

  • The CP1252 case was emulated on Linux; I did not run native Windows Python.

Sources

Not verified

  • Native Windows execution and the hosted CI jobs, which were queued at the final snapshot. The author’s separate Python 3.14.7 report was not independently run.

@oneandonlydean oneandonlydean self-assigned this Oct 30, 2026
@JJenkx

JJenkx commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Linux run at this PR's exact head 35f00deb91 (35f00deb91f5749b297ab3ec39b35dba19f8385f), not merged onto anything else; its base on main is a1782bdc02.

Arch Linux, kernel 7.2.9, Ryzen 9 7950X, g++ (GCC) 16.2.1 20260810, glibc 2.44, x86-64. Release, BUILD_TESTING=ON, Ninja -j4, private worktree.

  • ctest -R '^pr_overlap_export_cache$': passes in each of 3 runs (3 passes).

pr_overlap_export_cache is the Python test this PR changes; it ran through ctest with the system Python 3.

@JJenkx

JJenkx commented Oct 30, 2026

Copy link
Copy Markdown
Contributor Author

Qualification update — 2026-10-30, repeat 2

Actual Windows check at exact published 35f00deb91f5749b297ab3ec39b35dba19f8385f: Windows 11 Enterprise 10.0.22621, Ryzen 9 5950X, Python 3.15 with -X utf8=0 / PYTHONUTF8=0. The unchanged seven export-cache fixture tests pass at each ASCII, é and Ж TEMP path: 21/21. Restoring only the old locale-decoding helper (text=True in place of explicit UTF-8) keeps ASCII green but reproduces two FileNotFoundError errors in each non-ASCII path. Restoring the exact head gives another 21/21 passes. This is actual Windows execution, complementing the earlier Linux CP1252 model. Exact archived tools source, raw logs, script and FIFO receipt are attached; no production behavior, SDK/game/GPU or performance claim.

Evidence: https://github.com/JJenkx/AnyPS5/releases/download/linux-validation-20261023/owned-current-windows-20261030-repeat2.tar.gz, SHA256 e423c97373758bf59239e02c138b74f35f1a45488793bc92878a4c836f94a584. AI-assisted: yes (Codex).

AI-assisted: yes (Codex).

@oneandonlydean
oneandonlydean merged commit 32bf44c into boykopovar:main Oct 31, 2026
15 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants