Skip to content

feat(libc): support archive-backed stdio reads - #7447

Open
qpavyros wants to merge 1 commit into
boykopovar:mainfrom
qpavyros:codex/archive-stdio
Open

qpavyros wants to merge 1 commit into
boykopovar:mainfrom
qpavyros:codex/archive-stdio

Conversation

@qpavyros

Copy link
Copy Markdown

What

Allow libc fopen and fopen_s to read entries found only in AnyPS5's optional app0.zar packaging layer, while preserving disk overrides and case-insensitive lookup.

Materialize each opened entry into a temporary file in 64 KiB chunks, reopen it with genuine read-only OS access, and use the existing native stdio implementation. This covers reads, seeking, EOF/error state, narrow scanning and wide input without a separate stdio backend. Independent opens have independent positions; duplicated descriptors share the native backing and preserve archive metadata. Close, reopen and descriptor replacement retire the metadata.

Add archive-aware access and lstat; preserve the existing archive-aware stat path. Prevent unsupported descriptor operations from acting on temporary-file metadata.

Opening decompresses the whole entry and needs temporary disk space equal to its uncompressed size for each independent open. No performance improvement or title bring-up is claimed. Directory merging, fdopen on the existing virtual kernel archive descriptors and freopen to an archive-only path remain unsupported. Host stdio buffering rules still apply when mixing streams and descriptor I/O. This is not support for the console's PFS format.

Overlap review: tools/pr_overlap.py --branch codex/archive-stdio --repo boykopovar/AnyPS5 reports documentation-only conflicts in the existing libkernel.md archive debt paragraph with #5993, #6519, #6658, #6936, #7105 and #7149. These do not provide this stdio implementation and are not prerequisites. Several older branches conflict with the base, so their shared code paths were also inspected manually: the stream reopen work, macOS ports, error handling, directory handling, aliases and equeue changes do not duplicate archive-backed stdio. In particular, #7332 changes archive state lifetime for macOS, not archive stdio reads. Documentation conflicts will need resolving when either side merges.

Tested

  • Tested after rebasing onto 1b0c7cfa02bec9eef9f55ec31c4cdd5ca087675e.
  • Windows: MinGW GCC 15.2 Release build of libc/libkernel and relevant test targets; 16 targeted CTest cases passed.
  • Linux: Ubuntu under WSL, GCC 15 Release build of libc/libkernel and relevant test targets; 17 targeted CTest cases passed.
  • Five new archive stdio cases cover compressed block boundaries, EOF, seek/tell/rewind/ungetc, empty entries, narrow scanning, wide input, disk overrides, independent opens, descriptor duplication/replacement, positioned reads, metadata, denied mutations, mode/access errors, absent archives and damaged compressed blocks.
  • Existing archive, standard stream, file stream and host stream tests passed; Linux fopen error tests passed.
  • python tools/check_conventions.py --base origin/main passed.
  • No PS5 hardware or game title was tested. These are targeted library builds and synthetic filesystem fixtures, not a complete application build or a full test-suite run.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
  • No notes, investigation .md files or images added to the repository (attach them to this PR)
  • New third-party code is a submodule built from source
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Fixes: none
  • Unblocks: none
  • GPUs tested: n/a
  • AI-assisted: yes; implementation, code review and local test execution assisted by Codex

Copilot AI balanced review requested due to automatic review settings October 18, 2026 06:58

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@github-actions

github-actions Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request Shared files not compared
PR 3965 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 3987 docs/dev/debt/libc.md
PR 4627 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/include/FileStream.hpp
PR 4630 docs/dev/debt/libkernel.md
PR 5021 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/include/FileStream.hpp
core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libc.md
docs/dev/debt/libkernel.md
PR 5303 docs/dev/debt/libc.md
PR 6044 core/libs/prx/libc/CMakeLists.txt
PR 6084 core/libs/prx/libc/CMakeLists.txt
PR 6085 core/libs/prx/libc/CMakeLists.txt
PR 6086 core/libs/prx/libc/CMakeLists.txt
PR 6149 docs/dev/debt/libkernel.md
PR 6469 docs/dev/debt/libc.md
docs/dev/debt/libkernel.md
PR 6565 docs/dev/debt/libc.md
PR 6586 docs/dev/debt/libkernel.md
PR 6713 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 6740 docs/dev/debt/libkernel.md
PR 6819 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 6830 docs/dev/debt/libkernel.md
PR 6879 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libc.md
docs/dev/debt/libkernel.md
PR 6936 core/libs/prx/libkernel/File/src/ArchivedFile.cpp
core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 7049 docs/dev/debt/libkernel.md
PR 7081 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/src/FileAndHeap.cpp
docs/dev/debt/libc.md
PR 7088 docs/dev/debt/libkernel.md
PR 7090 docs/dev/debt/libkernel.md
PR 7105 docs/dev/debt/libkernel.md
PR 7121 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 7135 docs/dev/debt/libkernel.md
PR 7149 docs/dev/debt/libkernel.md
PR 7192 docs/dev/debt/libkernel.md
PR 7201 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/include/FileStream.hpp
core/libs/prx/libc/src/FileAndHeap.cpp
core/libs/prx/libkernel/File/src/Stdio.cpp
PR 7386 docs/dev/debt/libc.md
PR 7464 docs/dev/debt/libkernel.md
PR 7509 docs/dev/debt/libkernel.md
PR 7511 docs/dev/debt/libkernel.md
PR 7530 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 7532 docs/dev/debt/libkernel.md
PR 7544 docs/dev/debt/libc.md
PR 7565 docs/dev/debt/libkernel.md
PR 7605 docs/dev/debt/libkernel.md
PR 7648 docs/dev/debt/libc.md
PR 7653 docs/dev/debt/libkernel.md
PR 7715 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 7804 docs/dev/debt/libkernel.md
PR 7851 core/libs/prx/libc/CMakeLists.txt
PR 7853 docs/dev/debt/libkernel.md
PR 7937 docs/dev/debt/libkernel.md
PR 7965 docs/dev/debt/libkernel.md
PR 8010 docs/dev/debt/libkernel.md
PR 8087 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8150 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8279 docs/dev/debt/libc.md
PR 8284 docs/dev/debt/libc.md
PR 8287 docs/dev/debt/libc.md
PR 8306 docs/dev/debt/libc.md
PR 8311 core/libs/prx/libc/CMakeLists.txt
PR 8344 docs/dev/debt/libkernel.md
PR 8359 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 8373 docs/dev/debt/libkernel.md
PR 8376 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8384 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8397 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8408 docs/dev/debt/libkernel.md
PR 8417 docs/dev/debt/libc.md
PR 8448 docs/dev/debt/libkernel.md
PR 8483 docs/dev/debt/libkernel.md
PR 8499 docs/dev/debt/libkernel.md
PR 8509 core/libs/prx/libc/src/FileAndHeap.cpp
PR 8533 docs/dev/debt/libkernel.md
PR 8534 docs/dev/debt/libkernel.md
PR 8537 docs/dev/debt/libkernel.md
PR 8544 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 8546 docs/dev/debt/libc.md
PR 8547 docs/dev/debt/libkernel.md
PR 8550 core/libs/prx/libc/include/FileStream.hpp
core/libs/prx/libc/src/FileAndHeap.cpp
docs/dev/debt/libc.md
PR 8553 docs/dev/debt/libkernel.md
PR 8605 docs/dev/debt/libkernel.md
PR 8641 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 8658 core/libs/prx/libc/CMakeLists.txt
core/libs/prx/libc/include/FileStream.hpp
core/libs/prx/libc/src/FileAndHeap.cpp
core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 8670 docs/dev/debt/libkernel.md
PR 8687 core/libs/prx/libkernel/File/src/Stdio.cpp
PR 8700 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 8720 core/libs/prx/libc/src/FileAndHeap.cpp
docs/dev/debt/libc.md
PR 8725 docs/dev/debt/libkernel.md
PR 8726 docs/dev/debt/libkernel.md
PR 8728 docs/dev/debt/libkernel.md
PR 8730 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8742 docs/dev/debt/libkernel.md
PR 8797 core/libs/prx/libkernel/File/src/Stdio.cpp
PR 8855 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8858 docs/dev/debt/libkernel.md
PR 8863 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8864 docs/dev/debt/libc.md
PR 8865 docs/dev/debt/libc.md
PR 8866 docs/dev/debt/libkernel.md
PR 8872 docs/dev/debt/libc.md
docs/dev/debt/libkernel.md
PR 8880 docs/dev/debt/libc.md
PR 8890 docs/dev/debt/libkernel.md
PR 8894 docs/dev/debt/libkernel.md
PR 8901 core/libs/prx/libc/CMakeLists.txt
PR 8903 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8906 docs/dev/debt/libkernel.md
PR 8910 docs/dev/debt/libkernel.md
PR 8916 docs/dev/debt/libkernel.md
PR 8920 core/libs/prx/libc/src/FileAndHeap.cpp
PR 8921 docs/dev/debt/libkernel.md
PR 8937 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8965 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8974 core/libs/prx/libc/CMakeLists.txt
PR 8977 docs/dev/debt/libkernel.md
PR 8979 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 8985 docs/dev/debt/libkernel.md
PR 8986 docs/dev/debt/libkernel.md
PR 8988 docs/dev/debt/libkernel.md
PR 9001 docs/dev/debt/libc.md
docs/dev/debt/libkernel.md
PR 9005 docs/dev/debt/libkernel.md
PR 9019 core/libs/prx/libkernel/File/src/Stdio.cpp
PR 9038 docs/dev/debt/libc.md
PR 9059 docs/dev/debt/libkernel.md
PR 9060 docs/dev/debt/libkernel.md
PR 9071 docs/dev/debt/libc.md
PR 9074 docs/dev/debt/libkernel.md
PR 9087 docs/dev/debt/libc.md
PR 9095 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 9099 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 9101 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 9104 docs/dev/debt/libkernel.md
PR 9105 docs/dev/debt/libkernel.md
PR 9108 docs/dev/debt/libkernel.md
PR 9109 docs/dev/debt/libc.md
docs/dev/debt/libkernel.md
PR 9123 docs/dev/debt/libkernel.md
PR 9127 docs/dev/debt/libkernel.md
PR 9133 docs/dev/debt/libc.md
docs/dev/debt/libkernel.md
PR 9134 docs/dev/debt/libkernel.md
PR 9153 docs/dev/debt/libkernel.md
PR 9156 docs/dev/debt/libkernel.md
PR 9160 docs/dev/debt/libkernel.md
PR 9161 core/libs/prx/libkernel/File/src/Stdio.cpp
PR 9163 core/libs/prx/libc/CMakeLists.txt
PR 9164 docs/dev/debt/libkernel.md
PR 9165 docs/dev/debt/libkernel.md
PR 9167 docs/dev/debt/libkernel.md
PR 9170 docs/dev/debt/libc.md
PR 9181 docs/dev/debt/libc.md
PR 9187 docs/dev/debt/libkernel.md
PR 9199 docs/dev/debt/libkernel.md
PR 9216 docs/dev/debt/libc.md
PR 9218 docs/dev/debt/libkernel.md
PR 9223 core/libs/prx/libc/include/FileStream.hpp
PR 9232 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 9239 docs/dev/debt/libkernel.md
PR 9242 docs/dev/debt/libkernel.md
PR 9256 docs/dev/debt/libkernel.md
PR 9263 docs/dev/debt/libkernel.md
PR 9265 docs/dev/debt/libkernel.md
PR 9269 docs/dev/debt/libc.md
PR 9273 docs/dev/debt/libkernel.md
PR 9284 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 9293 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 9296 docs/dev/debt/libc.md
PR 9297 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 9298 docs/dev/debt/libkernel.md
PR 9302 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 9303 core/libs/prx/libkernel/File/src/Stdio.cpp
docs/dev/debt/libkernel.md
PR 9314 docs/dev/debt/libkernel.md
PR 9326 docs/dev/debt/libkernel.md
PR 9331 docs/dev/debt/libc.md
PR 9340 docs/dev/debt/libc.md
PR 9342 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 9344 core/libs/prx/libc/CMakeLists.txt
docs/dev/debt/libc.md
PR 9359 docs/dev/debt/libkernel.md
PR 9360 docs/dev/debt/libc.md
PR 9363 docs/dev/debt/libkernel.md
PR 9364 docs/dev/debt/libc.md
PR 9365 docs/dev/debt/libkernel.md
PR 9378 docs/dev/debt/libkernel.md
PR 9388 core/libs/prx/libc/CMakeLists.txt
PR 9392 docs/dev/debt/libkernel.md
PR 9395 docs/dev/debt/libc.md
PR 9408 docs/dev/debt/libc.md
PR 9423 core/libs/prx/libkernel/File/src/Stdio.cpp

@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 18, 2026
@qpavyros
qpavyros force-pushed the codex/archive-stdio branch from e8c4d03 to 073343e Compare October 18, 2026 20:21
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 18, 2026
@blawesom

Copy link
Copy Markdown
Contributor

Rebase handoff — rebased onto current main (authorship unchanged). One docs-debt conflict, unioned with main's entries.

Verified on Linux (CachyOS, kernel 7.2.9, GCC 16.2.1, Ninja, Release, BUILD_TESTING=ON, AMD RX 9070 XT / RADV): builds; conventions pass; full suite green apart from the two known main regressions (agc_driver_tessellation — Mesa, #5444; agc_driver_bda_device — #7584), neither from this PR.

To pick it up:

git fetch https://github.com/blawesom/AnyPS5 rebase/7447
git checkout <your branch>
git merge --ff-only FETCH_HEAD

Head is f725dfca4.

@adriadam10 adriadam10 self-assigned this Oct 19, 2026
@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 20, 2026
@github-actions github-actions Bot removed the needs rebase Conflicts with main, rebase needed label Oct 20, 2026
@github-actions github-actions Bot added the needs rebase Conflicts with main, rebase needed label Oct 21, 2026
@Felipebmaitan Felipebmaitan mentioned this pull request Oct 21, 2026
12 tasks done

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

build docs libc libkernel needs rebase Conflicts with main, rebase needed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants