Skip to content

fix(libkernel): fail getdents on a descriptor that is not open with EBADF - #4999

Closed
xXPBSXx wants to merge 1 commit into
boykopovar:mainfrom
xXPBSXx:fix/getdents-bad-descriptor-v2
Closed

xXPBSXx wants to merge 1 commit into
boykopovar:mainfrom
xXPBSXx:fix/getdents-bad-descriptor-v2

Conversation

@xXPBSXx

@xXPBSXx xXPBSXx commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

What

On Windows, getdents, getdirentries and sceKernelGetdents / sceKernelGetdirentries answered ENOTDIR (20) for a descriptor that is not open. FreeBSD 11.4's kern_getdirentries takes the descriptor with getvnode first, and getvnode fails an invalid descriptor (-1, closed, never opened) with EBADF (9).

ReadDirectoryDescriptor now returns EBADF when the descriptor is not open and keeps ENOTDIR for an open descriptor that is not a directory.

An existing assertion changes with it: GuestDirectoryBuffer.cpp expected ENOTDIR for sceKernelGetdents(-1, …). That check came in with the undersized-buffer work, where the descriptor argument was not the subject, and it records what the code answered rather than a source for the number. It now expects EBADF, and the test also covers a closed descriptor, getdirentries and the POSIX forms.

Not changed, and listed in the TechnicalDebt entry for libkernel: for an open descriptor that is not a directory (and for a pipe or socket) the code still answers ENOTDIR, where FreeBSD's kern_getdirentries answers EINVAL (vp->v_type != VDIR; getvnode on a non-vnode). The console's answer is not known, so these were left as they were; the test therefore no longer pins ENOTDIR for an open plain file.

No title evidence: found by calling getdents with invalid descriptors and comparing with the FreeBSD source. This is FreeBSD conformance.

Overlap (checked with git merge-tree against the heads of #4796, #4920, #4922, #4929, #4968, #4989 and #4993): no conflict. The new TechnicalDebt entry sits next to the Windows directory descriptors entry, not at the end of the file, so it does not collide with the entries other PRs append there. No new test file, so no CMake change. Since then #5253 (getdirentries basep) conflicts with this PR in docs/dev/debt/libkernel.md (git merge-tree against main 8cf6a0b, both clean against main) and changes the same extern "C" block of GuestDirectoryBuffer.cpp; whichever lands second rebases the debt entry.

Tested

Windows 10 (19045), MinGW-w64 GCC 15.2.0 (WinLibs r7), Release, in a combined build on main 0214069e3:

  • guest_directory_buffer passes with the new checks (the test declares getdents_nid_postfix and __error_nid_postfix itself: main removed them from the file's extern "C" block in the meantime, and without the declarations the test does not compile); the guest_directory*, guest_fdopendir, guest_file* and guest_filesystem tests pass.
  • Without the change to DirectoryDescriptor.cpp it fails at the getdents(-1) check (Directory buffer check failed at line 71).
  • python tools/check_conventions.py --base upstream/main passes.
  • Measured on a combined build of the twelve open branches of this series (this PR and the others) merged onto main 0214069e3: guest_directory_buffer takes 0.07 s (ctest, serial, Windows). The full guest_* / libc / libkernel group passes with this branch in the combined build (394 of 394, 0 failed; the build also contains the other branches of this series).
  • Not tested: Linux (DirectoryDescriptor.cpp is the Windows implementation; the Linux path answers with the host's error), a console, a title run.

Checklist

  • Based on current main; no other open PR implements the same functions
  • One topic per PR; follow-ups go in a new PR
  • No comments in code except technical debt
  • Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed in TechnicalDebt (docs/dev/debt/libkernel.md)
  • No notes, investigation .md files or images added to the repository
  • New third-party code is a submodule built from source (none added)
  • Generic behaviour, not specific to one title
  • Depends on: none
  • Unblocks: none
  • AI-assisted: yes

AI-assisted (Claude Code).

🤖 Generated with Claude Code

…ndows

getdents and getdirentries answered ENOTDIR for a descriptor that is not open. FreeBSD 11.4's kern_getdirentries takes the descriptor with getvnode first, which fails an invalid descriptor with EBADF. An open descriptor that is not a directory keeps ENOTDIR. The existing check for getdents(-1) expected ENOTDIR; it now expects EBADF and also covers a closed descriptor, getdirentries and the POSIX forms.

Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
@xXPBSXx
xXPBSXx force-pushed the fix/getdents-bad-descriptor-v2 branch from ae37373 to 39d6101 Compare October 13, 2026 16:14
@simonbronk

Copy link
Copy Markdown
Contributor

Tested on Linux x86-64 (Ubuntu 24.04 in a Colima VM with Rosetta on Apple M4, GCC 13.3), this PR (ae37373a9) merged into main 39890031e, Release build with BUILD_TESTING=ON.

  • ctest: 1062/1069 pass, including the tests in core/libs/tests/GuestDirectoryBuffer.cpp. The only failures are the 7 that also fail on main in this environment (Rosetta and VM limits: guest_math, amd64_only_converter, agc_driver_depth_compare_copy, tls_function_coverage_register_address_loads, linux_executable_exports, linux_entry_argv, linux_input_program_headers). No new failures.

AI-assisted: yes (Claude Code).

@xXPBSXx

xXPBSXx commented Oct 13, 2026

Copy link
Copy Markdown
Contributor Author

Thank you for the Linux test, @simonbronk.

One correction for the record: the head you tested (ae37373a9) has been replaced by 39d610155. After main's d31e283 removed the unused getdents_nid_postfix and __error_nid_postfix declarations from GuestDirectoryBuffer.cpp, the earlier head would not compile once merged with main (the merged test calls two functions main no longer declares); the new head declares both in the test. The code change (ReadDirectoryDescriptor answering EBADF for a descriptor that is not open) is the same. The description now states this, and that for an open descriptor that is not a directory the code still answers ENOTDIR where FreeBSD answers EINVAL (listed in the TechnicalDebt entry).

AI-assisted (Claude Code).

@oneandonlydean

Copy link
Copy Markdown
Collaborator

Closing: no title is known to reach this. With the queue at 750 open PRs, please hold off on new PRs that don't name a title needing the change until it's down. If a title stops on this on current main, reopen with the title ID and the log line.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants