[DavidSamir]

What

On Windows, access, remove, opendir and chdir report ENOENT for a path that runs through a regular file, such as save.dat/x. FreeBSD reports ENOTDIR for each of them ("A component of the path prefix is not a directory" in access(2), unlink(2), rmdir(2), chdir(2) and open(2), through which opendir opens its directory), and the Linux build does.

This continues #5504, which does the same for open, and takes four functions off the list its debt note records.

Cause. Each of the four asks the host and maps its error. For a path whose directory part contains a file, Windows answers ERROR_PATH_NOT_FOUND, the same as for a missing directory, and both become ENOENT.

Observed on main 18c438b with #5504, Windows 11 (10.0.26300), WinLibs GCC 15.2.0 r7, Release. file is a regular file:

	before	this PR	Linux build
access("file/x", F_OK)	-1, errno 2	-1, errno 20	-1, errno 20
remove("file/x")	-1, errno 2	-1, errno 20	-1, errno 20
opendir("file/x")	null, errno 2	null, errno 20	null, errno 20
chdir("file/x")	-1, errno 2	-1, errno 20	-1, errno 20
the same four on file/x/y	errno 2	errno 20	errno 20
the same four on missing/x	errno 2	errno 2	errno 2

I have not found a title that stops on this. It came out of running one probe of the path functions on the Windows build and on the Linux build and comparing the two outputs.

Seven commits:

1. fix(libc): report ENOTDIR from access for a path under a regular file on Windows: adds UnderRegularFile_nid_no_patch to General.cpp and uses it in access.
2. fix(libc): report ENOTDIR from remove for a path under a regular file on Windows.
3. fix(libc): report ENOTDIR from opendir for a path under a regular file on Windows.
4. fix(libc): report ENOTDIR from chdir for a path under a regular file on Windows: in SetWorkingDirectory, so fchdir and the other callers share it.
5. test(libc): access and remove a path under a regular file.
6. test(libc): opendir and chdir a path under a regular file.
7. docs(debt): list access, remove, opendir and chdir among the calls that report ENOTDIR under a regular file.

Windows only. UnderRegularFile_nid_no_patch walks a host path's prefixes from the nearest one up and answers for the first that exists: true when it is not a directory. Each of the four functions calls it only after its host call has failed with ENOENT, and then reports ENOTDIR. The host call is made once and its failure stands; this only names it. A call that succeeds, and one that fails for another reason, make the same host calls as before.

Not changed: stat, lstat, unlink, rename (for its source), chmod, utimes and fopen still report ENOENT for such a path on Windows; the debt note says so. The resolver's check for a .. component and sceKernelRmdir each keep their own copy of this walk; General.cpp around the resolver is being changed by #4938 and #4681, so I left those lines alone.

guest_filesystem gains two groups of checks beside the existing remove ones. For a name one and two levels under a regular file, access with F_OK and with R_OK and remove must fail with errno 20, and for a name under a missing directory with errno 2, with the file still there afterwards. Then the same for opendir and chdir.

Tested

Windows (73658bc on main 18c438b plus #5504, Windows 11 10.0.26300, WinLibs GCC 15.2.0 r7, Release):

• With General.cpp, General.hpp, Filesystem.cpp and Directory.cpp as in fix(libkernel): fail open under a regular file with ENOTDIR on Windows #5504 and the tests kept, guest_filesystem fails (exit 0xc0000409, Filesystem check failed at line 355 (guest errno 2, Win32 error 3)). With the change it passes 5/5 (--repeat until-fail:5), and guest_kernel_fcntl, guest_truncate, guest_dup, guest_file_vector, file_streams, guest_standard_streams, guest_posix_filesystem, guest_fdopendir, guest_kernel_rename and guest_path_case pass 3/3.
• A probe of 198 lines, with fix(libkernel): fail open under a regular file with ENOTDIR on Windows #5504's libraries, the branch's and on the Linux build: 31 path calls (the open family, the stat family, access, realpath, opendir, chdir, truncate, utimes, chmod, mkdir, rmdir, unlink, remove and rename) on each of file/x, file/x/y, file/, file/., file/.. and file/../file. Ten lines change: the eight of the table, and opendir("file/") and opendir("file/."), which also go from ENOENT to ENOTDIR. With the branch, every line for access, remove, opendir and chdir equals the Linux build's.
• Probes of 122 lines over the descriptor functions, 866 over streams, 188 over the file, directory, vector and mapping functions, 431 over time, number, character and string functions, 145 over threads, 95 and 54 over flock, 44 over pipe streams, 15 over fcntl status flags, 127 over freopen, 130 over directory descriptors, 117 over streams used the other way, 144 over truncating opens, 164 over unlink and remove, 156 over opens of names ending in a slash, 102 over rename, 135 over stat, chmod and utimes, 55 over the working directory and directory streams and 313 over fdopen, run with both sets of libraries: one line differs, remove("file/x") in the unlink probe (the change above), apart from a file size in the fdopen probe that changes from run to run with either set.
• Titles from the homebrew catalog, 35 s each with fix(libkernel): fail open under a regular file with ENOTDIR on Windows #5504's libraries and with the branch's, windows kept off-screen: ccleste (PPSA59322), RommPS (PPSA76677), SwanStationPS5 (PPSA98510), ProsperoTV (PPSA99003), DOOM (PPSA99666), ScreenTester5 (PPSA69420), Prospero Vibrate (PPSA99920), SymphonyStation5 (PPSA17641), ProsperoStore (PPSA99000), OpenNOW (PPSA99082), ProsperoRadio (PPSA99001) and PSXS5 (PPSA97510) are all still running at 35 s with both, with the same number of stderr lines for each title and the same last kind of log line.
• Full ctest --parallel 4 on a full build of the last commit: 1097 of 1097 pass, as with main 18c438b. That is the second run. In the first, guest_none_relocations timed out at its 20 s limit while I was doing other work on the machine; it takes 1.2 s on its own and passed 5/5 alone and in the second full run.
• Each commit alone: each of the seven was checked out in order, the libraries and the tests rebuilt, and guest_filesystem, guest_posix_filesystem, guest_fdopendir, guest_path_case and file_streams pass 3/3 at each. At no commit does a rebuilt library lose an exported name. The full build of everything was done at the last commit.
• tools/check_conventions.py --base origin/main passes. tools/pr_overlap.py --branch reports no overlap with open pull requests.

Linux (same commit, WSL2 Ubuntu 24.04, GCC 13.3.0, Release): libs builds, and guest_filesystem and guest_kernel_fcntl pass 3/3, so the new checks hold on the Linux build as it is. The changed code is Windows only.

Not checked on a console or with a commercial title.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: fix(libkernel): fail open under a regular file with ENOTDIR on Windows #5504 (this branch starts from it; its four commits are the first four listed here until it merges)
• Unblocks: none
• AI-assisted: yes

[CnRJay]

The behaviour is right. My concern is how the same check is spreading through the code. With this PR, the "is a prefix of this path a regular file" walk exists in at least four places:

• HasNonDirectoryPrefix in libkernel/File/src/Stdio.cpp (already on main, used by mkdir, rmdir and truncate);
• the inline loop in sceKernelOpen from #5504;
• UnderRegularFile_nid_no_patch here, in libc;
• the chmod/utimes/lutimes copy in #5549.

They're all meant to be the same rule, and the debt entry still lists stat, lstat, unlink, rename and fopen as still to do, so that's another five call sites. One helper (this PR's UnderRegularFile_nid_no_patch is the natural candidate, since libc is below libkernel) used by all of them, with HasNonDirectoryPrefix and the #5504 loop rewritten on top of it, would keep the rule in one place. Fixing an edge case (a prefix that is a symbolic link to a file, a drive root, or UNC paths) would then fix every caller at once. Alternatively, map ERROR_PATH_NOT_FOUND in the shared Windows error translation (FilesystemError) whenever the path is available, so new call sites get it for free.

Could #5504, #5530 and #5549 agree on that order? Whichever lands first can introduce the shared helper, and the others rebase onto it.

[DavidSamir]

Closing in favour of #5549, which contains these commits (and #5504's), to keep the queue shorter.

@CnRJay, thanks for the review. #5549 has no walk of its own: its PathFailure calls UnderRegularFile_nid_no_patch from here, and the three were one stack in that order. The open loop from #5504 and the two HasNonDirectoryPrefix copies on main are the same walk; I have the change that moves them onto the helper, but it names no title, so I am holding it while the queue is long. Testing the helper on host paths: a prefix that is a link to a file, drive roots, \\localhost\D$\... and \\?\D:\... paths all come out right; a prefix that is a dangling symbolic link counts as a file (ENOTDIR where FreeBSD gives ENOENT), because this toolchain's std::filesystem::status reports a dangling Windows link as an existing file.

AI-assisted: yes