[LeandroLP23]

What

A tool for running titles you already have the same way every time, so a run can be attached to a PR or an issue and compared with an earlier one.

python tools/title_run.py --games <directory> [ID ...] finds extracted packages (eboot.bin and sce_sys/param.json) under --games, unwraps the SELFs with tools/unwrap_self.py, relinks with the title's flags and lays the title out as docs/user/USAGE.md describes. It then runs each title --runs times (default 2: a cold run with empty AnyPS5 shader cache and pipeline journal, then warm runs) and decides each run's outcome by itself:

Outcome	When
relink-failed	the relinker (or unwrapping) fails
crashed	the title ends before the deadline with a non-zero code
exited	it ends before the deadline with code 0
stalled	alive, but no aps5-summary.jsonl line in the last 20 s (no presents), or too few presents for a frame to be dumped
blank	alive and presenting, but the last dumped frame is one colour
rendering	alive, presenting, and the last frame has content

It writes summary.json and a summary.md table with fixed columns (title, ID, version, cold, warm, FPS, first error) under build/title-runs/<commit>-<time>/, keeping the first, middle and last dumped frames of each run. --compare <summary.json> adds a Before column and lists regressions, improvements and titles missing from the new run; when IDs are given, only those are compared.

• Per-title settings live in tools/titles.json: extra relink flags, fixed_base (mapped to --windows-fixed-base/--linux-fixed-base), executables for LoadExec targets, run length and environment. It holds the flags the catalog titles need today (VLC, Ruffle Flash and XPSemu at 0x400000, RetroArch's picker/picker.bin, the --trap-sdk-syscall-stubs titles). Entries hold title IDs only, never names: a run is labelled with the package's own param.json title name, and discover() refuses a param.json whose titleId doesn't match the same pattern as the keys. .gitignore gets !tools/titles.json, next to the existing !demo/web/i18n/*.json.
• Processes. On Windows a Job Object follows the title and every process it starts (sceSystemServiceLoadExec hand-overs included), and the run's exit code is that of the newest process whose image is inside the title's folder. On Linux it is a process group. A title is always ended at the deadline (close, 15 s grace, then terminate), and nothing is left running.
• Not in the way. Audio is muted (SDL_AUDIODRIVER=dummy). On Windows there is no console window, the title's windows never get focus and go behind every other window as soon as they are created (a WinEvent hook, so they do not flash in front first). They keep their position and can be brought forward from the taskbar. They are not minimized, because presents stop while a window is minimized.
• Caches. Mesa and NVIDIA shader caches are pointed into the run's folder. AMD's Windows VkCache cannot be redirected and is left alone; summary.json records it as not isolated.
• Never sources data. It only reads packages the user points it at.

Unit tests: tools/tests/test_title_run.py, registered as title_run. They cover outcome classification from synthetic logs, summaries and BMPs (including truncated ones), relink arguments, the report and comparison, package discovery, and the process session with real child processes. Docs: a section in docs/dev/TRACING.md.

Tested

Windows 11 (26300), Radeon 860M (RDNA 3.5, driver 32.0.13058.8001), Ryzen AI 7 350, WinLibs GCC 15.2.0, Release.

• tools/tests/test_title_run.py on Linux and macOS (GitHub ubuntu-latest and macos-latest, Python 3.12, on a throwaway fork branch, because this pull request's CI skips the build jobs): 10 runs of 10 on each. The first try failed on macOS, where killpg on a group whose processes are all exiting fails with EPERM; alive(), close() and kill() now treat that as still exiting.

• python tools/tests/test_title_run.py: 24 OK. ctest -R '^(title_run|test_registration|conventions_doc_links)$': 3/3. tools/check_conventions.py --base origin/main passes.

• The local homebrew catalog, 66 titles, main 5859889b2 with this branch, one 60 s run each, then the 22 titles that did not reach rendering again with --runs 2 --compare <first summary.json>. Nothing was left running after either batch. Windows Firewall asked about titles that open a port (answering Cancel does not affect the run); titles are now prepared at a fixed path so it asks once per title.

• The 22 titles again on main 1c523bdb4, --runs 2 --compare against the table below: RommPS now renders cold, Mortalfin's cold run hit the 10 s GPU watchdog (has not reached submission 10) like ProsperoAI and RetroArch do, everything else matched. ProsperoPuzzles renders with fix(agc): run the unwatch hook when a watched import leaves the write watch #6308 on top (29.2 FPS, cold and warm). ScreenTester5 had shown rendering from two presents just before the deadline, which led to the frame rule for stalled.

  • rendering: 48. 40 on the first pass. SuperTuxKart (PPSA69421) and ProsperoEden (PPSA99008) needed --trap-sdk-syscall-stubs, now in titles.json. Mortalfin (PPSA99141) and Nuvio (PPSA99997) render on both runs of the second pass. ProsperoAI (PPSA99004), RetroArch (PPSA99169), RommPS (PPSA76677) and PS5 OpenGL Showcase (PPSA99005) render only warm: cold, they stop on the PS5 RADV port's 10 s submission wait while their first pipelines build, which the Cold and Warm columns now show directly.
  • The second pass, as the tool wrote it:

Title	ID	Version	Cold	Warm	FPS	First error	Before
XashPS5 V1.3	PPSA19111	01.030.000	exited	exited	0.0		exited
Middle Ages Ping Pong	PPSA28319	01.000.000	crashed	crashed	0.0		crashed
ScreenTester5	PPSA69420	01.000.000	stalled	stalled	0.0		stalled
SuperTuxKart	PPSA69421	00.001.000	rendering	rendering	6.1		relink-failed
RommPS	PPSA76677	01.001.001	exited	rendering	8.7	[debug:0] [RommPS] radv/ps5: the GPU has not reached submission 10 (at 9) in 10 s	exited
ProsperoAI	PPSA99004	01.005.000	crashed	rendering	7.3	radv/ps5: the GPU has not reached submission 2 (at 1) in 10 s	exited
PS5 OpenGL Showcase	PPSA99005	01.000.001	crashed	rendering	4.0		exited
ProsperoPuzzles	PPSA99006	01.000.010	crashed	crashed	0.0	what(): AGC graphics: storing storage image results pending over memory outside the write watch without the guest bytes	exited
ProsperoEden	PPSA99008	01.000.110	rendering	rendering	59.7		relink-failed
vkQuake	PPSA99010	01.000.000	exited	exited	0.0		exited
PS5 FSR4 Showcase	PPSA99011	01.000.010	exited	exited	0.0		relink-failed
Homebrew UI Lab	PPSA99050	01.000.010	crashed	crashed	0.0		exited
Surround Sound Studio	PPSA99051	01.000.003	crashed	crashed	0.0		exited
OpenNOW PS5 - Prototype	PPSA99082	00.002.038	stalled	stalled	0.0		stalled
Mortalfin	PPSA99141	01.000.002	rendering	rendering	81.5		exited
RetroArch	PPSA99169	01.000.000	exited	rendering	38.8	[debug:0] [Frontend Picker] radv/ps5: the GPU has not reached submission 12 (at 11) in 10 s	exited
PS5VR	PPSA99177	01.002.000	stalled	stalled	0.0		stalled
Quake III Arena	PPSA99303	01.001.000	exited	exited	0.0		exited
Wind Waker	PPSA99309	01.000.001	exited	exited	0.0		exited
Porpoise (DolphinPS5)	PPSA99764	03.000.000	stalled	stalled	0.0		stalled
PICO-8	PPSA99808	01.001.002	crashed	crashed	0.0		crashed
Nuvio	PPSA99997	01.000.007	rendering	rendering	59.9		stalled

• Before is the first pass. Its exited for the crashing titles was a bug found by that pass: a crash brings Windows Error Reporting into the job, and its exit code 0 was taken for the title's. The exit code now comes only from processes inside the title's folder, so these show as crashed (for example ProsperoPuzzles with 0xC0000409, the abort of AGC: since #6060, ProsperoPuzzles (PPSA99006) stops at 5 s with the storage-image write-back throw on Windows #6301).
• XashPS5, Wind Waker and Quake III Arena exit by design without the user's own game data (the valve folder, a GZLE01 disc image, .pk3 files); PICO-8 needs elfldr; FSR4 Showcase gives up on the build time of its FSR4 pipelines in a first run (AGC: PS5 FSR4 Showcase (PPSA99011) gives up in its first FSR4 pass while the queue worker builds 130k-800k-word compute pipelines (up to 66 s each on AMD) #6118). OpenNOW reached its sign-in screen at 25.8 FPS on d8888df9b (catalog) and stops after a few presents here: its sign-in screen redraws only when its view changes and still answers input, so stalled is expected there (OpenNOW PS5 (PPSA99082) presents only on view changes; older builds' continuous presents were likely padding residue #7501).
• A package whose eboot.bin is neither an ELF nor a SELF gives relink-failed with Refused: file is smaller than the SELF header area; the batch carries on and writes summary.json.

Not tested: real titles on Linux and macOS, NVIDIA, Intel. Known limits: a LoadExec child that starts and exits within one poll (0.1-0.5 s) is not seen, and the parent's exit code is used; app0 is hard-linked to the user's package when the file system allows it; two invocations sharing one --out share work/<ID>.

Checklist

• Based on current main; no other open PR implements the same functions
• One topic per PR; follow-ups go in a new PR
• No comments in code except technical debt
• Unimplemented paths throw (NotImplemented_nid_no_patch); silent stubs are listed under Silent stubs in their library's technical debt file
• No notes, investigation .md files or images added to the repository (attach them to this PR)
• New third-party code is a submodule built from source
• Generic behaviour, not specific to one title
• Depends on: none
• Unblocks: none (a test tool; the example runs cover VLC PPSA85300, ProsperoLight PPSA99002, ProsperoPuzzles PPSA99006 and RetroArch PPSA99169)
• GPUs tested: AMD Radeon 860M (RDNA 3.5), driver 32.0.13058.8001, Windows 11
• AI-assisted: yes (Claude Code)

[github-actions]

Some comparisons could not be completed because a pull request conflicts with the base branch. Review these shared files manually before concluding that there is no overlap.

Pull request	Shared files not compared
PR 674	CMakeLists.txt
PR 1835	CMakeLists.txt
PR 5021	.gitignore CMakeLists.txt
PR 6044	CMakeLists.txt
PR 6879	CMakeLists.txt
PR 6946	CMakeLists.txt
PR 7201	CMakeLists.txt
PR 7604	docs/dev/TRACING.md
PR 7642	CMakeLists.txt
PR 8739	CMakeLists.txt
PR 9408	docs/dev/TRACING.md
PR 9508	CMakeLists.txt

[LeandroLP23]

Two questions before this gets reviewed, so I can adjust it first:

1. @oneandonlydean @adriadam10: is tools/titles.json fine in the repository? It holds only title IDs, relink flags (--trap-sdk-syscall-stubs, --windows-fixed-base, --executable for LoadExec targets), run length and environment for the catalog homebrew, and no game files. Without it, everyone running the catalog has to rediscover those flags. If you'd rather keep per-title data out of the tree, --titles already takes any path, so it can become a local file and the tool stays.
2. @AlxcBTW: summary.json is meant to be importable by the blocker map. Per title it has name, version, seconds and runs[]. Each run has run, cache (cold/warm), outcome, exit_code, seconds, presents, fps, first_error, warnings and the kept frames. The top level adds build (commit, dirty, note), machine (os, gpu, driver, cpu), runs, seconds and driver_cache. Is anything missing that the map would need, or named in a way that does not fit it?

AI-assisted

[oneandonlydean]

Thanks for asking, and sorry for the wait. Yes, a tools/titles.json with title IDs and their relink flags is fine in the repo, as long as it keeps to title IDs (no game names). We'll review it under the normal process.

[oneandonlydean]

Thanks for waiting on the titles.json question. Review of d0a04fe67e:

1. titles.json keeps to title IDs. The approval above was for title IDs and their run settings, with no title names. Each entry currently has a "name". The tool already falls back to the package's own param.json titleName (title_run.py:538), so please drop the field from titles.json and from TITLE_FIELDS, so it isn't accepted again later.
2. Validate titleId from param.json before using it as a path. discover() takes titleId as is, while only the titles.json keys are checked against [A-Z]{4}[0-9]{5}. main() then builds work = <out>/work/<titleId>, and prepare() starts with shutil.rmtree(work, ignore_errors=True). With pathlib, an absolute titleId replaces the base, and ../.. climbs out of --out. So a malformed param.json anywhere under --games can make the tool delete a directory outside --out. Please apply the same pattern in discover() (and, if you like, check that work resolves under <out>/work), with a test.
3. Minor: test_bad_id_is_refused uses an app name as the invalid ID, and the TRACING.md paragraph names a specific app's sign-in page as its example. Neutral wording would keep names out of the tree entirely, e.g. "NOT-AN-ID" and "a screen that presents only when its view changes".

The rest looks good: driver shader caches are isolated through environment variables instead of being deleted, processes are contained (job object on Windows, a process group elsewhere), and test_title_run.py passes here on current main (28 tests, 3 Windows-only skips).

[LeandroLP23]

Thanks for the review. All three points are addressed in c2dce8d, rebased onto current main:

1. name is gone from titles.json and from TITLE_FIELDS. An entry that sets it fails with "unknown field name" (new test). The summary takes the name from the package's param.json.
2. discover() now checks the param.json titleId against the same pattern as the catalog keys (TITLE_ID) before the ID names anything. The new test feeds it ../../outside, an absolute path and a too-short ID. All three are refused, and the test fails when the check is removed. With the ID validated, <out>/work/<ID> can't leave --out, so I didn't add a second resolve check.
3. test_bad_id_is_refused uses NOT-AN-ID. The TRACING.md sentence no longer names an app, and the summary tests use a neutral name.

test_title_run.py: 30 tests pass on Windows. check_conventions.py passes.

[MarcZX10]

Reviewed exact head c2dce8dd35091e7442c7ac4ecae38b32c4fd5cf0. The added param.json title-ID validation prevents malformed IDs from becoming paths under <out>/work; I verified the regression cases for absolute and traversal IDs are covered. I ran python tools/tests/test_title_run.py on Linux: 30 tests passed, 3 platform-specific skips. Hosted conventions and relinker jobs pass; platform-specific runner integration is covered by the PR author’s reported runs, not this Linux run.

[MarcZX10]

Independent Linux validation against the current PR head: python3 -m unittest tools.tests.test_title_run passed 30 tests (3 Windows-specific skips), and git diff --check upstream/main...HEAD passes. I used a temporary harness with the PR files plus the existing tools/package_release.py; no repository files were changed. I did not validate the Windows process/window-management paths.

[ycbuzi]

Windows run of c2dce8d, merged onto main f894ba6 (92a59f7199, tools only), for the Windows process and window paths the Linux validation above did not cover. Windows 11 Home 10.0.26200, Python 3.14.6, libs and relinker built from that merge with MinGW-w64 GCC 15.2.0 (ucrt-posix-seh r7), Release. RTX 4070 SUPER, NVIDIA 616.92 (ANYPS5_GPU=4070), Ryzen 9 7900.

python -m unittest tools.tests.test_title_run: 30 tests, all pass, none skipped, so the three that skip on Linux ran here.

tools/title_run.py PPSA59322 PPSA98766 PPSA85300 --games ... --runs 2 on the catalog releases (each zip matching the catalog's SHA-256). Every run relinked, ran the full 60 s, was closed by the runner, and left three frames, none uniform:

Title	ID	Cold	Warm	FPS	First error
ccleste	PPSA59322	rendering	rendering	119.1
VLC	PPSA85300	rendering	rendering	76.3
DriftCheck	PPSA98766	rendering	rendering	57.1

VLC took its titles.json entry (--registry, fixed base 0x400000). ccleste logged destroyed with GPU results pending three times in each run; the other two logged neither warning. Not exercised: --compare, and a title that crashes, exits or stalls, so those outcomes were not reached by a real title here.