Skip to content

feat: opt-in single-sample MSAA fallback, MSAA trace and macOS title tooling - #3

Merged
giljakoki merged 18 commits into
mainfrom
claude/dazzling-hamilton-xm8ae9
Oct 11, 2026
Merged

giljakoki merged 18 commits into
mainfrom
claude/dazzling-hamilton-xm8ae9

Conversation

@giljakoki

Copy link
Copy Markdown
Owner

What

Follow-ups from running the first titles on an M5 Pro after #2.

AGC driver: multisampling

A Unity title (4x MSAA, PA_SC_AA_CONFIG = 0x0020c002) stopped at its first draws with "multisampling or coverage conversion is unsupported". Nothing in the driver handles more than one sample.

  • Opt-in APS5_MSAA_SINGLE_SAMPLE=1.
    • Multisampled colour and depth targets render as one-sample images at their own guest address.
    • A CB resolve (CB_COLOR_CONTROL mode 3, MRT0 → MRT1) becomes a GPU image copy, and an FMASK decompress (mode 5) is handled like a fast-clear eliminate.
    • Multisampled CMASK codes follow Mesa's fmask | color << 2.
    • Without the variable, behaviour is unchanged. The only difference is that error messages now name the variable where setting it would accept the state.
    • Still throws: multisampled texture reads, DCC over a multisampled target, per-sample shading, sample-mask exports with samples, partial CMASK fast clears and mismatched resolves.
    • Every approximation is listed in TechnicalDebt.md. Field names follow upstream's open full-MSAA PR (feat(agc): multisampled color and depth targets with CB resolves boykopovar/AnyPS5#419), to keep a later merge mechanical.
  • APS5_TRACE_MSAA=1.
    • Prints, de-duplicated, the MSAA-related context registers per distinct state, the image descriptors of multisampled and FMASK textures with the shader that reads them, and small uniform fills.
    • One run of a title shows what it needs next.
  • Docs: USAGE.md describes both variables. MACOS.md shows how to start a bundle with a variable (open --env NAME=1 Title.app, or Title.app/Contents/MacOS/launch from Terminal).

macOS tooling

  • tools/convert-ps5.sh, shipped in the package, converts a title folder into an app in one command:
    • it swaps in the dumper's decrypted <module>.elf copies, keeping the originals;
    • it unpacks fake-signed executables and modules;
    • it passes --module-dir for every folder that holds modules, except fakelib/;
    • modules in the title folder move to root-modules/, with links at their /app0 paths, which FMOD titles also load by path;
    • it checks for param.json before converting.
  • tools/unfself.py unpacks a fake-signed SELF (homebrew releases) into the ELF the relinker reads, and refuses encrypted or compressed segments. With it, the DOOM homebrew runs in game on an M5 Pro; MACOS.md records the result.
  • package_macos_app.py:
    • it stops before writing anything when sce_sys/param.json is missing;
    • it builds the bundle next to its destination and moves it into place only when complete, so a failure leaves no half-built app;
    • on macOS it clones the title's files (APFS copy-on-write) instead of copying them, and removes the quarantine attribute that made Gatekeeper scan a large title before its first start.
  • CI: the package's COMMIT names the pull request's head commit, not GitHub's merge commit.

Tested

  • Linux, full build (build-full):
    • the agc_driver subset gives 302 tests, 300 passed (27 skipped), with the variables unset, with APS5_MSAA_SINGLE_SAMPLE=1, and with APS5_TRACE_MSAA=1 added;
    • the 2 failures are the same with and without this change: agc_driver_image_address_dimension and agc_driver_image_plane_address_on_line_implicit_lod, both fixed upstream after this fork's base.
  • New tests:
    • agc_driver_graphics_single_sample_msaa, agc_driver_cmask_fast_clear_unimported, agc_driver_graphics_msaa_trace and agc_driver_multisample_trace_capture{,_draw};
    • the fallback cases in agc_driver_graphics and agc_driver_cmask_fast_clear;
    • each review fix was checked by removing it and seeing its test fail.
  • Review: three reviewers checked GPU semantics, tests and the trace. Their findings were fixed:
    • sample-mask exports are rejected;
    • a refused resolve names its reason;
    • DCC keys are checked before the flush;
    • the descriptor trace is reachable for multisampled textures.
  • Tools:
    • test_unfself and test_package_macos_app, including no half-built bundle on failure;
    • convert-ps5.sh run end to end on the DOOM release with the real relinker, and on Unity, FMOD and fakelib layouts with a stub relinker.
  • Titles on an M5 Pro, with the package of fix: upstream sync, failure reports and Windows test registration after the port; relinker fixes for a first Unity title #2 and these scripts:
    • DOOM homebrew in game;
    • Hades II gets past FMOD module loading;
    • the MSAA fallback itself has not run on a Mac yet: this pull request's macOS package is for that run.
  • tools/check_conventions.py passes.

Checklist

🤖 Generated with Claude Code

https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE


Generated by Claude Code

claude added 18 commits October 11, 2026 19:38
A pull request's run builds GitHub's merge of the head into the base branch, so COMMIT
named that merge (88ced7c for b11de50) and a user checking the download against the
pull request saw a commit they could not find. COMMIT now starts with the head commit and,
when the build was a merge, says which merge it was built as.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
…eave half a bundle

package_macos_app.py created the bundle and filled it in place. When a step failed, the
user was left with a bundle without its launch script and Info.plist, which macOS reports
as "The application cannot be opened because its executable is missing". A title dump
without sce_sys/param.json failed that way, with a Python traceback.

The packager now checks for sce_sys/param.json before it writes anything. The title's id
and name come from that file, and libkernel reads it while the title runs. It builds the
bundle as <name>.app.partial and renames it into place only when the bundle is complete,
so a failure removes the partial bundle and keeps an earlier one. Errors in the inputs are
printed as one line instead of a traceback.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
A title that stops at "multisampling or coverage conversion is
unsupported" says nothing else about how it uses MSAA: which draws set
PA_SC_AA_CONFIG, how its color and depth targets are programmed, how it
resolves (a CB resolve, an FMASK decompress, a shader reading a
multisampled texture or a copy) and what it writes into CMASK. Deciding
how far a fallback or full MSAA has to go needs that from one run.

APS5_TRACE_MSAA=1 prints to stderr, each distinct line once:

- "[msaa] state": the AA, scan-converter, sample-location, sample-mask,
  color-control, export and depth registers, the pixel program address
  and, per color slot with a format, its base, view, info, attrib,
  CMASK, FMASK, clear words, DCC base, high bases, ATTRIB2 and ATTRIB3.
  Printed in the draw precheck, before anything is decoded or rejected,
  for a draw whose PA_SC_AA_CONFIG is nonzero, whose CB_COLOR_CONTROL
  mode is a metadata pass (2 to 6), whose color slots have samples or
  fragments, or whose DB_Z_INFO has samples. The registers are read from
  the context directly, so the draw-recipe register log is unchanged.
- "[msaa] texture descriptor": the eight words of a texture descriptor
  of a multisampled type (14, 15), with ITERATE_256 or with an FMASK
  format, before its decode throws.
- "[msaa] buffer fill" / "[msaa] DMA_DATA fill": uniform fills of up to
  4 MiB, which is where CMASK, FMASK and HTILE clears show.

Each kind stops after 1024 distinct lines. Any value other than 1 stops
the title at its first draw. Without the variable nothing changes.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
Homebrew releases ship eboot.bin and their modules as fake-signed SELF containers, which
the relinker rejects ("The input is a SELF container, not an ELF"). Their segments are
stored unencrypted and uncompressed, so tools/unfself.py rebuilds the ELF: the header and
program headers follow the SELF entries, each entry with blocks holds the data of the
program header its id names, and the SCE version segment is the tail of the file. An
encrypted or compressed entry (a retail title) is refused.

Checked on the DOOM homebrew (lowbit/ps5-doom v1.1.1): both eboot.bin and
sce_module/libc.prx unpack and relink with --macos.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
…of commercial titles

The DOOM homebrew (lowbit/ps5-doom v1.1.1) runs in game on an M5 Pro at a stable frame
rate with keyboard input, from the macOS package of PR #2 with its eboot.bin unpacked by
tools/unfself.py and its bundled libc.prx set aside. The section also says where the first
commercial titles tried stop.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
The DOOM run used the macOS package built from b11de50, the head of PR #2, not main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
tools/convert-ps5.sh, shipped in the macOS package with unfself.py, runs the steps a first
user did by hand for each title:
- it swaps decrypted <module>.elf copies in for the encrypted modules beside them, as a
  dumper writes them, keeping the originals in <title>-encrypted-backup;
- it unpacks a fake-signed eboot.bin or module with unfself.py;
- it picks the decrypted executable, adds --module-dir for a Unity title's Media/Modules
  and Media/Plugins, and stops early without sce_sys/param.json;
- it warns about modules still encrypted and about "<name> 1.prx" copies;
- it relinks, packages and opens the app. Extra arguments go to the relinker, and
  INPUT=<file> adds a key mapping.

Run end to end here on the DOOM homebrew with the real relinker, and on a Unity-style
folder with a stub relinker.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
…PS5_MSAA_SINGLE_SAMPLE=1

A title that renders with MSAA stops at its first multisampled draw: the
precheck rejects a nonzero PA_SC_AA_CONFIG ("multisampling or coverage
conversion is unsupported (register 0x2f8 = 0x0020c002)") and the decode
rejects MSAA_ENABLE and multisampled CB_COLOR_ATTRIB and DB_Z_INFO. Full
MSAA is upstream pull request boykopovar#419. Until it lands, rendering such titles
with one sample is better than not rendering them, as long as the user
asks for it and whatever it cannot do still throws.

APS5_MSAA_SINGLE_SAMPLE=1 renders multisampled color and depth targets
as the one-sample images of their guest addresses:

- PA_SC_AA_CONFIG's sample counts, MAX_SAMPLE_DIST and centroid controls,
  MSAA_ENABLE, CB_COLOR_ATTRIB NUM_SAMPLES and NUM_FRAGMENTS, the FMASK
  bits of CB_COLOR_INFO and DB_Z_INFO NUM_SAMPLES are accepted.
  ColorTarget and DepthTarget carry the sample count, and the metadata
  pass its new modes and source, under pull request boykopovar#419's names so its
  merge stays mechanical. Coverage conversion, per-sample shading, sample
  masks, DCC over a multisampled target, multisampled mips, 3D targets
  and array slices, and slices of multisampled depth still throw.
- A multisampled CMASK is read with Mesa's MSAA codes (fmask | color << 2).
  An all fast-cleared one is materialized over the whole target on the
  CPU path and left at 0xCC (every sample on fragment 0, color expanded),
  an expanded one is kept, and a partly cleared one throws because the
  multisampled CMASK tile order is not modeled.
- CB_COLOR_CONTROL mode 3 (resolve, MRT0 into MRT1) becomes a device
  copy between the targets' resident images, which is exact for
  one-sample content. StorageTexture::CopyFrom takes a resolve flag for
  it: the copy may change the tile mode, marks a DCC destination's keys
  uncompressed and does not need the destination to be host-imported,
  since a render target's write-back takes the CPU path when it is not.
  Mode 5 (FMASK decompress) is a fast-clear eliminate of multisampled
  targets. Neither mode is decoded without the variable.
- Each multisampled target is reported once on stderr.

Without the variable nothing changes but messages: a rejection that the
variable would lift ends with "; APS5_MSAA_SINGLE_SAMPLE=1 renders
multisampled targets with one sample", and a multisampled color target
is reported by its CB_COLOR_ATTRIB before its CB_COLOR_INFO. Any other
value stops the title at its first draw. TechnicalDebt.md lists the
approximations.

agc_driver_graphics checks the default messages, that neither pass
decodes without the variable and the multisampled CMASK cases on the CPU
path; agc_driver_graphics_single_sample_msaa runs the decode cases with
the variable. agc_driver_cmask_fast_clear draws into, eliminates and
resolves a multisampled target on Vulkan, into a DCC destination too,
and agc_driver_cmask_fast_clear_unimported repeats that without host
imports.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
…ng variables for a macOS bundle

A user whose title stops on a multisampled draw sees a message naming
APS5_MSAA_SINGLE_SAMPLE but finds nothing about it in the documents, and
a macOS user cannot set a variable on a double-clicked bundle.

USAGE.md gets a section next to APS5_STORE_AT_FLIP: what the fallback
does and does not cover (no anti-aliasing, a CB resolve becomes a copy,
multisampled textures and the cases in TechnicalDebt.md still stop the
title), what APS5_TRACE_MSAA=1 prints and that other values stop the
title. MACOS.md says how a bundle gets a variable: the launcher passes
its environment on, so `open --env NAME=1 Title.app` or running
Title.app/Contents/MacOS/launch from Terminal with the variable set,
while the title is not already running.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
A title that ships middleware as a module outside sce_module and Unity's two folders,
such as an FMOD libfmod.prx the executable needs, was packaged without it and failed at
launch with dyld's "Library not loaded: @rpath/libfmod.prx". The script now finds every
folder below the title that holds .prx, .sprx or decrypted .elf modules. In each folder it
swaps in the decrypted copies and unpacks fake-signed modules, and it passes --module-dir
for every folder except sce_module, sce_modules and prx. It leaves sce_sys and fakelib/
alone; some dumpers put console system libraries in fakelib/, and AnyPS5 provides those
itself. Modules directly in the title folder stop the script with a message, since the
relinker takes module folders below it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
…ubfolder

A title can ship a module its executable links directly in the title folder: an FMOD
libfmod.prx beside eboot.bin, which the PS5 loader also finds there. The relinker takes
only module folders below the title folder, and treating the title folder itself as one
would pick up the executable too, so the script stopped on such a title. It now moves
those modules, with their decrypted .elf copies, into root-modules/ and passes it as a
module folder. A module the executable links loads from there. One the title opens by
its /app0 path at run time would not be found.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
…er APS5_MSAA_SINGLE_SAMPLE=1

The single-sample fallback accepted a pixel shader's sample-mask export
(DB_SHADER_CONTROL MASK_EXPORT_ENABLE with SPI_SHADER_Z_FORMAT 9) in a
draw whose PA_SC_AA_CONFIG has samples. The pipeline then runs with one
sample, where Vulkan keeps only bit 0 of gl_SampleMask: an exported
0b1110, which writes three of four samples on the hardware, drops the
pixel. Before the fallback this state threw at 0x2f8, and
TechnicalDebt.md says sample masks throw.

DecodeState and DrawRejection now reject the export when the fallback is
on and PA_SC_AA_CONFIG's MSAA_NUM_SAMPLES or MSAA_EXPOSED_SAMPLES is
nonzero. A one-sample rasterizer keeps the export, as the hardware also
reads only bit 0 there. Without the variable nothing changes: the check
is skipped and the 0x2f8 rejection comes first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
…p0 paths

A title can both link a module that sits in its title folder and load the same module by
path. FMOD titles call sceKernelLoadStartModule("/app0/libfmod.prx") at start-up and stop
when it fails. After the module moved to root-modules/, the load found no
/app0/libfmod.prx.guest.prx and returned SCE_KERNEL_ERROR_ENOENT. The title then trapped
in its libc (int 0x45), which macOS reports as a general protection fault. The script now
puts a link at <app0>/<module>.guest.prx to root-modules/<module>.guest.prx in the bundle,
so the load resolves to the image the executable already linked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
The resolve built its error message in the same call that ran
StorageTexture::CopyFrom, and C++ leaves the order of a call's argument
evaluations open. GCC, the compiler of the Linux and Windows builds,
built the message first, while the refusal reason was still null, so
every refused resolve reported "(unknown)" instead of "shape",
"released" or "no recorder".

The copy now runs first and the message is built only when it is
refused. agc_driver_cmask_fast_clear resolves into a destination of
another storage format and requires "(shape)" in the error.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
…over a DCC fast clear

The resolve test read the destination's DCC keys only after expect(),
whose flush marks the keys uncompressed by itself. Deleting the key
marking that StorageTexture::CopyFrom does for a resolve left every test
passing, although until that flush the guest keys still said "clear"
over the resolved texels.

The keys are now also checked right after the resolve, under the GPU
mutex once the device is idle. A new case fast-clears the destination's
keys the way a title's DCC clear reaches the driver (the keys written to
0, StorageTexture::NoteKeysFill), leaves its texels alone and resolves
into it: the resident image's refresh at the resolve takes the cleared
keys, so only the copy's own marking makes them uncompressed before the
flush. Without that marking both checks before the flush fail, with host
imports and without.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
The message of a refused stencil replacement is 95 characters with
two-digit values, written into a 96-byte buffer. The masks come from the
32-bit fields of VkStencilOpState, so GCC 13 at -O3 warns that the text
may be truncated (-Wformat-truncation) once its inlining of
stencilFace() changes, as it does when more code is added to State.cpp.
The values never exceed 0xff, so nothing was cut, but a buffer of 128
bytes holds every value the format can print and keeps the build free
of the warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
…ader is captured

APS5_TRACE_MSAA printed "[msaa] texture descriptor" from
DecodeTextureResource, which runs only once a shader has compiled. A
shader that reads a multisampled descriptor (image types 14 and 15)
never compiles: DescriptorBindingBuilder::Populate throws "multisampled
image resources have no descriptor image shape" first. The descriptor
the line exists for was never printed, so a run could not show which
multisampled texture a shader reads.

The draw and dispatch captures now trace every captured image descriptor
before the shader is compiled, with the shader's address. Populate takes
each guest image descriptor from the capture's snapshot, so the capture
sees every descriptor DecodeTextureResource would; its call is removed,
which keeps FMASK-format and ITERATE_256 descriptors from printing twice.

The state, descriptor and fill lines are built by pure functions, and
MultisampleTraceLog collects the printed lines for tests:

- agc_driver_graphics checks each line's format and when it is printed;
- agc_driver_graphics_msaa_trace runs with APS5_TRACE_MSAA=1 and checks
  that each distinct line is printed once;
- agc_driver_multisample_trace_capture and its draw variant submit a
  dispatch and a draw whose shader loads from a 2D MSAA descriptor and
  check the line naming the shader before the compile fails;
- agc_shader_memory checks that the capture holds that descriptor while
  the compile throws.

USAGE.md now says the descriptor lines name the reading shader, and that
an invalid APS5_TRACE_MSAA stops the title at its first draw, compute
dispatch or buffer fill, since the fills and the captures read it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
…bute

The packager copied a title's files into the bundle, so every converted title took its own
size again on disk: tens of gigabytes for a large title. Files a browser or another app
downloaded also carry com.apple.quarantine, and a bundle whose copied title has it makes
Gatekeeper scan all of it ("Verifying ...") before the first start, which for a large
title seemed to hang.

On macOS the title's files are now cloned with cp -c (APFS copy-on-write: no extra space,
copied at once). The packager falls back to a copy where cloning fails, for example on a
volume that is not APFS, and off macOS. The quarantine attribute is removed from the
finished bundle before it is moved into place.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HSL58TWgY3qJuxYjEgaYDE
@giljakoki
giljakoki merged commit 2dee572 into main Oct 11, 2026
17 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants