Skip to content

feat: temporal guides + still-scene convergence fix - #1

Merged
DennisSmolek merged 22 commits into
mainfrom
feat-temporal-guides
Jul 29, 2026
Merged

feat: temporal guides + still-scene convergence fix#1
DennisSmolek merged 22 commits into
mainfrom
feat-temporal-guides

Conversation

@DennisSmolek

Copy link
Copy Markdown
Member

Temporal guides program (M1–M6) + the post-parity core work, plus a still-scene
convergence fix found by the program's first full-pipeline consumer.

Merging this triggers publish.yml path B — the branch carries feat: commits
since v0.1.0, so it auto-bumps and publishes 0.2.0.

Temporal guides — a new public surface

Dilated motion, dilated depth, and disocclusion are frame properties, not
upscaler properties: every temporal effect upstream (SSGI/SSR reprojection,
SVGF-class denoisers, TAA) re-derives worse versions privately. The upscaler now
publishes its internal working set as upscaler.guides (ordinary three
textures), and the frame can be driven split so the geometry guides exist
before the final color does:

upscaler.dispatchGuides({ depth, velocity }, camera); // right after the G-buffer
// … effects sample guides.dilatedMotion / .disocclusion / .dilatedDepth …
upscaler.dispatchUpscale({ color, deltaTime }, camera);
  • path: 'guides' runs the early stage alone, for apps that never upscale.
  • Reactivity is bidirectional — an explicit mask merges (per-pixel max)
    with the auto-generated one; effects can write into guides.reactive mid-frame.
  • MomentsPass — standalone, signal-agnostic (E[x], E[x²]) over any float
    texture plus one coarse level, with zero coupling to the upscaling pipeline.
  • temporalGuides(depth, velocity, camera) publishes the same bundle as TSL
    texture nodes; upscale(..., { guides }) shares one computation across the graph.
  • Contracts per product (format, space, resolution, latency) live on the
    TemporalGuides type and in TEMPORAL-GUIDES-SPEC.md. Live references:
    examples/12-temporal-guides (raw), examples/13-guides-node (TSL).

Cross-repo acceptance (M6): PASS. An external SSGI/SVGF consumer ran its
identical temporal stack against its private guides pass vs this bundle:
still-camera stability bit-identical, teleport reconvergence 1.275 s vs
1.288 s, across-arm diff below within-arm noise. Their verdict — drop-in
replacement. @experimental is therefore off the raw surface and MomentsPass;
it stays on the TSL node only, whose contract is the accepted one but whose
graph plumbing has no external consumer yet.

The monolithic dispatch() is unchanged in behavior: byte-identical captures
and perf-neutral (−2.7%, within noise) against the pre-split pipeline.

Core fixes

Still-scene convergence (consumer report 3, confirmed ours). On a still
camera the output never converged — flickering Disocclusion silhouettes and a
forever-rolling AccumulationAge, at every ratio including NativeAA. Reproduced
in our own bench with no consumer code. The decisive metric was the frame diff
between two frames at the same jitter phase one period apart, where benign
per-phase shimmer nets ~0 and we measured 0.182. Three stacked defects:

  1. Depth-clip vote starvation (reconstruct.ts) — only taps in front of the
    current surface could vote or carry weight, so one bilinear tap straddling
    last frame's texel-quantized dilated-depth boundary became the sole voter and
    disoccluded still edges every phase. Now every valid tap votes and the best
    tap wins; genuine trails still read ~1.
  2. Clip-magnitude history aging (accumulate.ts, removed) — aging the sample
    count by clip strength pinned equilibrium age low at any contrasty edge, so
    alpha never shrank. FSR2 never does this.
  3. Clip write-back (accumulate.ts) — the blend stores the clipped
    history, so each phase's variance box re-snapped the buffer regardless of
    alpha. Fixed with STILL_CLAMP_RELAX, gated on stillness × convergence ×
    absence of disocclusion/shading-change/reactivity.

Q1 2×: 0.211 → 0.112 consecutive, 0.182 → 0.018 phase-locked. NativeAA:
0.081 / 0.003. New Q12 cornell scenario: 0.024, disocclusion view black, age
saturated. Motion scenarios (Q3/Q4) verified regression-free.

Grazing-angle disocclusion flicker — jitter-delta-compensated reprojection
plus a tolerance widened by the dilation ring's own depth relief.

Post-parity adoptions (all measured)

  • RCAS sharpens in conditioned tonemap space, inverting once — −34% RCAS,
    −5.7% total
    , capture-identical.
  • Host pre-exposure (preExposureTexture) honored end-to-end: DeltaPreExposure
    history correction + host-invariant metering. Byte-identical when absent.
  • AMD's viewport/depth-scaled disocclusion tolerance, inside our fused pass.
  • Multi-scale shading-change detector (shadingChange.ts) replacing the 3×3
    heuristic — 0.044 ms, 5× cheaper than the source-style candidate, measurably
    fewer false positives under motion.

Tooling

  • scripts/measure-convergence.mjs — headless CDP convergence meter
    (consecutive + phase-locked frame diffs, debug-view PNGs).
  • Bench scenarios Q11 (host pre-exposure) and Q12 (cornell still-convergence).
  • Parity candidate bundles remain runnable for A/B.

Verification

npm test (172 passed) · npm run typecheck · npm run lint · npm run build
all clean. GPU verification is recorded per item in bench/docs/NEXT-STEPS.md
and TEMPORAL-GUIDES-SPEC.md; consumer-side evidence in
GUIDES-HANDOFF-RESPONSE.md.

Known follow-up (not blocking)

src/shaders/candidate*.ts — ~55 kB of parity-research shader source — is
statically imported by Upscaler.ts and so ships in dist. Bench-only, gated
behind _candidateBundle, and not tree-shakeable as wired. Worth moving behind
a bench-only entry point in a later release; deliberately not touched here,
since it means editing a core path right before a publish.

🤖 Generated with Claude Code

DennisSmolek and others added 21 commits July 16, 2026 14:53
Fix five defects that prevented the authored source-parity bundles from
ever compiling on a device (isInf, reserved keyword, r8unorm storage,
unused sampler binding, /0.5 velocity falloff), fix the cross-variant
analyzeAbba label crash so production-vs-candidate A/B runs work, and
add --help to the benchmark runner.

Measured (ratio 2, ABBA, noise floor <=2%): filter bundle +36%,
structural +6.5% incremental, full SPD resolver +76% GPU compute vs
production, with no visual regression or win on Q0/Q1/Q3 captures.
No bundle adopted; evidence in bench/PARITY-DECISIONS.md and PARITY.md
(new consumer-facing parity rationale).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
PARITY.md stays at root as the consumer-facing rationale; the internal
ledgers (PARITY-PROGRESS/CANDIDATES/DECISIONS, THREE-TEMPORAL-COMPARISON)
move to bench/docs/. NEXT-STEPS.md captures the four surviving adoption
items with per-item validation gates. CLAUDE.md points at the new layout.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… DeltaPreExposure, AMD disocclusion

Item 1 (RCAS cost investigation, resolved): the resolver candidate's "47%
cheaper RCAS" was per-tap work, not value ranges — production paid a 1x1
exposure load + tonemapInvert division + exposure division per tap behind
FLAG_INPUT_REINHARD. Production RCAS now sharpens the accumulate history's
conditioned tonemap-space texels (the [0,1) range the limiter math assumes)
and inverts the conditioning once on the result: RCAS 0.103 -> 0.068 ms
(-34%), total pipeline -5.7% (warm ABBA blocks, ratio 2). Captures on
Q0/Q1/Q3 + Q9 HDR stress: RMSE <= 1.8/255, HDR-bulb ROI <= 9/255 — visually
identical, no overshoot. The per-tap form is frozen as RCAS_PER_TAP_SHADER
under the rcas-fsr315-limiter / rcas-fsr315-numeric bench identities; both
isolating timing variants (rcas-hoisted-exposure-v1, rcas-tonemap-space-v1)
stay in the registry.

Item 2 (host pre-exposure, DeltaPreExposure semantics): the pyramid
publishes preExposureTexture's value in the exposure texel's .b and meters
auto-exposure host-invariantly (GPU-found: without this the conditioning
chases a host step for ~2s and the drift fires the shading-change detector
full-screen on flat regions). Accumulate ratio-corrects reprojected history
in linear space across a host change (new binding 11 = previous exposure
texel). Validated on the new Q11 host-pre-exposure scenario (bench drives
the scene MRT color and preExposureTexture together; E00 manifest extended):
a 2.5x step + ramp leaves the detector at baseline, never resets
accumulation age, and output brightness tracks the drive. Without the input
the pass is byte-identical (PNG-compared pre/post).

Item 3 (AMD disocclusion constant): the fused reconstruct pass replaces the
DEPTH_SEPARATION_SCALE / DEPTH_SIMILARITY_FLOOR guesses with AMD's
per-bilinear-tap confidence voting and viewport/depth-scaled tolerance
(1.37e-5 * halfViewportWidth * maxDepth, ffx_fsr2_depth_clip.h via the
GPU-verified candidate port). Q3: thin stable silhouette outlines, still
scenes near-black, age resets confined to trails; finals RMSE <= 1.1/255;
pass time unchanged. Fused single-pass structure kept (source's scatter +
separate pass measured +30%/+22% for no visual win).

GPU verification: all runs on Apple Metal via CDP; examples 01/05/07 smoke
clean (no WGSL errors) on the default-constructed Upscaler. Docs updated
(PARITY.md, shaders README fidelity table, NEXT-STEPS, CLAUDE.md). Only
NEXT-STEPS item 4 (Phase-5 SPD shading-change detector) remains open.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…PS 4)

Replaces accumulate's inline 3x3-neighborhood shading heuristic with
src/shaders/shadingChange.ts: one fused half-resolution dispatch whose 8x8
workgroup covers a 16x16 render tile, making the 4x4 and 8x8 block
reductions workgroup-local — each luma comparison is evaluated once with
hoisted frame-info loads, no mip textures, no separate resolve pass. The
pass maintains a 1-frame luma history, reprojects it jitter-delta-aligned
(bilinear), neutralizes disoccluded texels, and compares block-MEAN luma
per scale gated by a base + contrast-scaled (coefficient of variation)
noise floor. The strongest gated scale feeds accumulate's existing
SHADING_AGE aging path (new binding 12; zero dummy when
settings.detectShadingChanges is off — the pass simply isn't dispatched).
Locks keep their self-referential break, untouched.

Five GPU tuning iterations (evidence in bench/results/raw/E00/pre-spd-*
and post-spd-v*): the source-style mean-of-per-texel-signed-ratios floors
at ~0.10 still-scene response — the ratio metric weights the darker side
of alias residue, a coherent bias signed averaging cannot cancel — and
neither jitter alignment nor bilinear reprojection alone fixes it.
Averaging luma BEFORE the ratio, adaptive floors, and dropping the
flicker-dominated 2x2 scale hit the full acceptance matrix:

- Q1 still scene: mean ~2/255 (old detector ~1.1) — near-black
- Q4 camera orbit over high-frequency content: worst 3.6 vs old 4.9 —
  measurably FEWER false positives (the item's whole point)
- Q9 light steps: clean single-frame spikes at 137/113 per 255 (old: 84
  with a ~20-frame decay tail); slow ramps deliberately quiet (1-frame
  comparison; blend tracks ramps — Q9 ramp finals show no lag, RMSE 0.78)
- Q11 host pre-exposure step: quiet (hostRatio correction inside the pass)
- Finals within 1.6/255 RMSE of the old detector everywhere

Cost: 0.044 ms at ratio 2 — 5x cheaper than the candidate's two-pass form
(0.231 ms measured). Examples 01/07/09 GPU-smoke clean. Bench gains the
'shadingChange' timing label. SHADING_LO/HI are gone (the floors live in
shadingChange.ts); SHADING_AGE stays in accumulate.ts.

Docs updated: CLAUDE.md (roadmap item closed, landmine + debug-protocol
entries rewritten), shaders README (adopted section with the rejected
designs and why), NEXT-STEPS item 4, PARITY.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e status everywhere

- PARITY.md: expanded into the shareable design-notes document with a new
  'What we changed — enhancements beyond a port' section (fused reconstruction,
  conditioned-space RCAS, the multi-scale shading-change detector incl. the
  block-mean-ratio bias finding, host-invariant exposure), each with upstream
  behavior, our form, and measurements. Stale 'open items' section closed.
- README: 'Phases' table replaced by a current-state Status section; PARITY.md
  now linked from How-it-works and Status.
- bench/docs: PARITY-PROGRESS.md (concluded program's process scaffolding)
  deleted, its load-bearing facts folded into PARITY-DECISIONS.md;
  NEXT-STEPS.md retitled as a closed adoption record (header wrongly said
  item 4 still open); PARITY-CANDIDATES.md gained the program-conclusion note;
  DENOISING-DIRECTION.md moved into bench/docs/.
- CLAUDE.md: phase-ledger roadmap restructured into 'Feature status' +
  'Deferred / out of scope', keeping every mechanism note and trap.
- Code comments: stale phase references removed; accumulate.ts and
  luminancePyramid.ts headers corrected (they described shipped features as
  future work). No WGSL strings touched — shader fingerprints unchanged.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… (SSGI/SVGF)

Maps bench/docs/FSR3-BRIEF.md onto the actual pipeline: publish the existing
internal textures as a contracted guides bundle, split dispatch into an early
geometry stage and a late luma stage, add bidirectional reactive merge, and
a standalone signal-agnostic MomentPyramid. Additive only; existing dispatch
stays byte- and perf-identical.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…hange)

_encodeTemporal now composes _encodeGuides (reconstruct only — the
signal-agnostic geometry stage) and _encodeLate (everything needing beauty
color) on one encoder, so the monolithic dispatch keeps its single submit.
Seam for the split-dispatch guides API (TEMPORAL-GUIDES-SPEC M2).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mental)

- upscaler.guides (TemporalGuides): the production working set as three
  StorageTextures — dilated motion/depth, disocclusion, reactive, shading
  change, exposure, locks, history — with ping-pongs resolved to the most
  recently written half.
- dispatchGuides()/dispatchUpscale(): the split frame, so geometry guides
  exist post-G-buffer, pre-color; path 'guides' runs the early stage alone.
- GpuTimer merges per-label results (a split frame is two submits).
- examples/12-temporal-guides: split-dispatch demo + headless GPU harness.

GPU gates: Q0 captures byte-identical vs pre-publish (24/24), example 12 and
the guides-only path validation-clean headlessly.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cross-frame gather comparison (our deliberate cheaper form of upstream's
same-frame scatter) shipped with three instabilities, visible as per-jitter-
phase full disocclusion flicker on a distant ground plane (found via the
temporal-guides example): a running-AND tap veto that upstream doesn't have
(any tap at/behind the current surface zeroed the pixel, order-dependently),
no jitter-delta compensation in the reprojection, and a fixed tolerance that
reads a steep slope's per-texel depth change as surface separation.

Fix: reference per-tap skip semantics, jitter-delta-compensated prevUV (same
derivation as shadingChange), and the separation tolerance widened by the 3x3
dilation ring's own depth relief (free, geometry-derived, no tuned constants).

Measured: example-12 disocclusion quadrant frame-to-frame flips drop from
12-14% of pixels to ~1.5% (moving-content baseline); Q3 fence trails and
silhouette outlines unchanged; Q0 accumulation age rises slightly (history
surviving where it was falsely reset); final-image delta RMSE ~1.2/255.
Candidate bundle shaders keep the old form (frozen bench identities).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The generator gains an incoming-mask binding and max-merges it with the
opaque-diff response instead of being suppressed by an explicit mask; the
published guides.reactive target is the merged result, and effects may write
reactivity into it between the split dispatches. Aliasing guard: passing
guides.reactive back as `reactive` while reactiveOpaqueColor is set throws
(the generator writes that texture). Filter-bundle candidate path binds the
inert dummy; structural bundle's source-policy generator is untouched.

GPU-verified on example 05 (manual / auto / off modes, no validation errors).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…des M5)

Per-pixel (E[x], E[x²]) of a configurable scalar (Rec.709 linear luma or
YCoCg Y, runtime flag in the pass's own constants buffer) over any float
texture, plus one coarse level of 4x4 block means — SVGF's statistics half,
decoupled from every beauty/exposure assumption. One fused 8x8 dispatch,
workgroup-local reduction; rgba16float outputs (.rg used; rg16float is not a
core WebGPU storage format). Not wired into the upscaling pipeline at all.

GPU-verified against a CPU reference on a seeded 32x32 DataTexture in BOTH
spaces: validation-clean, max relative error <0.1% (f16 tolerance 1%),
coarse variance non-negative. Example 12 exposes MomentsPass + THREE for the
headless harness.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…priority

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ed-build config

GUIDES-HANDOFF-RESPONSE.md (demo-16, first full-pipeline consumer) landed:
linked build + M2 split-dispatch contract accepted and verified live on
cornell/sponza, nothing blocked. Two friction items, both ours:

- UpscalerNode._renderer was assigned in setup() and never read — deleted
  (it broke the consumer's noUnusedLocals when aliasing at src/).
- The handoff's "no duplicate-three hazard" claim was wrong for
  out-of-root consumers: two node_modules → two three cores. Folded their
  verified vite block (three→three/webgpu collapse, dedupe, optimizeDeps
  exclude) into GUIDES-HANDOFF.md and repointed the tsconfig-paths
  guidance at dist/index.d.ts, with the keep-dist-current commitment.

dist/ rebuilt so the consumer's declaration-based typecheck matches src.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
temporalGuides(depth, velocity, camera) publishes the temporal-guides
products into a THREE.PostProcessing graph as texture nodes
(getTextureNode(name): stable identity, ping-ponged products re-pointed
per frame, nearest 1x1 placeholder pre-configure so r32float format
inference never sees a filterable stand-in). Two modes, decided by wiring:

- standalone: the node owns a guides-only Upscaler sized to its depth
  input; late products are null and consuming one warns once with the
  linked-mode pointer.
- linked: upscale(..., { guides }) adopts the node's upscaler
  (_acquireUpscaler) and the split frame runs in-graph — guides dispatch
  right after the G-buffer, effects consume the products, the upscale
  node finishes with dispatchUpscale, falling back to the monolithic
  dispatch on frames where the early stage couldn't run (inputs not yet
  GPU-backed, mid-frame reconfigure). New Upscaler.guidesPending getter
  is the branch point.

Gates (spec M4): examples 07/09 GPU-verified unchanged; new
examples/13-guides-node consumes disocclusion in a toy effect (orange
trailing-silhouette tint, pre-upscale) — dispatch-spy probe shows the
pure split path steady-state (120 guides + 120 late, 0 monolithic over
1s) on one shared upscaler; standalone mode CDP-driven under a
validation error scope (clean, early products live, late null + warning).
172 tests, typecheck, lint, build green; examples/README.md table brought
current (07-13); dist rebuilt for the consumer's declaration typecheck.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Reproduced the ssgiDev demo-16 non-convergence in our own bench (Q1 still
camera: 0.211 consecutive-frame meanAbsDiff sustained, 0.182 at the SAME
jitter phase one period apart — aperiodic history churn, not the benign
per-phase pattern). Three stacked core defects:

1. reconstruct: the depth-clip vote only counted positive-separation taps,
   so one bilinear tap straddling the previous frame's texel-quantized
   dilation boundary re-disoccluded every still silhouette per phase. Now
   every valid tap votes (agreement = full confidence) and the best tap
   wins; genuine trails still read ~1.
2. accumulate: clip-magnitude aging of the sample count made convergence
   unreachable at any contrasty edge (rolling accumulation-age moiré at
   every ratio incl. NativeAA). Removed — FSR2 never ages on rectification
   strength; the clip + shading detector own stale shading.
3. accumulate: the blend stores the CLIPPED history, so each jitter phase's
   variance box re-snapped the buffer regardless of alpha (phase-locked
   0.183 with 1+2 fixed; 0.005 with the clip off). New STILL_CLAMP_RELAX:
   still + converged + quiet pixels get a ×9-widened box; motion,
   disocclusion, shading change, or reactivity restores full rectification.

Shipped ladder (Q1 2x, consecutive/phase-locked): 0.211/0.182 → 0.112/0.018.
NativeAA 0.081/0.003. New Q12 cornell-still-convergence scenario (enclosed
box + IGN-dithered Vogel point-light shadows, the consumer's aggravator +
pose): 0.024/0.012, disocclusion view fully black, age saturated. Q3/Q4
motion scenarios verified regression-free on GPU. New headless churn meter:
scripts/measure-convergence.mjs (consecutive + same-phase diffs, debug-view
PNGs). Evidence: bench/docs/NEXT-STEPS.md §5; PAPER-NOTES §6; consumer reply
appended to GUIDES-HANDOFF-RESPONSE.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…e raw surface

The consumer's cross-repo A/B (report 2) passed: their SSGI temporal stack
fed by our bundle measured bit-identical still-camera stability against
their private front-end, 1.275 s vs 1.288 s teleport reconvergence, and
MomentsPass was separately field-verified in their SVGF demo. That was the
spec's own exit criterion for the tag.

Dropped from `Upscaler.guides` / `guidesPending` / `dispatchGuides` /
`dispatchUpscale`, the `TemporalGuides` type, and `MomentsPass`.

Kept on `TemporalGuidesNode` / `temporalGuides` / `UpscalerNode`'s `guides`
option, with the reason narrowed in the doc: the contract those expose is
the accepted one, but the node's graph plumbing (node identity, per-frame
re-pointing, linked-vs-standalone wiring) has only our own GPU verification
(examples/13-guides-node) and no external consumer, so its ergonomics may
still shift.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@DennisSmolek

Copy link
Copy Markdown
Member Author

Follow-ups from this branch are now tracked as issues rather than left in the description:

None block merging.

The shader tests import bench modules, which import the package by its own
name. Node's self-reference resolves that through package.json `exports` →
`dist/`, so the test run depended on build output that CI hasn't produced
yet — `npm test` runs before `npm run build`. It passed locally only
because a stale dist/ was lying around.

Alias it to src/index.ts in the vitest config, matching what
bench/vite.config.ts already does. Verified by deleting dist/ and running
the CI sequence in order: lint, typecheck, test (172), build.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@DennisSmolek
DennisSmolek merged commit 13be5ed into main Jul 29, 2026
1 check passed
@DennisSmolek
DennisSmolek deleted the feat-temporal-guides branch July 29, 2026 08:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant