Add topological modeling for action forward dynamics - #223
Open
teerthsharma wants to merge 1 commit into
Open
Conversation
Collaborator
|
heads up for reviewers: NVIDIA/NeMo-Relay#282 |
Collaborator
|
Interesting feature. @teerthsharma is the idea that if FD retains the same connected components, the prediction is good? And this is used as a proxy for FD eval? |
Author
Same connected components by topology yes !! |
Cosmos3 action forward-dynamics rollouts are inspected frame by frame, so
failures that live in structure rather than pixels -- an object mask splitting
across an autoregressive chunk boundary, a contact loop opening, a stable
component collapsing -- do not surface in pixel error. This adds an optional,
side-effect-free evaluation layer over rollout masks that reports connected
components, enclosed-background hole proxies, Euler characteristic, drift,
finite-difference topology dynamics, and a convergence certificate.
Default Cosmos3 generation behavior, model weights, GPU requirements, and the
dependency set are unchanged. The core path is pure Python over lists; ripser is
optional and off by default.
Files:
- cookbooks/cosmos3/generator/action/topology_helpers.py
- cookbooks/cosmos3/generator/action/topological_modeling.md
- cookbooks/cosmos3/generator/action/README.md
- tests/cookbooks/cosmos3/generator/action/test_topology_helpers.py
Determinism
The report is a comparison artifact, so its bytes are fixed by the input alone.
Four sources of variance are closed:
- Labels are evaluated in sorted order. Previously `{"object": ..., "gripper":
...}` and `{"gripper": ..., "object": ...}` produced reports whose frame list,
CSV row order, and finite-difference trace differed while the data did not.
- `rank_topology_specialists` breaks score ties on `specialist_id`. Previously a
tie resolved to whichever candidate the caller listed first.
- Persistent homology is either explicitly absent or explicitly required.
`PersistentHomologyConfig` drops the `enabled` flag, which silently did
nothing while `backend` defaulted to `"none"`; `backend="ripser"` now raises
`ImportError` rather than degrading to a different set of numbers on a machine
without the package.
- `RolloutSpec` rejects a non-positive `fps`. `fps=-15` clamped the
finite-difference `dt` to `1e-9` and reported a topology speed of 2.08e9
against a gate threshold of 2.0.
Digital topology
Two conditions make `components - holes` an Euler characteristic rather than two
unrelated counts, and `TopologyConfig` now enforces both.
Complementary connectivity: `foreground_connectivity` and
`background_connectivity` must be 8/4 or 4/8, the digital Jordan curve
condition. On the diagonal ring `.#. / #.# / .#.`, the 8/8 pair reports zero
holes because the enclosed pixel reaches the border diagonally, and the 4/4 pair
splits the ring into four components and reports chi = 3. Only the complementary
pairs return chi = 0. A matching pair raises `ValueError`.
One filtered foreground: `min_component_area_px` is applied before holes are
counted. A ring of area 8 under a threshold of 16 previously reported zero
components and one hole, chi = -1 for an empty foreground; it now contributes
neither.
Persistent homology
`ripser` is called with an `(n, 2)` float array. The previous call passed a list
of pixel tuples and raised `'list' object has no attribute 'shape'`, so the
enrichment path failed in every environment where ripser was installed and was
reported as status `"failed"`. A skipped-unless-installed annulus test covers it.
`PersistentHomologySummary.betti_0`/`betti_1` are renamed `h0_bars`/`h1_bars`.
They are diagram cardinalities, not Betti numbers: a Rips H0 diagram carries one
bar per input point, so `h0_bars` always equals `sampled_points`. The Betti
proxies the module stands behind remain the combinatorial `components` and
`holes`.
Validation
41/41 unittest cases pass from this commit on Python 3.11.9, Windows 11, with
numpy 1.26.4 and ripser 0.6.14 present so the optional path is exercised
rather than skipped. The suite grows from 16 cases to 41. Run unchanged against the prior
helper on the same interpreter, it reports 13 failures and 1 error out of 41, so
every determinism, digital-topology, persistent-homology, and metric-axiom claim
above is pinned by a case that fails without the corresponding fix.
Ground-truth coverage asserts the value a correct implementation must return:
solid disk chi = 1, annulus chi = 0, figure-eight chi = -1, six separated blobs
chi = 6, diagonal ring chi = 0, and a scattered-points negative control that
must report zero holes. `topology_state_distance` is checked against the four
metric axioms over 4 vectors: identity, positivity on 6 distinct pairs, symmetry
on 6 pairs, and the triangle inequality over all 64 triples.
Cost
A 16-frame single-label rollout, one run per size on the machine above, prior
helper to this commit: 64x64 45ms to 30ms, 128x128 187ms to 128ms,
256x256 734ms to 541ms, 480x640 4.296s to 3.007s. The reduction comes from
hoisting the neighbor-offset table out of the flood-fill inner loop, where
cProfile attributed 0.35s of 2.43s at 256x256 to 1,048,576 rebuild calls, and
from converting each mask to a boolean grid once per frame instead of four
times. Complexity is unchanged at O(pixels) per frame per pass.
Limits
The stability score is a hand-tuned heuristic in [0, 1], not a calibrated
quantity, and is marked as such in the source; its constants should be fit
against labelled rollouts before scores are compared across runs. The
convergence gate thresholds are defaults, not derived bounds. A rollout shorter
than `betti_window` yields no windows and scores 1.0 vacuously; `betti_windows`
is reported in the metrics so that case is distinguishable, but the check itself
still passes. Timings are single-run on one machine with no confidence interval
and no cross-platform control. The metric is diagnostic and should not be
reported as a generation improvement without a controlled validation protocol.
Relates to NVIDIA#222
teerthsharma
force-pushed
the
codex/topological-modeling-action-fd
branch
from
August 11, 2026 19:22
94b8229 to
73e97ed
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds Topological Modeling for Cosmos Action Forward Dynamics as one cohesive optional evaluation layer for Cosmos3 action FD rollouts.
It covers the full modeling scope:
The helper is side-effect-free. It does not alter default Cosmos3 inference and does not require GPU/model weights/segmentation dependencies for CI-style validation.
Files
cookbooks/cosmos3/generator/action/topology_helpers.pyripseris availablecookbooks/cosmos3/generator/action/topological_modeling.mdcookbooks/cosmos3/generator/action/README.mdtests/cookbooks/cosmos3/generator/action/test_topology_helpers.pyVerification
All passed locally.
Maintainer context
Opened proposal issue before implementation: #222
Non-goals