Repository navigation
Add split-narrow-bridges: split accidentally-merged objects at thin necks - #86
Merged
Merged
Conversation
Implements the edt_watershed strategy from docs/split_narrow_bridges_plan.md for splitting accidentally-merged blob-like objects (nuclei, cells) at thin necks, preserving all original foreground voxels. Per-object dispatch reuses the Skeletonize/Measure bbox pattern; the two-phase compute-then-write design avoids concurrent chunk writes by computing per-object split labels into scratch files first, then applying them in a single spatially-disjoint blockwise pass. New split pieces get IDs strictly above the dataset's max existing ID, so they never collide with untouched objects. SkeletonGraphSplit (for thin/branched mitochondria) is stubbed with NotImplementedError pending the graph-based design. Wires up the `split-narrow-bridges` CLI entry point following the existing RunProperties convention.
Adds the skeleton_graph strategy for splitting merged objects: builds the object's skeleton graph and cuts at nodes that are a genuine local minimum of EDT radius (thinner than every neighbor), not just nodes below a flat threshold. A flat threshold also flags an ordinary monotonic taper (e.g. a blob with a thin spur) as a candidate neck, since radius shrinks steadily to the spur's tip without ever dipping and rising again -- exactly the over-splitting failure mode this feature is meant to avoid. Requiring a true local minimum (thick-thin-thick) fixes that. Getting the size-gate accounting right took two more wrong turns, both now documented in docs/split_narrow_bridges_plan.md: a pi*r^2*length tube-volume proxy overestimated a 3-voxel stub as ~6 voxels, and a straight-line nearest- skeleton-node assignment leaked most of a compact blob's volume onto a thin stub's nodes. The fix is a marker-based watershed on a flat field restricted to the mask, which assigns every voxel to its nearest node geodesically (respecting the mask's actual shape) and gives exact per-node voxel counts. Neither strategy is organelle-specific -- skeleton_graph is a general topology-aware alternative to edt_watershed, just better suited to thin/ branched shapes (mitochondria being the motivating case, not a special case). Adds two more tests: a dumbbell split via skeleton_graph, and a trunk+stub shape verifying the size gate keeps a spurious small protrusion attached.
EDTWatershedSplit now rejects a split whose boundary isn't actually thin (_merge_thick_boundaries), not just too small -- catches two density peaks in one continuously-fat, irregularly-shaped blob that a size gate alone would miss. Verified against real cerebellum nucleus data: correctly no-ops when boundaries are genuinely thin, and correctly merges a synthetic fat-waisted two-lobe blob that a size-only gate would wrongly keep split. The first version of that gate recomputed every region's mask + dilation + neighbor boundaries from scratch after every single merge -- fine for a few small fragments, but 2.4 hours on a real ~100M-voxel merged-nucleus object with ~50 initial peaks. Rewritten to compute all pairwise boundary info in one array pass, then do the merging via union-find on that small label-adjacency graph instead of rescanning the array -- same result (verified bit-identical), ~40x faster (8650s -> 216s on that object). Adds ComputeEDT: blockwise, multi-label-aware EDT over an entire segmentation, persisted as its own dataset. edt.edt() (Seung-lab) computes, per voxel, distance to the nearest *differently*-labeled voxel when given the raw labeled array directly -- exact per-instance EDT everywhere in one pass, not an approximation. A windowed EDT can only overestimate distance (background outside the window is invisible to it), so a fixed padding only needs to exceed the thinness threshold callers care about, not any object's physical scale -- validated on real data (id 2307, 9.25M voxels): 100% agreement with direct per-object EDT on the "<500nm" decision despite raw-value differences deep in thick interiors where the threshold doesn't care.
…ume gate Add dask_util helpers (resolve_concurrency_limit, rescale_idi_concurrency, wave_uses_shared_job_cpuset) so ImageDataInterface concurrency scales with actual available cores and per-worker-process fair share under a shared LSF/SLURM/SGE job cpuset, wired into SplitNarrowBridges and Skeletonize. Default SplitNarrowBridges's precompute_edt to False after root-causing a production KilledWorker failure in ComputeEDT's blockwise EDT precompute on jrc_mus-cerebellum-3 (unbounded padded block size against per-slot memory budget); per-object wave-scheduled EDT is already memory-safe. Remove max_object_volume_nm_3 (pre-split whole-object skip filter) -- conceptually at odds with splitting's purpose, since it discards exactly the oversized objects that need splitting. Use filter_ids or CleanConnectedComponents after the fact instead. Default minimum_subregion_volume_nm_3 to minimum_volume_nm_3 when unset, so a split producing an undersized piece is rejected up front (voxels stay attached to the parent) instead of being accepted and then silently deleted by the final volume filter.
Support a list of independent inputs (not necessarily channels of the same image -- e.g. separate prediction datasets) via consensus_config, each thresholded on its own intensity_threshold_minimum/maximum (and optionally its own gaussian_smoothing_sigma_nm/invert), then combined into one mask by a per-voxel vote: foreground if at least minimum_consensus_count of the inputs agree. Defaults to unanimous agreement across all inputs. Connected components then runs on the combined mask exactly as on a single thresholded input.
…rrectness TubeDirectionSplit adds simultaneous-cut, prominence, and axis-direction gates for splitting merged tubular objects (mitochondria) where radius-only reasoning fails on mesh-like skeleton topology. Also fixes MorphologicalOperations to support opening/closing with correct halo sizing and true-edge handling, adds an LRU byte-budgeted subpieces cache and vectorized spatial index to the write phase (previously ~2.8 blocks/sec on production data), and raises max_pieces_per_object to 1024.
…smatch snapping bug Measure now tracks a valid mask alongside raw intensity reads so ROI padding past a dataset's bounds isn't silently counted as intensity 0. Also fixes to_ndarray_tensorstore's grid snapping to anchor in offset-relative space, which previously read the wrong native voxels when offset wasn't itself a multiple of voxel_size.
Skips split attempts entirely for objects below a total-volume threshold, before any mask/EDT/skeleton work happens, rather than relying solely on per-cut gating after the fact. Also fixes a latent id-collision bug this surfaced: new split ids were derived from max(self.ids), which would have undercounted once small objects got filtered out of that list.
…to 0.5.0 Both CLI tools existed but were never listed in the README's tool table. Also documents the new minimum_object_volume_to_split_nm_3 pre-filter in the design doc, including why it doesn't repeat the removed max_object_volume_nm_3 mistake (skipping small objects up front is safe; skipping large ones isn't, since "too large" is usually the exact merged- object signature this tool exists to split).
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #86 +/- ##
==========================================
+ Coverage 84.15% 84.65% +0.50%
==========================================
Files 28 30 +2
Lines 4341 4945 +604
==========================================
+ Hits 3653 4186 +533
- Misses 688 759 +71 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Neither had real coverage: skeleton_graph had only two synthetic-geometry unit tests and no test of the cyclic topology it's documented to handle incorrectly; tube_direction had zero unit tests despite docstring claims about real mito data that were never captured as a reproducible test. Ship only edt_watershed, which has real-data validation on record. Design history for the removed strategies is kept in docs/split_narrow_bridges_plan.md in case they come back with real tests. Also clarifies in the README where split-narrow-bridges sits in the predictions-to-segmentations pipeline (downstream of connected-components/ mws, upstream of clean-connected-components).
docs/split_narrow_bridges_plan.md was a working design journal, not user-facing documentation -- doesn't belong in main. Inlined the handful of comments that pointed to it directly instead of leaving dangling references.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
@stuarteberg
Summary
Main feature:
split-narrow-bridges— splits accidentally-merged instances at thin necks, preserving every original foreground voxel and only relabeling. Works on any existing instance segmentation.neck_radius_nm: physical (nm) threshold — any cross-section narrower than this is a candidate neck to cut; thicker connections are left alone.neck_radius_mode="adaptive"can derive this per-object instead of using one fixed value for every object.minimum_object_volume_to_split_nm_3: skips objects too small to attempt a split at all, before any per-object work happens.edt_watershed; two others (skeleton_graph,tube_direction) were built but removed for lack of test coverage.compute-edtCLI tool precomputes the distance transform once for reuse across repeated runs.Unrelated improvements, but also included
ConnectedComponents.to_ndarray_tensorstore.