Astra instance-segmentation - #3006
Merged
PawelPeczek-Roboflow merged 9 commits intoSep 17, 2026
Merged
PawelPeczek-Roboflow merged 9 commits into
PawelPeczek-Roboflow merged 9 commits into
Conversation
`roboflow_core/open_ai@v7` gains an `instance-segmentation` task: the
model returns one outline polygon per instance of the requested classes
and the block decodes it into masked predictions
(`instance_segmentation_prediction` kind), so the result plugs straight
into mask/polygon visualization, crops and dataset upload.
Prompt contract (shared, in `common/vlm_decoding/segmentation.py`): a
flat `[x1, y1, x2, y2, ...]` vertex list in absolute pixel coordinates of
the uploaded image, wrapped as `{"segmentations": [{"label", "polygon"}]}`
and enforced via structured outputs. Chosen from a sweep on the 40
vlm-exam images that carry mask ground truth: absolute pixels, 0-1 floats
and 0-1000 ints tie on mask AP@50 (0.850 / 0.853 / 0.828, no significant
pairwise difference), but floats cost ~75% more output tokens and 0-1000
quantisation loses on small objects in large frames. The image is sent at
its ORIGINAL resolution as JPEG - unlike detection's 2048px downscale -
because pre-resizing cost -0.14 mask AP@50 on the images above the cap.
The task defaults to `gpt-6-astra` when the manifest omits
`model_version` (an explicit literal or selector is always honoured); the
block-wide `gpt-5.1` default predates the task.
Decoding is lenient in the same way as detection: `segmentations` /
`detections` wrappers or a bare list, flat / pair / point vertex shapes,
polygon key aliases, numeric strings, out-of-frame clamping; degenerate
polygons are skipped and an answer with no usable polygon surfaces as
`error_status`. Masks are rasterised with `sv.polygon_to_mask` at the
original resolution. Under `ENABLE_TENSOR_DATA_REPRESENTATION` the decode
is handed back as `InstanceDetections` with a dense boolean mask stack.
The shared `predictions` kind union grows by
`instance_segmentation_prediction`; the per-block tests pinning it are
updated accordingly.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
(cherry picked from commit f1d2462)
The shared `describe_vlm_prediction_outputs` / `actual_vlm_prediction_outputs` helpers hard-coded the `predictions` kind union, so widening it for segmentation made every VLM block (Claude, Gemini, Qwen, ...) advertise `instance_segmentation_prediction` although none of them can produce it, and dragged their tests along. The helpers now take the block's supported task list and derive the union from it (`prediction_kinds_for_tasks`); blocks that pass nothing keep the detection + classification union they always declared. Only `open_ai@v7` passes its task list, so it is the only block whose declared kinds change, and the other nine test suites are back to their `main` state. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit bbee50e)
- Skip polygons that enclose no area (repeated, collinear or clipped-away vertices): `fillPoly` would paint them as a one-pixel line and the serialiser would then drop the instance silently. Drop them at decode time, logged, and keep the empty-mask check as a second guard. - Rasterise into a preallocated `(N, H, W)` stack instead of a list plus `np.stack`, so the dense masks are never held twice. - Keep the frame size on `metadata[IMAGE_DIMENSIONS_KEY]` for an empty result, as `empty_detections_with_image_metadata` does, so the numpy and tensor-native representations agree. - Reword the default-model comment: the block default was never benchmarked for segmentation, it is not incapable of it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 8f009a0)
Review feedback: the (N, H, W) mask stack was sized by the raw entry count before polygons were validated, and dense masks cost 1 B/pixel/instance (100 instances on 4000x3000 is ~1.2 GB) with no ceiling. Rather than cap it, stop rasterising: every entry is validated first (well-formed polygon, non-zero enclosed area), then each survivor is encoded polygon -> COCO RLE with `pycocotools.frPyObjects` - no dense mask is ever materialised, so memory is proportional to the run count however many instances the model returns. The decoder now emits `mask=None` plus `data["rle_mask"]`, the RLE-first contract of `segment_anything3@v3` / `instance_segmentation@v4`: the visualization, crop and upload blocks decode lazily per instance, the Auto Label worker keeps native RLE losslessly, and the app stores RLE anyway. This also removes the previous lossy polygon -> raster -> contour -> polygon round trip in serialization. - `predictions` for the task is declared as `[rle_instance_segmentation_prediction, instance_segmentation_prediction]` (RLE first, dense kept so consumers declared on it still connect). - Boxes come from the RLE (`toBbox`) so box and mask always agree. - Tensor-native carrier is `InstanceDetections` with `InstancesRLEMasks`. - Test mask reader decodes RLE in both representations. Verified with a real Astra call through the ExecutionEngine on a 49-chip image: 49 instances, `mask=None`, 49 RLEs, serialized as `rle_mask`, mask + polygon visualizations rendered from RLE. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 816109e)
CI runs isort 5.13.2, which keeps the InstanceDetections import on one line where isort 8 wraps it; match the pinned version so static code analysis passes. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> (cherry picked from commit 2a7eb60)
PawelPeczek-Roboflow
requested review from
dkosowski87,
grzegorz-roboflow,
hansent,
probicheaux,
rafel-roboflow and
yeldarby
as code owners
September 16, 2026 18:01
…anyal runs skip mergability check witg main
Collaborator
Author
PawelPeczek-Roboflow
changed the title
Collaborator
Author
|
I am approving this PR to be merged to fast track and deployed. |
PawelPeczek-Roboflow
changed the title
PawelPeczek-Roboflow
merged commit Sep 17, 2026
a80e66e
into
fast-track/post-v1.6.0-pt1
7 checks passed
PawelPeczek-Roboflow
deleted the
feat/openai-segmentation-task-v1.6.0
branch
September 17, 2026 09:36
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Mirror of #2997 but with proper base commit and cherry-picked work from @Erol444 on top of last release.
Type of Change
Look at #2997
Testing
Test details:
Checklist
Additional Context