Repository navigation
AI-assisted annotation: a hosted model paints a plane, you review and accept it - #111
Draft
davidackerman wants to merge 10 commits into
Draft
davidackerman wants to merge 10 commits into
davidackerman wants to merge 10 commits into
Conversation
fill_unpainted only fills voxels that are 0. paint_box does the same by default, and with overwrite=True writes every voxel its labels cover, returning how many painted voxels it changed. Its undo entry is the one restore_box takes, so the view-labels Undo takes it back. fill_unpainted now calls it and behaves as before.
…nfig and keys A provider interface (a plane, a prompt and a colour in; a mask out) with two providers: Gemini on Vertex AI, signed in with Application Default Credentials so no key is held, and a local fake for tests and demos. The organelle catalog, prompts and colour-to-mask step come from the ai_annotate_gemini branch; half-strength catalog colours can now be found. Off unless ~/.cellmap_flow/ai_annotate.yaml enables it. The file names the providers and models the browser may pick from and holds no secret: an inline key is refused, a key file must be private to its owner, and every key read is blanked from the log, which the dashboard streams to the browser. A daily call limit, bounded retries on 429/503 only, timeouts, and a size and format check on returned images before they are decoded. A non-square plane is padded to a square for Gemini, which answers in fixed aspect ratios, and the answer cropped back.
geometry plans the plane around a point: the annotation voxels it writes, the raw it reads over the same extent, and the image sent, no more than 1024 pixels a side. Which plane follows the viewer's single-panel layout, read through its display dimensions (z, y, x in cellmap-flow's viewers, so neuroglancer's yz panel is the XY plane). pipeline reads it, builds the request and labels the mask's objects: an object continuing one in the plane before or after keeps its id, and new ids skip those planes'. staging keeps a result for review under a server-made id; audit appends each request, result and decision to the session's ai_annotate_log.jsonl.
…cept or reject Routes under /api/finetune/ai-annotate for the config, the settings, a run, its status, a resend with an edited prompt, and accept or reject. Shift+G runs at the voxel under the mouse. One job per session; nothing is sent until the user has acknowledged, for the dataset, the destination the page showed them, and a run or resend checks the model, the acknowledgement and the dataset prefixes again. Accept writes the plane into the annotation volume with paint_box and the view-labels undo stack: unlabelled voxels only, or with overwrite the model's objects over what is under them while its background still fills only 0s.
Provider, model and target pickers, the prompt prefilled from the catalog and editable, the destination to acknowledge, a run button beside the Shift+G hint, and the review: input, model output and overlay side by side, the prompt to edit and resend, Accept (with Overwrite existing labels) and Reject. Server and model text is only ever set as text.
…ate extra docs/ai_annotate.md: turning it on, Vertex AI setup, what leaves the cluster and where, keys, accept modes, prompts, the audit log and costs. The pixi environments install the extra; the lock adds google-genai and its few dependencies and moves websockets to 16.1.1, which it caps.
…d a quick start credentials_file under a vertex provider names a Google credentials file, such as a copy of gcloud's application-default login on group storage, so the dashboard signs in the same way on any cluster node with nothing to export first. The config holds only the path. The file is held to the key file rule (owned by you, chmod 600), read only when a call is made, its secret fields registered for redaction, and handed straight to the SDK. Left out, the SDK finds Application Default Credentials as before. docs/ai_annotate.md gains a quick start: sign in once, write the config, start the dashboard with pixi, acknowledge, Shift+G.
…ions does Label it, AI-assisted annotation and Good regions each open and close from their heading, as AI-assisted annotation already did, and stay as the user left them across reloads. The off message for AI-assisted annotation is shorter and its file path wraps inside the panel. Show Annotated Regions redraws the boxes once, on a click (a layer update drops the draw tool), so it is now Redraw Annotated Regions, styled as the patch tools' buttons are; the dark theme left the old outline all but invisible, so it read as a link.
…hdrawn, and offer a volume where one is missing An expired user login (Google's "Reauthentication is needed") now says to sign in again and, with a credentials_file, to copy the new login over it, instead of reporting no credentials at all. A call that fails before it reaches the model (signing in, the config) is taken back off the day's count, which the panel now says is the config's own cap. Unticking the acknowledgement withdraws it, so the next run asks again; settings saved without the field leave it as it is. Where an annotation volume is needed and missing (an AI annotation run or Shift+G, Seed, All Background, Split, Mark as Good), the answer carries needs_volume and the page offers to create one with New Volume.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This rebuilds the
ai_annotate_geminibranch's click-to-annotate feature on current main. That branch stays as a reference only: main has moved too far for a rebase.In the Finetune tab, with an annotation volume open:
Providers
Providers sit behind a small interface: a plane, a prompt and a colour go in, and a mask comes out. This PR ships two:
vertex_gemini(gemini-3-pro-image on Vertex AI). It signs in with Application Default Credentials, so the dashboard holds no key.fake, which runs locally, for tests and demos.The interface is built so OpenAI-style and self-hosted models can be added later.
Keys and what leaves the cluster
~/.cellmap_flow/ai_annotate.yamlturns it on. That file lists the providers and models the page may choose from, and the browser can't name an endpoint, project or key.credentials_file), e.g. a copy of gcloud's login on group storage so it works on any cluster node. The config holds only the path; the file must be the user's own andchmod 600, and the credentials go straight to the SDK.[REDACTED]in the log, which the dashboard streams to the browser.allowed_dataset_prefixesif that list is set.ai_annotate_log.jsonlin the session's corrections folder.Labels
Compared with the old branch
.envloader and the MinIO proxy that allowed PUT/DELETE are not carried over.Please check by hand
Shift+G in each single-panel layout sends the expected plane. The viewer's axes are z, y, x, so neuroglancer's yz layout is the XY plane, xz is XZ, and xy is YZ. In a 4-panel layout the plane of the top-left panel is used. The status line names the plane that was sent.
Not in this PR
minio/minio123) and a public bucket. That predates this PR.Dependency
ai-annotate = ["google-genai>=1.0"], installed in every pixi environment.websocketsbelow 17, so it moves from 17.2 to 16.1.1. Only uvicorn's optional extra uses websockets, and it accepts ≥13.docs/ai_annotate.mdcovers setup, costs and troubleshooting.