diff --git a/README.md b/README.md index f50d8e4..45a4b9d 100644 --- a/README.md +++ b/README.md @@ -48,7 +48,7 @@ agents, and container power users. | `runner/tao_ds.py` | Host-side Docker launcher used by the `tao_ds` shell function from `scripts/envsetup.sh`. | | `docker/` | Base development image Dockerfile, requirements, build script, and digest manifest. | | `release/` | Python package metadata plus release-container build scripts. | -| `ci/` and `.gitlab-ci.yml` | Static-test helpers and merge-request pipeline wiring. | +| `.github/workflows/` and `.pre-commit-config.yaml` | Pull-request checks: lint, license headers, DCO, README drift, and secret scan. | | `tests/` | Unit and integration-style tests for conversion, analytics, auto-label, mining, and config behavior. | | `tao-core/`, `tao-pytorch/` | In-repo submodules used by local development and container builds. | @@ -115,11 +115,11 @@ subtasks are discovered from each command package's `scripts/` directory. ### Base Image Source -`runner/tao_ds.py` and `ci/utils.py` resolve the immutable base image +`runner/tao_ds.py` resolves the immutable base image (`nvstaging/tao/data_services_base_image`) from `docker/manifest.json`, choosing the architecture-specific digest for the host. The pinned digests are intentionally not duplicated here — they -live in `docker/manifest.json` (and the CI / Jenkins / release files), and a static CI -check (`ci/run_static_tests.py`) verifies those digest references stay in sync. +live in `docker/manifest.json` and `release/docker/Dockerfile.release`; update both +together when the base image is rebuilt. ## Container Builds diff --git a/docs/agent_onboarding.md b/docs/agent_onboarding.md index d15900c..7693a16 100644 --- a/docs/agent_onboarding.md +++ b/docs/agent_onboarding.md @@ -1,6 +1,30 @@ # Agent Onboarding -Use this guide to get oriented without disturbing a user-owned worktree. +Use this guide to get oriented without disturbing a user-owned worktree. For a +full directory walk, read the [Codebase tour](codebase_tour.md). + +## Mental Model + +`scripts/envsetup.sh` sets `NV_TAO_DS_TOP` and defines a shell function named +`tao_ds`. That function runs `runner/tao_ds.py`, which starts the base Docker +image from `docker/manifest.json`, mounts this source tree at `/workspace`, and +executes any command passed after `--`. + +Inside the container, `setup.py` installs one console script per function +(refer to Terminology in [Architecture](architecture.md)): +`annotations`, `augmentation`, `analytics`, `auto_label`, `image`, +`gap_analysis`, `tmm`, and `embedding`. All of them use the shared command dispatcher in +`nvidia_tao_ds/core/entrypoint/entrypoint.py` to discover subtasks from +`scripts/`, require `-e/--experiment_spec_file`, and run the selected script as +a fresh Python subprocess under Hydra. + +```text +console command (setup.py) + -> nvidia_tao_ds/core/entrypoint/entrypoint.py + -> nvidia_tao_ds//scripts/.py (fresh subprocess) + -> Hydra spec validated against nvidia_tao_ds/config// + -> function logic (conversion, DALI, model inference, mining, ...) +``` ## First Pass @@ -12,54 +36,53 @@ git -c filter.lfs.process= -c filter.lfs.required=false status --short --branch git remote -v find . -maxdepth 2 -type d sed -n '1,220p' README.md -sed -n '1,220p' .gitlab-ci.yml rg -n "console_scripts|entry_points" setup.py rg -n "ArgumentParser|manifest.json|--gpus|--tag|--run_as_user" runner scripts docker release rg -n "hydra_runner|default_specs|get_subtasks|entrypoint|console_scripts" nvidia_tao_ds setup.py -rg -n "pytest|run_static|pre-commit|flake8|pylint" .gitlab-ci.yml ci tests +ls .github/workflows/ +sed -n '1,80p' .pre-commit-config.yaml ``` -Treat `tao-core/` and `tao-pytorch/` as submodules or vendored source in this -checkout. They may be dirty for reasons unrelated to your task. Do not reset, -update, or rewrite them unless the task explicitly requires it. +Treat `tao-core/` and `tao-pytorch/` as submodules in this checkout. They may +be dirty for reasons unrelated to your task. Do not reset, update, or rewrite +them unless the task explicitly requires it. Initialize them before running +anything: -## Mental Model - -`scripts/envsetup.sh` sets `NV_TAO_DS_TOP` and defines a shell function named -`tao_ds`. That function runs `runner/tao_ds.py`, which starts the base Docker -image from `docker/manifest.json`, mounts this source tree at `/workspace`, and -executes any command passed after `--`. - -Inside the container, `setup.py` installs package console scripts such as -`annotations`, `augmentation`, `analytics`, `auto_label`, `image`, -`gap_analysis`, `tmm`, and `embedding`. Most commands use the shared launcher in -`nvidia_tao_ds/core/entrypoint/entrypoint.py` to discover subtasks from -`scripts/`, require `-e/--experiment_spec_file`, and pass the selected YAML to -Hydra. Mining and RCCA commands use thinner dispatchers that forward Hydra -arguments directly to their selected script. +```sh +git submodule update --init +``` ## Source Truths -| Question | Source Of Truth | +| Question | Source of truth | | :--- | :--- | | Which package commands exist? | `setup.py` `console_scripts` | | Which host launcher flags exist? | `runner/tao_ds.py` `parse_cli_args` | | Which base image is pulled? | `docker/manifest.json` | -| Which subtasks exist? | `nvidia_tao_ds/$DOMAIN/scripts/*.py` | -| Which example specs exist? | `nvidia_tao_ds/$DOMAIN/experiment_specs/*.yaml` | -| Which dataclass schema is used? | `nvidia_tao_ds/config/$DOMAIN/...` | -| Which static tests run in GitLab? | `.gitlab-ci.yml` and `ci/run_static_tests.py` | +| Which subtasks exist? | `nvidia_tao_ds//scripts/*.py` | +| Which example specs exist? | `nvidia_tao_ds//experiment_specs/*.yaml` | +| Which specification template does a subtask load? | The script's `@hydra_runner(config_name=...)`, not the subtask name | +| Which dataclass schema is used? | `nvidia_tao_ds/config//...` (`analytics` is the configuration package for `data_analytics/`) | +| Which static tests run in CI? | `.pre-commit-config.yaml` via `.github/workflows/static-tests.yml` | | Which README content is generated? | `tools/update_readme_supported_commands.py` | -## Worktree Safety +## Common Agent Questions + +| Question | Where to look | +| :--- | :--- | +| Where is the shared dispatcher behavior (GPU handling, subprocess, telemetry)? | `nvidia_tao_ds/core/entrypoint/entrypoint.py` | +| Why does a CPU command need a GPU host? | The dispatcher calls `nvidia-smi` unconditionally | +| Why does my new subtask ignore `num_gpus`? | Multi-GPU is keyed on the literal subtask name `generate` | +| Why is there no `status.json` for a subtask? | The script lacks `@monitor_status`; refer to the coverage list in the [Codebase tour](codebase_tour.md) | +| Which model does auto-label use? | `auto_label/scripts/generate.py` dispatches on `cfg.autolabel_type` | +| Where do LLM/VLM calls happen? | `nvidia_tao_ds/core/llm_clients/` | +| How does the API relate to the CLI? | Dev-mode Flask app versus the tao-core microservice; refer to [Architecture](architecture.md) | -Before editing, capture the status and decide which files you own. The docs -rollout normally touches `README.md`, `docs/`, `tools/update_readme_supported_commands.py`, -`.pre-commit-config.yaml`, and `.gitlab-ci.yml`. +## Worktree Safety -Avoid broad cleanup. Do not remove user-created tests, local examples, cache -directories outside your own generated outputs, or submodule changes unless the -user asks. +Before editing, capture the status and decide which files you own. Avoid broad +cleanup. Do not remove user-created tests, local examples, cache directories +outside your own generated outputs, or submodule changes unless the user asks. ## Targeted Checks @@ -68,10 +91,13 @@ For documentation and generated README changes: ```sh python tools/update_readme_supported_commands.py --check python -m py_compile tools/update_readme_supported_commands.py -git diff --check -- README.md docs/*.md docs/assets/*.svg tools/*.py .pre-commit-config.yaml .gitlab-ci.yml +git diff --check -- README.md docs/*.md docs/assets/*.svg tools/*.py .pre-commit-config.yaml rg -n "TBD|PLACEHOLDER|example\\.com" README.md docs ``` -For source-adjacent command or config changes, add focused pytest runs from -[testing_and_debugging.md](testing_and_debugging.md). GPU, Docker, private -checkpoint, NGC, and full dataset tests are usually outside a docs-only change. +For source-adjacent command or configuration changes, run the same static +checks CI runs (`pre-commit run` on the changed files) and add focused pytest +runs from [Testing and debugging](testing_and_debugging.md). GPU, Docker, private +checkpoint, NGC, and full dataset tests are usually outside a documentation-only change. + +All commits need a DCO sign-off (`git commit -s`); CI enforces it. diff --git a/docs/architecture.md b/docs/architecture.md index 3377b80..8ae0821 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -1,95 +1,237 @@ # Architecture -TAO Data Services is a source package plus a Dockerized runtime. Host users -enter through `tao_ds`; in-container users enter through package console scripts. +This guide explains what TAO Data Services is, how customers run it, and how a +command flows through the source code. For a directory-level orientation, read +the [Codebase tour](codebase_tour.md) first. + +## What TAO Data Services Is + +TAO Data Services is the dataset-preparation backend of the TAO ecosystem: a +collection of dataset-preparation functions covering annotation conversion, data augmentation, +auto-labeling, dataset analytics, data mining, and model-gap analysis. This +repository is its source. It ships to users as one container, +`nvcr.io/nvidia/tao/tao-toolkit:-dataservices`, alongside the sibling +backends built from tao-pytorch (training) and tao-deploy (TensorRT inference). + +The product surface is the set of console commands this package installs +(`annotations`, `augmentation`, `auto_label`, `analytics`, `image`, +`embedding`, `tmm`, `gap_analysis`) plus their subtasks. Everything else in +this repository exists to build, validate, or develop that container. + +## How Users Run Data Services + +As of TAO 7.0 there is one supported user surface, with the container CLI +underneath it: + +1. **Agent and TAO skills (current).** Users load the `tao-skills` plugin in a + coding agent and ask for the outcome ("convert these COCO annotations to + KITTI"). The skills and the TAO Execution SDK dispatch a job to the user's + compute backend (local Docker, DGX Cloud Lepton, Brev, SLURM, or + Kubernetes), which runs the data-services container and invokes the same + console commands documented here. +2. **Container CLI (the layer underneath).** Inside the + `-dataservices` container, users or automation run the commands + directly: `annotations convert -e spec.yaml`, `embedding image_embeddings + -e spec.yaml`, and so on. The public data-mining and auto-label + documentation shows this form. + +Two older surfaces were **removed in TAO 7.0** and should not appear in new +code or documentation: the **TAO Launcher** (`tao dataset annotations convert +...`, deprecated in 6.0) and **FTMS** (the Fine-Tuning MicroService REST API +plus the `nvidia-tao-client` CLI/SDK). This repository still contains +FTMS-era code paths (`nvidia_tao_ds/api/`, the microservice `CMD` in the +release Dockerfile, `JOB_ID`-gated log teeing); treat them as legacy +integration surfaces, not as the way users run the product. + +## Terminology + +| Term | Meaning here | +| :--- | :--- | +| Function | One dataset-preparation capability with its own console command and package, such as `annotations` or `augmentation`. Functions are classified by compute: **GPU functions** (`augmentation`, `auto_label`, `embedding`, `tmm`) and **CPU functions** (`annotations`, `analytics`, `image`, `gap_analysis`). Despite the product name "Data Services," these are batch CLI functions, not long-running services. | +| Subtask | One operation of a function, implemented as one module in the function's `scripts/` package and selected as the first CLI argument: `annotations convert`, `analytics analyze`. | +| Specification (spec) | The YAML experiment file passed with `-e`, validated against the function's dataclass schema. | +| Shared command dispatcher | `nvidia_tao_ds/core/entrypoint/entrypoint.py` — the in-repo code every console command delegates to. Not a product. | +| `tao_ds` | The **development-only** container launcher defined by `scripts/envsetup.sh` (a shell function over `runner/tao_ds.py`). Users never see it; public docs reuse the name as a nickname for the data-services container itself. | +| TAO Launcher | The removed `tao` CLI product (deprecated 6.0, removed 7.0). Not related to `tao_ds` or to the dispatcher above. | +| FTMS | The removed Fine-Tuning MicroService REST API and `nvidia-tao-client`. Its in-repo remnants are noted above. | + +## Developer Runtime: From Console Command to Script ![Runtime dispatch flow](assets/runtime_flow.svg) -## Runtime Dispatch +For development, `tao_ds` (dev-only, see Terminology) stands in for whatever +runs the container in production: 1. `source scripts/envsetup.sh` exports `NV_TAO_DS_TOP` and defines `tao_ds`. 2. `tao_ds` runs `runner/tao_ds.py`, resolves the base image from - `docker/manifest.json`, mounts the repository as `/workspace`, sets - `PYTHONPATH=/workspace:$PYTHONPATH`, and starts Docker with the requested - GPUs, mounts, environment variables, shared memory, ulimits, UID/GID, and - optional service-mode ports. + `docker/manifest.json`, mounts the repository as `/workspace`, and starts + Docker with the requested GPUs, mounts, environment variables, shared + memory, ulimits, UID/GID, and optional service-mode ports. 3. Commands after `tao_ds --` execute inside the container. -4. `setup.py` installs package commands for data-service domains. -5. Standard commands call `nvidia_tao_ds/core/entrypoint/entrypoint.py`, which - discovers subtask modules from each package's `scripts/` directory, validates - `-e/--experiment_spec_file`, injects `--config-path` and `--config-name`, and - launches the selected Python script. -6. Script modules use `nvidia_tao_ds/core/hydra/hydra_runner.py` to run Hydra, - disable `.hydra` output directories, register optional dataclass schemas, and - preserve stdout-oriented logging. - -## Command Families - -| Command | Package | Dispatch Pattern | -| :--- | :--- | :--- | -| `augmentation` | `nvidia_tao_ds/augmentation` | Shared launcher with `-e`; MPI path for multi-GPU `generate`. | -| `auto_label` | `nvidia_tao_ds/auto_label` | Shared launcher with `-e`; `torchrun` path for multi-GPU `generate`. | -| `annotations` | `nvidia_tao_ds/annotations` | Shared launcher with `convert`, `merge`, `slice`, and QA conversion subtasks. | -| `analytics` | `nvidia_tao_ds/data_analytics` | Shared launcher; config package is named `analytics`. | -| `image` | `nvidia_tao_ds/image` | Shared launcher for image validation. | -| `embedding` | `nvidia_tao_ds/mining/embedding` | Direct dispatcher that forwards Hydra args to `image_embeddings`. | -| `tmm` | `nvidia_tao_ds/mining/tmm` | Direct dispatcher that forwards Hydra args to `nearest_neighbors`. | -| `gap_analysis` | `nvidia_tao_ds/rcca/gap_analysis` | Direct dispatcher for `vcn_aoi` and `vlm_bcq`. | - -`get_subtasks()` also wires a shared `default_specs` helper. Before documenting -or relying on that helper for a domain, check -`nvidia_tao_ds/core/utils/default_specs.py` and the matching -`nvidia_tao_ds/config/` layout; nested domains such as mining and RCCA do not -follow the same flat module pattern as `annotations` or `augmentation`. + +From here on the flow is identical for users and developers: + +4. `setup.py` installs one console script per function (`annotations`, + `augmentation`, `auto_label`, `analytics`, `image`, `embedding`, `tmm`, + `gap_analysis`). +5. Every console script is a thin argparse shell over the shared command + dispatcher in `nvidia_tao_ds/core/entrypoint/entrypoint.py`, which: + * Discovers subtask modules from the function's `scripts/` package and + injects the synthetic `default_specs` subtask. + * Validates `-e/--experiment_spec_file` and converts it into Hydra + `--config-path`/`--config-name` flags, passing unknown CLI tokens through + as Hydra overrides. + * Applies multi-GPU settings only for subtasks in `multigpu_support` + (default: `['generate']`) and, when more than one GPU is requested, + selects the runner: `mpirun` for `augmentation`, `torchrun` for + `auto_label`, and plain `python` otherwise. + * Spawns the selected script as a **fresh Python subprocess**, streaming + stdout (and to `$TAO_MICROSERVICES_TTY_LOG/$JOB_ID/microservices_log.txt` + when `JOB_ID` is set). + * Reports telemetry and determines success from **both** the exit code and + the last record of `results_dir/status.json`. +6. Script modules use `nvidia_tao_ds/core/hydra/hydra_runner.py` to register + the dataclass schema and run Hydra, and `@monitor_status` from + `core/decorators.py` to create the results directory, write `status.json`, + and map exceptions to user-facing status messages. + +## Function Inventory + +| Function | Package | Dispatch pattern | Compute | +| :--- | :--- | :--- | :--- | +| `annotations` | `nvidia_tao_ds/annotations` | Shared dispatcher: `convert`, `merge`, `slice`, `qa_to_llava_annotation`. | CPU | +| `augmentation` | `nvidia_tao_ds/augmentation` | Shared dispatcher; DALI pipelines; MPI path for multi-GPU `generate`. | GPU | +| `auto_label` | `nvidia_tao_ds/auto_label` | Shared dispatcher; `torchrun` path for multi-GPU `generate`; fans out on `autolabel_type`. | GPU or remote LLM | +| `analytics` | `nvidia_tao_ds/data_analytics` | Shared dispatcher: `analyze`, `validate`, `kpi_analyze`. Config package is named `analytics`. | CPU | +| `image` | `nvidia_tao_ds/image` | Shared dispatcher: `validate` (corrupted-image removal). | CPU | +| `embedding` | `nvidia_tao_ds/mining/embedding` | Shared dispatcher: `image_embeddings`, `text_embeddings` (CLIP and SigLIP to Parquet). | GPU | +| `tmm` | `nvidia_tao_ds/mining/tmm` | Shared dispatcher: `nearest_neighbors`, `unique_neighbor_matching` (RAPIDS). | GPU | +| `gap_analysis` | `nvidia_tao_ds/rcca/gap_analysis` | Shared dispatcher: `object_detection`, `vcn_aoi`, `vlm_bcq`. | CPU | + +`get_subtasks()` wires a shared `default_specs` helper into every command, but +it only works for the flat functions (`annotations`, `augmentation`, +`auto_label`, `image`, `analytics`); `nvidia_tao_ds/core/utils/default_specs.py` does not support the nested +mining and RCCA domains. ## Configuration Flow -Most command scripts combine three files: +![Configuration flow](assets/config_flow.svg) + +Most command scripts combine four pieces: | Layer | Example | Purpose | | :--- | :--- | :--- | -| Entrypoint wrapper | `nvidia_tao_ds/annotations/entrypoint/annotations.py` | Parses the subtask and delegates to the shared launcher. | +| Entrypoint wrapper | `nvidia_tao_ds/annotations/entrypoint/annotations.py` | Parses the subtask and delegates to the shared dispatcher. | | Script subtask | `nvidia_tao_ds/annotations/scripts/convert.py` | Owns task logic and the `@hydra_runner(...)` declaration. | | Experiment spec | `nvidia_tao_ds/annotations/experiment_specs/annotations.yaml` | Example YAML selected by `-e` or direct Hydra overrides. | -| Dataclass schema | `nvidia_tao_ds/config/annotations/default_config.py` | Structured defaults and schema used by Hydra/API/default-spec generation. | +| Dataclass schema | `nvidia_tao_ds/config/annotations/default_config.py` | Structured defaults and schema used by Hydra, the API, and default-spec generation. | -Do not infer spec names from subtask names. The source of truth is each -script's `@hydra_runner(config_name=...)`. For example, the `annotations` -`convert` subtask uses `config_name="annotations"`, while `qa_to_llava_annotation` -uses `config_name="qa_to_llava"`. +The configuration precedence at runtime is: -## Shared Services +```text +dataclass defaults -> experiment YAML (-e) -> command-line Hydra overrides +``` -| Module | Responsibility | -| :--- | :--- | -| `nvidia_tao_ds/core/entrypoint/entrypoint.py` | Subtask discovery, spec path conversion, GPU override precedence, process launch, telemetry, and log teeing. | -| `nvidia_tao_ds/core/hydra/hydra_runner.py` | Local wrapper around Hydra's internal runner with schema registration and TAO-friendly logging overrides. | -| `nvidia_tao_ds/core/utils/default_specs.py` | Default experiment YAML generation from dataclass configs. | -| `nvidia_tao_ds/core/logging/` | TAO Data Services logging helpers. | -| `nvidia_tao_ds/core/llm_clients/` | OpenAI-compatible, Gemini, and base LLM client abstractions used by auto-label workflows. | +Two schema conventions coexist: + +* **Flat functions** keep their schemas in `config//`, anchored by a + `default_config.py::ExperimentConfig`. Subtasks may add sibling modules: + `annotations` pairs `ExperimentConfig` (convert) with `merge_config.py` and + `slice_config.py`, while `qa_to_llava_annotation` defines its configuration + inline in the script. +* **Newer nested functions** (mining, rcca) define one configuration module per + subtask, named after the subtask (for example, + `config/mining/tmm/nearest_neighbors.py::NearestNeighborsConfig`). -## API Service +Schemas declare their fields through typed factories from `config/utils/types.py` +(`STR_FIELD`, `INT_FIELD`, `FLOAT_FIELD`, `BOOL_FIELD`, `LIST_FIELD`, +`DICT_FIELD`, `DATACLASS_FIELD`, ...). Each factory attaches metadata — +`description`, `display_name`, `valid_options`, `valid_min`/`valid_max`, +`default_value`, and `automl_enabled`) consumed by the API schema layer and +default-specification generation. -`nvidia_tao_ds/api/app.py` is a Flask application with OpenAPI endpoints under -`/api/v1`. It exposes neural-network action discovery, schema lookup, action -submission, job status, health, OpenAPI JSON/YAML, Swagger, and Redoc routes. +Do not infer specification names from subtask names. The source of truth is each +script's `@hydra_runner(config_name=...)`. For example, `annotations convert` +uses `config_name="annotations"`, `qa_to_llava_annotation` uses +`config_name="qa_to_llava"`, and `augmentation generate` uses +`config_name="kitti"`. -The API uses `nvidia_tao_core.api_utils.module_utils` to map installed console -scripts to actions, `dataclass2json_converter` for schema materialization, and -`json_schema_validation` before queueing jobs. A background thread processes the -shared `process_queue`. Service-mode container launch is handled by -`runner/tao_ds.py --run_as_service`. +## Models and Weights + +The functions orchestrate models from other TAO repositories rather than +defining their own: + +| Consumer | Model | Source | +| :--- | :--- | :--- | +| `auto_label generate` (`autolabel_type=grounding_dino`) | Grounding DINO (text-prompted boxes, iterative refinement) | `tao-pytorch` submodule (`nvidia_tao_pytorch.cv.grounding_dino`); user-supplied checkpoint | +| `auto_label generate` (`autolabel_type=mal`) | MAL (box-to-segmentation pseudo-labels) | `tao-pytorch` submodule (`nvidia_tao_pytorch.cv.mal`); PL checkpoint | +| `auto_label generate` (VLM workflows) | Remote VLM/LLM (Gemini or any OpenAI-compatible endpoint) | `core/llm_clients/` (`create_client`); needs `GOOGLE_API_KEY` or `OPENAI_API_KEY` | +| `embedding image_embeddings` / `text_embeddings` | CLIP and SigLIP | Hugging Face `transformers` by model ID, or a TAO CLIP checkpoint via `nvidia_tao_pytorch.multimodal.clip` | + +The schema side mirrors this: `config/auto_label/default_config.py` composes +Grounding DINO and MAL configuration dataclasses imported from `nvidia_tao_core`. + +## Shared Services + +| Module | Responsibility | +| :--- | :--- | +| `nvidia_tao_ds/core/entrypoint/entrypoint.py` | Subtask discovery, spec path conversion, GPU override precedence, process launch, telemetry, log teeing, status-based failure detection. | +| `nvidia_tao_ds/core/hydra/hydra_runner.py` | Local wrapper around Hydra's runner with schema registration and TAO-friendly logging overrides. | +| `nvidia_tao_ds/core/decorators.py` | `@monitor_status` (results dir, `status.json`, error classification) and `@experimental`. | +| `nvidia_tao_ds/core/utils/default_specs.py` | Default experiment YAML generation from dataclass configs (flat functions only). | +| `nvidia_tao_ds/core/logging/` | `StatusLogger` and dual logging into `status.json`. | +| `nvidia_tao_ds/core/llm_clients/` | `LLMClient` ABC, Gemini and OpenAI-compatible clients, `create_client` factory used by auto-label workflows. | +| `nvidia_tao_ds/core/utils/dataset_loading.py` | Shared COCO/KITTI loading used across functions. | + +## API Service (Legacy FTMS Integration) + +FTMS was removed as a product surface in TAO 7.0, but its integration code is +still in this repository. There are two API stories, and confusing them is the +most common orientation mistake in this repository: + +* **Dev-mode API:** `nvidia_tao_ds/api/app.py` is a self-contained Flask app + exposing neural-network action discovery, schema lookup, action submission, + job status, and health routes under `/api/v1`, plus OpenAPI, Swagger, and + Redoc routes at the root. It maps + installed console scripts to actions via + `nvidia_tao_core.api_utils.module_utils`, validates specifications with + `json_schema_validation`, and processes jobs on a background queue thread. It + is reachable only through the dev launcher: `tao_ds --run_as_service`. The + contract snapshot is `nvidia_tao_ds/api/openapi.json`. +* **Production microservice:** the release container + (`release/docker/Dockerfile.release`) does **not** run `app.py`. It sets + `FLASK_APP=nvidia_tao_core.microservices.app` and runs tao-core's + microservice; the broader control plane (datasets, experiments, job + orchestration) lives in TAO API, not this repository. + +## TAO Core and TAO PyTorch Boundaries + +Both are git submodules; initialize them before running anything: + +```sh +git submodule update --init +``` + +* **`tao-core/`** provides telemetry (soft-imported), status callbacks + (imported by `core/logging/logging.py`), API utilities used by `api/app.py`, + and the Grounding DINO and MAL configuration dataclasses composed into + `config/auto_label/`. The release image builds and installs its wheel and + uses its microservice app as the container entrypoint. For local + development, `scripts/envsetup.sh` puts `/workspace/tao-core` on + `PYTHONPATH`. +* **`tao-pytorch/`** supplies the actual model code for Grounding DINO, MAL, + and TAO CLIP checkpoints consumed by `auto_label` and `embedding`. ## Extension Points Add a new workflow by choosing the smallest surface: -| Change | Typical Files | +| Change | Typical files | | :--- | :--- | -| New subtask in an existing command | `nvidia_tao_ds/$DOMAIN/scripts/$SUBTASK.py`, `experiment_specs/`, `config/`, tests. | -| New top-level command | New package with `entrypoint/`, `scripts/`, `experiment_specs/`, config package, `setup.py` entry, tests, README generator update. | -| New API-visible action | Command/config support plus `api/app.py` and any `tao-core` module utility mapping changes. | -| New container dependency | `docker/requirements-pip.txt`, `docker/Dockerfile`, `docker/build.sh`, `docker/manifest.json` after push. | +| New subtask in an existing command | `nvidia_tao_ds//scripts/.py`, `experiment_specs/`, `config/`, tests. Remember: multi-GPU only applies if the subtask is named `generate`, and `status.json` requires `@monitor_status`. | +| New top-level command | New package with `entrypoint/`, `scripts/`, `experiment_specs/`, a config package, a `setup.py` entry, tests, and a README-generator run. | +| New API-visible action | Command/config support plus `api/app.py` and any tao-core module-utility mapping changes. | +| New container dependency | `docker/requirements-pip.txt`, `docker/Dockerfile`, then `docker/build.sh` and a `docker/manifest.json` digest update after push. Video/codec dependencies must respect the FFmpeg codec allow-list baked into the Dockerfile. | -Use [new_data_service_command.md](new_data_service_command.md) before adding a -new command or promoting a pattern as canonical. +Refer to [New data-service command](new_data_service_command.md) before adding +a new command or promoting a pattern as canonical. diff --git a/docs/assets/config_flow.svg b/docs/assets/config_flow.svg new file mode 100644 index 0000000..d806e36 --- /dev/null +++ b/docs/assets/config_flow.svg @@ -0,0 +1,56 @@ + + TAO Data Services configuration flow + Configuration flows from dataclass defaults to experiment YAML to Hydra overrides and into service scripts. + + + + + + + + Configuration Flow + + + Dataclass schema + config/<service> + defaults + metadata + + + Experiment YAML (-e) + experiment_specs/ + named by config_name + + + Hydra overrides + results_dir=/results + CLI dotlist + + + Structured cfg + cfg passed to + script main(cfg) + + + default_specs + flat services only + + + Service code + convert/augment/label + + + + + + + + + diff --git a/docs/assets/module_map.svg b/docs/assets/module_map.svg new file mode 100644 index 0000000..ec6703d --- /dev/null +++ b/docs/assets/module_map.svg @@ -0,0 +1,87 @@ + + TAO Data Services package module map + The nvidia_tao_ds package groups service families over shared launcher, config, logging, and LLM-client modules, with models supplied by submodules. + + + + + + + + nvidia_tao_ds Module Map + + + SERVICE FAMILIES (one console script each) + + + CPU services + annotations (convert/merge/slice) + data_analytics ("analytics") + image, rcca/gap_analysis + + + GPU services + augmentation (DALI/mpirun) + auto_label (torchrun) + mining/embedding, mining/tmm + + + config/ + flat: default_config.py + nested: per-subtask modules + utils/types.py: *_FIELD factories + + + api/ + dev-mode Flask app + (tao_ds --run_as_service) + openapi.json contract + + + SHARED CORE + + + core/entrypoint/entrypoint.py + subtask discovery, spec-to-Hydra, GPU + handling, subprocess launch, telemetry + + + core/ + hydra_runner, monitor_status, + StatusLogger, default_specs + + + core/llm_clients/ + Gemini + OpenAI-compatible + clients for VLM auto-labeling + + + EXTERNAL MODEL AND INFRA SOURCES (git submodules) + + + tao-pytorch/ + Grounding DINO, MAL, TAO CLIP model code + used by auto_label and embedding + + + tao-core/ + telemetry, status callbacks, api_utils, GDINO/MAL config + dataclasses; release-container microservice app + + + + + + + + diff --git a/docs/codebase_tour.md b/docs/codebase_tour.md new file mode 100644 index 0000000..70fdf29 --- /dev/null +++ b/docs/codebase_tour.md @@ -0,0 +1,209 @@ +# Codebase Tour + +This is a guided walk through the TAO Data Services repository for developers +picking up the codebase for the first time. It answers three questions: what +each directory is for, how the Python package is organized into modules, and +where the sharp edges are. + +For what the product is, how customers run it, and definitions of the terms +used below (function, subtask, dispatcher, `tao_ds`), read the opening sections +of [Architecture](architecture.md) first; this repository builds the +`nvcr.io/nvidia/tao/tao-toolkit:-dataservices` container, and the +console commands below are its user-facing surface. + +![TAO Data Services module map](assets/module_map.svg) + +## Repository Layout + +```text +tao-dataservices/ +├── nvidia_tao_ds/ # The installable Python package (everything shipped in the wheel) +│ ├── annotations/ # Format conversion, merge, slice (CPU) +│ ├── augmentation/ # DALI-based data augmentation (GPU, MPI multi-GPU) +│ ├── auto_label/ # Pseudo-label generation: Grounding DINO, MAL, VLM workflows (GPU / remote LLM) +│ ├── data_analytics/ # Dataset analytics, validation, KPI analysis (command name: analytics) +│ ├── image/ # Corrupted-image validation (CPU) +│ ├── mining/ # embedding (CLIP/SigLIP -> parquet) and tmm (RAPIDS nearest-neighbor mining) +│ ├── rcca/ # gap_analysis: model-failure / coverage analysis +│ ├── core/ # Shared command dispatcher, Hydra runner, decorators, logging, LLM clients +│ ├── config/ # Hydra dataclass schemas, one package per function +│ ├── api/ # Dev-mode Flask API (refer to the API section in Architecture) +│ ├── backbone/ # Vendored FAN/ConvNeXt/Swin/ViT code (legacy, unused elsewhere) +│ └── dataclass_to_rst/ # Doc tooling: config dataclasses -> RST tables for public docs +├── runner/tao_ds.py # Host-side Docker launcher (not shipped in the wheel) +├── scripts/envsetup.sh # Defines the tao_ds shell function; installs git hooks +├── docker/ # Base development image: Dockerfile, manifest.json, requirements +├── release/ # Release image, version metadata (release/python/version.py) +├── internal/ # Unpackaged dev scripts (dataset split, KITTI visualization) +├── tools/ # update_readme_supported_commands.py (generates the README command table) +├── tests/ # pytest suites (flat files + tests/autolabel + tests/mining) +├── docs/ # This documentation +├── .github/ # GitHub Actions workflows and shared hook scripts +├── tao-core/ # Git submodule: shared TAO config/microservice/telemetry code +├── tao-pytorch/ # Git submodule: model code consumed by auto_label and embedding +├── setup.py # Package definition and console_scripts (one per function) +└── Makefile # Wheel build targets (build, install, develop, clean) +``` + +The rules of thumb are: + +* If it ships to users, it lives under `nvidia_tao_ds/`. +* If it launches or builds containers, it lives in `runner/`, `docker/`, or + `release/`. +* If it validates the repository, it lives in `tests/`, `.pre-commit-config.yaml`, or + `.github/workflows/`. + +## The Package in One Table + +| Module | Role | Key files | +| :--- | :--- | :--- | +| `/entrypoint/` | An approximately 35-line argparse shell delegating to the shared dispatcher. | For example, `annotations/entrypoint/annotations.py` | +| `/scripts/` | One module per subtask; each owns its `@hydra_runner(...)` declaration. | For example, `annotations/scripts/convert.py` | +| `/experiment_specs/` | Example YAML specifications selected by `-e`. | For example, `annotations/experiment_specs/annotations.yaml` | +| `core/entrypoint/entrypoint.py` | The shared command dispatcher: subtask discovery, spec-to-Hydra translation, GPU handling, subprocess launch, telemetry, and status-based failure detection. | `get_subtasks`, `launch` | +| `core/hydra/hydra_runner.py` | Schema registration and Hydra invocation. | `hydra_runner` decorator | +| `core/decorators.py` | `@monitor_status` (results dir, `status.json`, error classification) and `@experimental`. | | +| `core/logging/` | `StatusLogger`, dual logging into `status.json`. | `logging.py` | +| `core/llm_clients/` | LLM/VLM client abstraction used by auto-label workflows. | `LLMClient` ABC, `GeminiClient`, `OpenAICompatibleClient`, and the `create_client` factory | +| `core/utils/` | `default_specs` pseudo-subtask, shared COCO/KITTI loading, video helpers. | `default_specs.py`, `dataset_loading.py`, `video_utils.py` | +| `config//` | Dataclass schemas (refer to [Two Configuration Conventions](#two-configuration-conventions)). | `default_config.py` and per-subtask modules | +| `config/utils/types.py` | Typed field factories (`STR_FIELD`, `INT_FIELD`, `DATACLASS_FIELD`, ...) that attach UI/validation metadata. | | +| `api/` | Dev-mode Flask app exposing the functions as API actions. | `app.py`, `openapi.json` | + +## Function Inventory + +Despite the product name "Data Services," the units of this repository are +batch CLI **functions**, not long-running services; they are classified by +compute as GPU functions or CPU functions. Each console script (registered in +`setup.py`) maps to one function package: + +| Command | Package | Subtasks | Runs on | +| :--- | :--- | :--- | :--- | +| `annotations` | `annotations/` | `convert`, `merge`, `slice`, `qa_to_llava_annotation` | CPU | +| `augmentation` | `augmentation/` | `generate` | GPU (DALI; multi-GPU via `mpirun`) | +| `auto_label` | `auto_label/` | `generate` (dispatches on `autolabel_type`) | GPU (`torchrun`) or remote LLM APIs | +| `analytics` | `data_analytics/` | `analyze`, `validate`, `kpi_analyze` | CPU | +| `image` | `image/` | `validate` | CPU | +| `embedding` | `mining/embedding/` | `image_embeddings`, `text_embeddings` | GPU | +| `tmm` | `mining/tmm/` | `nearest_neighbors`, `unique_neighbor_matching` | GPU (RAPIDS `cudf`/`cuml`) | +| `gap_analysis` | `rcca/gap_analysis/` | `object_detection`, `vcn_aoi`, `vlm_bcq` | CPU | + +The `auto_label generate` subtask fans out on `cfg.autolabel_type`: + +| `autolabel_type` | What it does | Model source | +| :--- | :--- | :--- | +| `grounding_dino` | Text-prompted box pseudo-labels with iterative refinement | Grounding DINO built from the `tao-pytorch` submodule; user-supplied checkpoint | +| `mal` | Box-to-segmentation pseudo-labels | MAL from `tao-pytorch`; PL checkpoint | +| `image_grounding` | VLM two-step expression extraction + grounding | Remote VLM via `core/llm_clients` | +| `image_referring_expression` | VLM four-step referring-expression workflow | Remote VLM | +| `video_reasoning_annotation` | Multi-step video captioning/QA workflow | Remote VLM + LLM | + +## Anatomy of One Function: `annotations` + +```text +nvidia_tao_ds/annotations/ +├── entrypoint/annotations.py # argparse shell -> shared dispatcher +├── scripts/convert.py # @hydra_runner(config_name="annotations", schema=ExperimentConfig) +├── scripts/{merge,slice,qa_to_llava_annotation}.py +├── conversion/ # The format-conversion engine +│ └── mapping.py # CONVERSION_MAPPING: {input_format: {output_format: converter_fn}} +├── merger.py # Merger ABC -> COCOMerger, LLaVAMerger, ODVGMerger +├── slicer.py # Slicer ABC -> COCO{Random,Number,Category,Filename}Slicer +└── experiment_specs/*.yaml # 11 example specs +nvidia_tao_ds/config/annotations/ +├── default_config.py # ExperimentConfig (convert) +├── merge_config.py # MergeConfig +└── slice_config.py # SliceConfig +``` + +Command flow for `annotations convert -e spec.yaml results_dir=/results`: + +1. The `annotations` console script calls + `annotations/entrypoint/annotations.py:main`. +2. `core/entrypoint/entrypoint.py::get_subtasks()` discovers `scripts/*.py` as + subtasks and injects the synthetic `default_specs` subtask. +3. `launch()` converts `-e` into Hydra `--config-path`/`--config-name` flags, + passes unknown CLI tokens through as Hydra overrides, and **spawns a fresh + `python .../scripts/convert.py` subprocess**, teeing stdout (and to + `$TAO_MICROSERVICES_TTY_LOG/$JOB_ID/microservices_log.txt` when `JOB_ID` is + set). +4. Inside the child, `@hydra_runner(..., schema=ExperimentConfig)` validates the + YAML, and `@monitor_status(...)` creates the results directory and writes + `status.json`. +5. `run_conversion(cfg)` dispatches through + `CONVERSION_MAPPING[input_format][output_format]`. +6. Back in the parent, failure is detected two ways: the subprocess exit code + **and** the last record of `results_dir/status.json` — a subtask that exits + 0 but logged `FAILURE` is still reported as failed. + +The subprocess step matters for debugging: breakpoints set in a `scripts/*.py` +are never hit when you launch via the console command. Run the script module +directly with `--config-path`/`--config-name` to debug it in-process. + +## Two Configuration Conventions + +The configuration tree has two generations of layout: + +* **Flat (older functions):** Schemas live in `config//`, anchored by + `default_config.py::ExperimentConfig`: `annotations`, `augmentation`, + `auto_label`, `image`, and `analytics` (the schema package for + `data_analytics/`). Subtasks may add sibling modules; `annotations` pairs + `ExperimentConfig` (convert) with `merge_config.py` and `slice_config.py`. +* **Per-subtask (newer functions):** One configuration module per subtask: + `config/mining/embedding/image_embeddings.py` (`ImageEmbeddingsConfig`), + `config/mining/tmm/nearest_neighbors.py` (`NearestNeighborsConfig`), + `config/rcca/gap_analysis/*.py`, and so on. + +Fields in both use the factories from `config/utils/types.py`, which attach +`description`, `valid_options`, `valid_min`/`valid_max`, and related metadata +consumed by specification generation and the FTMS/API layer. + +## Sharp Edges + +The following behaviors surprise every new developer; they are collected in one place: + +* **`data_analytics` versus `analytics`.** The package directory is + `data_analytics/`, the console script is `analytics`, and the configuration package + is `config/analytics/`. `core/utils/default_specs.py` carries an explicit + alias map to reconcile them. +* **`default_specs` does not support mining or rcca.** It only recognizes + functions with a flat `nvidia_tao_ds//entrypoint/` layout and a + `config//default_config.py`. The nested `mining/*` and `rcca/*` functions + fail with "Module ... is not supported" even though `default_specs` appears + in their subtask lists. +* **Every command requires `nvidia-smi`**, including pure-CPU ones; the shared + dispatcher unconditionally counts GPUs. The CLI cannot run on a GPU-less host. +* **Multi-GPU parsing is keyed on the literal subtask name `generate`.** + `launch(multigpu_support=['generate'], ...)` — a new GPU subtask with any + other name silently runs single-GPU and ignores spec-level + `num_gpus`/`gpu_ids`. `augmentation` uses `mpirun`, `auto_label` uses + `torchrun`; the choice is keyed on the `network=` string. +* **Not every subtask writes `status.json`.** `@monitor_status` is present on + annotations `convert`/`merge`/`slice`, `augmentation generate`, + `auto_label generate`, analytics `analyze`/`validate`, and `image validate` + — and absent on `qa_to_llava_annotation`, `kpi_analyze`, all of `mining/*`, + and all of `rcca/*`. For the latter group the dispatcher's status-based + failure check is a no-op. +* **`augmentation`'s default `config_name` is `"kitti"`**, not `generate`; its + `experiment_specs/` holds `kitti.yaml` and `coco.yaml`. Always read the + `@hydra_runner(config_name=...)` line rather than inferring from the subtask + name. +* **`vllm_captioning` is a dangling auto-label option.** It appears in the + configuration's `valid_options`, but `ExperimentConfig` has no such field and + the script raises `NotImplementedError`; selecting it fails with an + `AttributeError` from configuration validation. +* **`nvidia_tao_ds/backbone/` is vendored legacy code** (FAN/ConvNeXt/Swin/ViT; + the FAN-derived files carry an additional NVlabs Source Code License-NC + notice) imported by nothing else in the package. Do not extend it. +* **The Docker image enforces a codec policy.** The base image builds an + LGPL-only FFmpeg (VP9/mjpeg encode only; H.264 decode is hardware-only) and + OpenCV without FFmpeg, with build-time assertions that fail the image if + restricted codecs reappear. Any video-handling change must stay inside that + allow-list (`core/utils/video_utils.py` uses `libvpx-vp9`). +* **Two API stories exist.** `nvidia_tao_ds/api/app.py` is a dev-mode Flask app + reachable only via `tao_ds --run_as_service`; the release container instead + runs the tao-core microservice app. Refer to [Architecture](architecture.md). +* **No pytest configuration file exists.** There is no `conftest.py` or + `pytest.ini`; several GPU/data-heavy tests skip themselves when + `CI_PROJECT_DIR` is set (a GitLab-era guard) and need private scratch + datasets mounted. Prefer path-based test selection. diff --git a/docs/development_workflows.md b/docs/development_workflows.md index bb339bf..9057782 100644 --- a/docs/development_workflows.md +++ b/docs/development_workflows.md @@ -1,12 +1,18 @@ # Development Workflows +This guide gives concrete recipes for common source-code changes. + These recipes assume you have already run: ```sh +git submodule update --init source scripts/envsetup.sh ``` -## Build Or Install The Package +`envsetup.sh` sets `NV_TAO_DS_TOP`, defines the `tao_ds` shell function, and +installs the repository's pre-commit hooks. + +## Build or Install the Package Inside the development container: @@ -24,54 +30,81 @@ python3 setup.py develop `release/python/version.py` owns the package name, package description, version components, and package metadata used by `setup.py`. -## Change A Command Or Subtask +## Trace a Command to Code + +Use this when a task mentions a command such as `annotations convert`. + +```sh +rg -n "annotations" setup.py +sed -n '1,80p' nvidia_tao_ds/annotations/entrypoint/annotations.py +sed -n '1,120p' nvidia_tao_ds/annotations/scripts/convert.py +rg -n "config_name=" nvidia_tao_ds/annotations/scripts/ +``` + +Then inspect the specifications, schema, and tests: + +```sh +find nvidia_tao_ds/annotations/experiment_specs -maxdepth 1 -type f | sort +find nvidia_tao_ds/config/annotations -maxdepth 1 -type f | sort +find tests -name '*annotation*' -o -name '*merger*' | sort +``` + +## Change a Command or Subtask 1. Find the command in `setup.py`. 2. Open its package under `nvidia_tao_ds/`. 3. Read the package `entrypoint/` wrapper and the target file in `scripts/`. -4. Check the matching example YAML under `experiment_specs/`. +4. Check the matching example YAML under `experiment_specs/` and the + `@hydra_runner(config_name=...)` declaration. 5. Check the matching dataclass schema under `nvidia_tao_ds/config/`. 6. Update focused tests under `tests/`. 7. Run `python tools/update_readme_supported_commands.py` if command metadata, subtask files, launcher options, or image digests changed. -Standard command wrappers require `-e/--experiment_spec_file` for normal -subtasks. Mining and RCCA direct dispatchers forward Hydra arguments directly, -so mirror their existing tests before changing their CLI behavior. +When adding a subtask, remember two dispatcher behaviors keyed on names: only +subtasks named `generate` get multi-GPU handling, and only scripts decorated +with `@monitor_status` write `status.json` (which the dispatcher reads to detect +failures). -## Update Config Defaults +## Update Configuration Defaults -Flat command configs live under `nvidia_tao_ds/config//`. The analytics -command is implemented in `nvidia_tao_ds/data_analytics/` but its config package -is `nvidia_tao_ds/config/analytics/`. +Flat command configurations live under `nvidia_tao_ds/config//`. The analytics +command is implemented in `nvidia_tao_ds/data_analytics/` but its configuration +package is `nvidia_tao_ds/config/analytics/`. -Nested domains use nested config packages: +Nested domains use nested, per-subtask configuration packages: -| Domain | Config Path | +| Domain | Config path | | :--- | :--- | | TMM mining | `nvidia_tao_ds/config/mining/tmm/` | | Embedding mining | `nvidia_tao_ds/config/mining/embedding/` | | RCCA gap analysis | `nvidia_tao_ds/config/rcca/gap_analysis/` | -After config changes, run focused config and command tests: +Declare new fields with the factories from +`nvidia_tao_ds/config/utils/types.py` (`STR_FIELD`, `INT_FIELD`, ...), including +a `description`; the metadata feeds the API schema layer and default-specification +generation. + +After configuration changes, run focused configuration tests: ```sh pytest tests/test_config_modules.py -q -pytest tests/mining/test_mining_entrypoints.py -q ``` -## Update The API Service +## Update the API Service -API routes live in `nvidia_tao_ds/api/app.py`. The service relies on installed -console scripts and `nvidia_tao_core.api_utils.module_utils` for action -discovery, so API-visible command changes may require coordinated changes in -`tao-core/`. +Dev-mode API routes live in `nvidia_tao_ds/api/app.py`. The service relies on +installed console scripts and `nvidia_tao_core.api_utils.module_utils` for +action discovery, so API-visible command changes may require coordinated +changes in `tao-core/`. -For schema behavior, check the relevant dataclass config and any tests that +For schema behavior, check the relevant dataclass configuration and any tests that exercise API or default-spec imports. For endpoint inventory, compare -`nvidia_tao_ds/api/app.py` and `nvidia_tao_ds/api/openapi.json`. +`nvidia_tao_ds/api/app.py` and `nvidia_tao_ds/api/openapi.json`. The release +container runs the tao-core microservice app, not this Flask app; refer to +[Architecture](architecture.md). -## Update The Base Development Image +## Update the Base Development Image Use `docker/build.sh` for the base image: @@ -84,30 +117,25 @@ cd "$NV_TAO_DS_TOP/docker" Single-platform builds can load locally. Multi-platform builds require `--push` because Docker buildx cannot load multiple architectures into the local Docker -daemon at once. After pushing, update `docker/manifest.json` with the digest -printed by the build script. +daemon at once. -**Update every digest reference, not just `docker/manifest.json`.** The base-image -digest is hardcoded in several files. A base-image rebuild must update all of them for -the target architecture (`x86` and/or `arm`), or CI, Jenkins, and the release build will -keep pulling the stale image: +**Update every digest reference, not just `docker/manifest.json`.** After +pushing, update the new digest in both places that pin it: | Location | What it drives | | :--- | :--- | -| `docker/manifest.json` | `tao_ds` launcher + `ci/utils.py` (static/functional tests); per-arch `x86`/`arm` digests — the source of truth | -| `.gitlab-ci.yml` | the `image:` the GitLab static-test jobs run in | -| `Jenkinsfile.dev`, `Jenkinsfile.develop`, `Jenkinsfile.nightly` | the `base-image` container the Jenkins jobs run in | -| `Jenkinsfile.release` | `BUILD_X86` / `BUILD_ARM` release base images | -| `release/docker/Dockerfile.release` | `FROM` base digests for the release image | +| `docker/manifest.json` | The `tao_ds` launcher; this is the source of truth for the per-architecture `x86` and `arm` digests | +| `release/docker/Dockerfile.release` | Default `X86_DIGEST`/`ARM64_DIGEST` build args for the release image `FROM` lines | -Find them all with `grep -rl .` and replace per architecture (the `x86` -and `arm` digests are distinct strings, so a per-digest `sed` is safe). The -`check_base_image_digests_consistent` static test (`ci/run_static_tests.py`) fails the build -if any of these files references a digest not in `docker/manifest.json`, so a partial bump -is caught in CI. The README intentionally does **not** carry the digests (they are internal -`nvstaging` references, not useful in a public repo) — do not re-add them. +Find any stragglers with `grep -rl .` and replace per architecture +(the `x86` and `arm` digests are distinct strings, so a per-digest `sed` is +safe). The README intentionally does **not** carry the digests, because they +are internal `nvstaging` references that are not useful in a public +repository; do not re-add them. Dependency changes to requirements or Docker +files also trip the `dependency-guard` pre-commit hook, which requires +explicit acknowledgment. -## Update The Release Image +## Update the Release Image Use `release/docker/deploy.sh` for the release image: @@ -120,14 +148,19 @@ The script can build the source wheel through `tao_ds -- make build`, build `release/docker/Dockerfile.release`, push the release image, and clean wheels when `--wheel` is used. +Any change touching video or codecs must respect the FFmpeg codec allow-list +baked into `docker/Dockerfile` (an LGPL-only build with VP9 and mjpeg encode and no libx264, hevc, or aac). The image build asserts this and fails if restricted codecs +reappear. + ## Keep README Generated Content Fresh `README.md` contains a generated section for launcher options, console scripts, -subtasks, and base-image digests. Update or check it with: +subtasks, and the base-image manifest pointer. Update or check it with: ```sh python tools/update_readme_supported_commands.py python tools/update_readme_supported_commands.py --check ``` -The local pre-commit hook and GitLab `static_tests` job run the `--check` mode. +The local pre-commit hook and the GitHub Actions `static-tests` workflow run +the `--check` mode. diff --git a/docs/index.md b/docs/index.md index da0e616..0b3bcb1 100644 --- a/docs/index.md +++ b/docs/index.md @@ -6,10 +6,16 @@ container power users working from the TAO Data Services source tree. The root `README.md` is intentionally concise. Start here when you need the repository mental model, command flow, configuration flow, or validation map. +This repository is the source of the TAO data-services container +(`nvcr.io/nvidia/tao/tao-toolkit:-dataservices`). For what the product +is, how customers run it, and the terminology the rest of these documents use, +read the opening sections of [Architecture](architecture.md). + ## Start Paths -| Role | Start With | Then Read | +| Role | Start with | Then read | | :--- | :--- | :--- | +| New developer picking up the repository | [Codebase tour](codebase_tour.md) | [Architecture](architecture.md) | | Coding agent or new maintainer | [Agent onboarding](agent_onboarding.md) | [Architecture](architecture.md), [Testing and debugging](testing_and_debugging.md) | | Feature developer | [Development workflows](development_workflows.md) | [New data-service command](new_data_service_command.md) | | Container power user | [Container power users](container_power_users.md) | [Architecture](architecture.md) | @@ -19,27 +25,28 @@ repository mental model, command flow, configuration flow, or validation map. | Document | Purpose | | :--- | :--- | -| [agent_onboarding.md](agent_onboarding.md) | First-pass audit commands, worktree safety, and source-of-truth files. | -| [architecture.md](architecture.md) | Runtime dispatch, Hydra config, API service, package layout, and extension points. | -| [development_workflows.md](development_workflows.md) | Recipes for common source, config, Docker, release, and README changes. | -| [testing_and_debugging.md](testing_and_debugging.md) | CI static checks, targeted pytest commands, GPU-sensitive paths, and failure triage. | -| [container_power_users.md](container_power_users.md) | `tao_ds`, mounts, GPUs, base-image digests, service mode, and direct Docker equivalents. | -| [new_data_service_command.md](new_data_service_command.md) | Source-backed guide for adding or extending commands and subtasks. | +| [Codebase tour](codebase_tour.md) | Annotated repository tree, module map, function inventory, and the sharp edges new developers hit. | +| [Agent onboarding](agent_onboarding.md) | First-pass audit commands, worktree safety, and source-of-truth files. | +| [Architecture](architecture.md) | Runtime dispatch, Hydra config, models and weights, API service, and extension points. | +| [Development workflows](development_workflows.md) | Recipes for common source, configuration, Docker, release, and README changes. | +| [Testing and debugging](testing_and_debugging.md) | CI static checks, targeted pytest commands, GPU-sensitive paths, and failure triage. | +| [Container power users](container_power_users.md) | `tao_ds`, mounts, GPUs, base-image digests, service mode, and direct Docker equivalents. | +| [New data-service command](new_data_service_command.md) | Source-backed guide for adding or extending commands and subtasks. | ## Repository Anchors -| Path | What To Look For | +| Path | What to look for | | :--- | :--- | -| `setup.py` | Package metadata and in-container console script entrypoints. | -| `scripts/envsetup.sh` | `NV_TAO_DS_TOP` setup and the host-side `tao_ds` shell function. | +| `setup.py` | Package metadata and in-container console script entry points. | +| `scripts/envsetup.sh` | `NV_TAO_DS_TOP` setup, the host-side `tao_ds` shell function, and git-hook installation. | | `runner/tao_ds.py` | Docker launcher, GPU selection, mount handling, service mode, and manifest lookup. | | `nvidia_tao_ds/core/entrypoint/entrypoint.py` | Shared subtask discovery, experiment spec handling, GPU override handling, and subprocess launch. | | `nvidia_tao_ds/core/hydra/hydra_runner.py` | Local Hydra wrapper used by script modules. | | `nvidia_tao_ds/config/` | Dataclass-backed schemas and default-spec sources. | | `nvidia_tao_ds/*/experiment_specs/` | Example YAML specs used by script subtasks. | -| `nvidia_tao_ds/api/app.py` | Flask API routes, job queue handoff, schema validation, and OpenAPI endpoints. | +| `nvidia_tao_ds/api/app.py` | Dev-mode Flask API routes, job queue handoff, schema validation, and OpenAPI endpoints. | | `docker/manifest.json` | Immutable base-image registry, repository, and architecture-specific digests. | -| `.gitlab-ci.yml` and `ci/` | Merge-request checks and static-test helpers. | +| `.pre-commit-config.yaml` and `.github/workflows/` | Pull-request checks: lint, license headers, DCO, README drift, and secret scan. | ## Command Layers @@ -59,7 +66,13 @@ launcher, or image-manifest metadata changes. ## Diagrams +Architecture and workflow diagrams are checked in as SVG files under +`docs/assets/` and embedded with normal Markdown image links. The SVG files are +the canonical editable sources. + | Diagram | Source | | :--- | :--- | | Runtime dispatch flow | [assets/runtime_flow.svg](assets/runtime_flow.svg) | +| Configuration flow | [assets/config_flow.svg](assets/config_flow.svg) | +| Package module map | [assets/module_map.svg](assets/module_map.svg) | | Container launch and build flow | [assets/container_flow.svg](assets/container_flow.svg) | diff --git a/docs/testing_and_debugging.md b/docs/testing_and_debugging.md index bc5bec1..7ab1f61 100644 --- a/docs/testing_and_debugging.md +++ b/docs/testing_and_debugging.md @@ -1,77 +1,147 @@ -# Testing And Debugging +# Testing and Debugging This repository mixes static checks, unit tests, command dispatch tests, -container checks, and GPU/data-sensitive workflows. Pick the smallest test that +container checks, and GPU- and data-sensitive workflows. Pick the smallest test that covers your change. -## GitLab Static Tests +## Static Checks -`.gitlab-ci.yml` runs the `static_tests` job in the TAO Data Services base -image. The job now checks generated README drift before running -`ci/run_static_tests.py`. +CI runs static checks on pull requests through GitHub Actions +(`.github/workflows/static-tests.yml`), which executes the repository's pre-commit +hooks against the changed files: -`ci/run_static_tests.py` runs: +* SPDX license headers (`.github/hooks/check_license_header.py`) +* `pylint` (`.pylintrc`), `pydocstyle`, and `flake8`, scoped to + `nvidia_tao_ds/` +* The generated README drift check + (`tools/update_readme_supported_commands.py --check`) -| Tool | Scope | -| :--- | :--- | -| `pylint --rcfile .pylintrc` | Modules listed in `ci/utils.py` `TEST_MODULES`. | -| `pydocstyle --ignore=D4,D200,D203,D205,D210,D212,D213,D301,D400,D401` | Same module list. | -| `flake8 --ignore=E24,W504,E501` | Same module list. | +CI skips the `trufflehog` and `dependency-guard` hooks (secret scanning runs +in the separate `secret-scan.yml` workflow); both still run in the local +pre-commit hook, and `dependency-guard` requires explicit acknowledgment for +requirements and Docker file changes. + +Reproduce locally: + +```sh +pip install pre-commit +pre-commit install +pre-commit run --from-ref origin/main --to-ref HEAD +``` -`ci/utils.py` resolves the Docker image from `docker/manifest.json` for local -static runs. In CI it runs directly in the job image. +Separate workflows enforce DCO sign-off on every commit (`dco.yml`), secret +scanning (`secret-scan.yml`), and PR title format (`pr-title.yml`). +Functional (GPU) tests run through the externally triggered `blossom-ci.yml` +workflow, not on every push. -## Docs-Only Validation +## Documentation-Only Validation + +For documentation-only changes, run these checks: ```sh python tools/update_readme_supported_commands.py --check python -m py_compile tools/update_readme_supported_commands.py -git diff --check -- README.md docs/*.md docs/assets/*.svg tools/*.py .pre-commit-config.yaml .gitlab-ci.yml +git diff --check -- README.md docs/*.md docs/assets/*.svg tools/*.py .pre-commit-config.yaml rg -n "TBD|PLACEHOLDER|example\\.com" README.md docs ``` GPU, Docker build, private checkpoint, NGC, and full dataset tests are outside the normal blast radius for documentation-only changes. +## Test Environment Setup + +The base development image has pytest and the runtime dependencies +(`docker/requirements-pip.txt`), but **not** the TAO packages themselves. Set +up once per container: + +```sh +# On the host +git submodule update --init # tao-core/ and tao-pytorch/ are empty otherwise +source scripts/envsetup.sh +tao_ds --gpus all -- bash + +# Inside the container (repository is mounted at /workspace) +pip install /workspace/tao-core/. # provides nvidia_tao_core (status callbacks, api_utils) +export PYTHONPATH=/workspace/tao-pytorch:$PYTHONPATH # for tests that import nvidia_tao_pytorch +pytest tests/test_config_modules.py -q # smoke check +``` + +Notes: + +* `nvidia_tao_core` is not in `docker/requirements-pip.txt` (only its + transitive dependencies are), so tests fail with import errors until you + install the submodule. This is the step the README historically missed. +* `nvidia_tao_ds` itself is importable because the launcher sets + `PYTHONPATH=/workspace`; run `python setup.py develop` instead if you need + the console commands. +* Tests that exercise Grounding DINO or TAO CLIP (`test_text2box.py`, + `tests/mining/test_image_embeddings.py`) import `nvidia_tao_pytorch` from + the submodule; the mining tests fall back to stubs when it is absent. +* GPU- and data-heavy tests additionally need the private scratch datasets + mounted (paths under `/media/scratch_metropolis2/tao_ci/...`). + ## Targeted Pytest Map -| Change Area | Suggested Tests | +| Change area | Suggested tests | | :--- | :--- | | Config package moves or default spec support | `pytest tests/test_config_modules.py -q` | -| Mining dispatchers | `pytest tests/mining/test_mining_entrypoints.py -q` | -| Embedding logic | `pytest tests/mining/test_image_embeddings.py -q` | -| Nearest-neighbor mining | `pytest tests/mining/test_nearest_neighbors.py -q` | +| Shared launcher / exit-code behavior | `pytest tests/test_entrypoint_exit_code.py -q` | +| Embedding logic | `pytest tests/mining/test_image_embeddings.py tests/mining/test_text_embeddings.py -q` | +| Nearest-neighbor mining | `pytest tests/mining/test_nearest_neighbors.py tests/mining/test_unique_neighbor_matching.py -q` | | Annotation merge/slice | `pytest tests/test_merger_slicer.py -q` | -| COCO/KITTI conversion | `pytest tests/test_coco_kitti_conversion.py tests/test_coco_odvg_conversion.py -q` | -| AICity conversion | `pytest tests/test_aicity_ovpkl_conversion.py -q` | +| COCO, KITTI, and ODVG conversion | `pytest tests/test_coco_kitti_conversion.py tests/test_coco_odvg_conversion.py -q` | +| AICity / PAS conversions | `pytest tests/test_aicity_ovpkl_conversion.py tests/test_nvidia_paidf_pas_to_tao_clip_conversion.py -q` | | QA to LLaVA conversion | `pytest tests/test_qa_to_llava_annotation.py tests/test_llava_merger.py -q` | | Analytics | `pytest tests/test_data_analytics.py -q` | -| Auto-label prompt and parsing logic | `pytest tests/autolabel -q` | -| Logging changes | `pytest tests/test_dual_logging.py tests/test_baselogger_recursion.py -q` | +| Auto-label VLM workflows | `pytest tests/autolabel -q` | +| Auto-label Grounding DINO | `pytest tests/test_text2box.py -q` | +| Gap analysis | `pytest tests/test_od_gap_analysis.py tests/test_vcn_aoi.py tests/test_vlm_bcq.py -q` | +| Logging changes | `pytest tests/test_dual_logging.py -q` | +| Dataset loading helpers | `pytest tests/test_dataset_loading.py -q` | + +There is no `conftest.py` or pytest configuration file. Two skip conventions +matter: + +* GPU- and data-heavy tests (`test_data_analytics.py`, `test_augment.py`) skip + themselves when `CI_PROJECT_DIR` is set, a GitLab-era guard that GitHub + Actions does not set. They need the private scratch datasets mounted. +* Some analytics tests skip unless `WANDB_API_KEY` is set. -`ci/run_functional_tests.py` runs `pytest tests -v --color=yes`. Use it when a -change spans multiple domains and the environment has the needed dependencies. +A few tests stub or skip `nvidia_tao_core` imports; initialize the submodules +before running the full suite. ## Common Failures -| Symptom | Likely Cause | Where To Check | +| Symptom | Likely cause | Where to check | | :--- | :--- | :--- | -| `Experiment spec file was not found` | Standard entrypoint was called without a valid `-e` path. | `nvidia_tao_ds/core/entrypoint/entrypoint.py` | -| Hydra cannot find a config | The script `config_name` does not match the YAML name or config path. | The target `scripts/*.py` `@hydra_runner(...)` | -| `nvidia-smi` assertion failure | Requested `num_gpus` exceeds visible GPUs. | `launch()` GPU override logic and spec `gpu_ids` | -| Docker pull or inspect fails | Local tag is missing or `docker/manifest.json` digest is stale/inaccessible. | `runner/tao_ds.py`, `docker/manifest.json` | -| Default spec generation rejects a module | Config package shape is not supported by `default_specs.py`. | `nvidia_tao_ds/core/utils/default_specs.py` | -| API action missing | Installed console scripts or `tao-core` module mappings do not expose it. | `setup.py`, `tao-core/nvidia_tao_core/api_utils/module_utils.py` | -| W&B tests skip | `WANDB_API_KEY` is not set. | `tests/test_data_analytics.py` | +| `Experiment spec file was not found` | The standard entry point ran without a valid `-e` path. | `nvidia_tao_ds/core/entrypoint/entrypoint.py` | +| Hydra cannot find a config | The script `config_name` does not match the YAML name or config path. | The target `scripts/*.py` `@hydra_runner(...)`; for example, `augmentation generate` defaults to `kitti.yaml` | +| `nvidia-smi` assertion failure | Requested `num_gpus` exceeds visible GPUs, or the host has no GPU (the launcher calls `nvidia-smi` unconditionally, even for CPU subtasks). | `launch()` GPU logic in the shared entrypoint | +| Command reports FAIL despite exit 0 | The launcher also reads the last record of `results_dir/status.json`. | `_status_reports_failure` in the shared entrypoint | +| New GPU subtask runs on one GPU only | Multi-GPU parsing is keyed on the subtask name `generate`. | `launch(multigpu_support=...)` | +| Import errors for `nvidia_tao_core` / `nvidia_tao_pytorch` | Uninitialized submodules. | `git submodule update --init`; in containers `envsetup.sh` sets `PYTHONPATH` | +| `default_specs` rejects a module | Only flat functions are supported; mining and RCCA are not. | `nvidia_tao_ds/core/utils/default_specs.py` | +| Docker pull or inspect fails | Local tag missing or `docker/manifest.json` digest stale/inaccessible. | `runner/tao_ds.py`, `docker/manifest.json` | +| Video write fails or codec missing | The base image enforces an LGPL codec allow-list (VP9/mjpeg encode only). | `docker/Dockerfile`, `core/utils/video_utils.py` | +| API action missing | Installed console scripts or tao-core module mappings do not expose it. | `setup.py`, tao-core `api_utils.module_utils` | +| Weights and Biases tests skip | `WANDB_API_KEY` is not set. | `tests/test_data_analytics.py` | ## Debugging Runtime Commands -Print the Docker command without hiding what will run: +Run a command through the launcher: + +```sh +tao_ds --gpus all -- annotations convert -e nvidia_tao_ds/annotations/experiment_specs/annotations.yaml +``` + +The shared entrypoint launches each subtask as a fresh Python subprocess, so +breakpoints set in a `scripts/*.py` are never hit through the console command. +Debug in-process by running the script directly: ```sh -tao_ds --gpus all --volume "$PWD:/workspace" -- annotations convert -e nvidia_tao_ds/annotations/experiment_specs/annotations.yaml +python nvidia_tao_ds//scripts/.py \ + --config-path /abs/path/to/spec/dir --config-name ``` -For standard entrypoints, command-line Hydra overrides follow the experiment -spec. For mining and RCCA direct dispatchers, pass Hydra args directly to the -subtask script through the dispatcher. +Command-line Hydra overrides follow the experiment specification, for example, +`results_dir=/results data.input_format=COCO`. diff --git a/tools/update_readme_supported_commands.py b/tools/update_readme_supported_commands.py index c04c888..db7d5b6 100644 --- a/tools/update_readme_supported_commands.py +++ b/tools/update_readme_supported_commands.py @@ -184,11 +184,11 @@ def render_generated_section(repo_root: Path) -> str: "", "### Base Image Source", "", - f"`runner/tao_ds.py` and `ci/utils.py` resolve the immutable base image", + f"`runner/tao_ds.py` resolves the immutable base image", f"(`{repository}`) from `docker/manifest.json`, choosing the architecture-specific", "digest for the host. The pinned digests are intentionally not duplicated here — they", - "live in `docker/manifest.json` (and the CI / Jenkins / release files), and a static CI", - "check (`ci/run_static_tests.py`) verifies those digest references stay in sync.", + "live in `docker/manifest.json` and `release/docker/Dockerfile.release`; update both", + "together when the base image is rebuilt.", ] ) lines.append(END)