Skip to content

[TAO-2382][Feature] Add dinov3 evaluate action (shared core/evaluation suite) - #81

Open
vpraveen-nv wants to merge 3 commits into
feature/core-eval-testsfrom
feature/dinov3-eval
Open

[TAO-2382][Feature] Add dinov3 evaluate action (shared core/evaluation suite)#81
vpraveen-nv wants to merge 3 commits into
feature/core-eval-testsfrom
feature/dinov3-eval

Conversation

@vpraveen-nv

@vpraveen-nv vpraveen-nv commented Aug 5, 2026

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Mirrors the nvdinov2 evaluate action for the DINOv3 SSL family (Epic G), plus two small core/evaluation robustness fixes found during validation:

  • TAO-2382dinov3 evaluate: registers dinov3 in ADAPTER_REGISTRY (reuses DinoV2Adapter; DinoV3PlModel(DinoV2PlModel) exposes the same teacher.backbone token dict); adds ssl/dinov3/scripts/evaluate.py (checkpoint via restore_pretrained_weights, accepts stripped teacher/student .pth or timm-format .pth/.safetensors); DINOv3EvaluateExpConfig(EvaluateConfig, EvalSuiteConfig) + evaluate_spec.yaml (KNN-only, ImageNet normalization); README table regen.
  • TAO-2386 — fixes cache_tag fallback when the config field is present but None (crashed any run with evaluate.cache_dir set).
  • TAO-2383 — tolerates truncated JPEGs in the eval classification dataset (ImageFile.LOAD_TRUNCATED_IMAGES = True, same policy as the mae/classification_pyt/radio dataloaders).

Why are the changes needed?

Completes embedding-quality evaluation coverage for the DINOv3 family through the shared core/evaluation suite, and fixes two crashes hit during full-scale validation (a cache_tag crash with cache_dir set, and a truncated ImageNet JPEG killing full-split extraction).

Related issues

JIRA: TAO-2381 / TAO-2382 / TAO-2383 / TAO-2386. N/A for GitHub issues.

Does this PR introduce any user-facing change?

Yes — new dinov3 evaluate action. Also: runs with evaluate.cache_dir set no longer crash, and truncated JPEGs no longer abort evaluation dataset extraction.

How was this patch tested?

Validation on a4u8g-0146 (single A100):

Check Result
Unit tests (tests/core/evaluation + dinov3 config) 39 passed
timm feature-parity (published ViT-B/16, 256 + 768 px) pass, 162/162 tensors remapped
dinov3 evaluate ImageNet KNN, full 1.28M index @224 knn_top1 = 82.99% (published DINOv3 ViT-B/16 k-NN ballpark)
C-RADIOv4-H parity through the same core KNN path @512 86.42% vs paper 86.59 (Δ −0.17)

pylint 10.00/10 on touched files.

Was this patch authored or co-authored using generative AI tooling?

Yes — portions were co-authored with an AI coding assistant.

Release note

Added a dinov3 evaluate action for embedding-quality (KNN) evaluation of DINOv3 checkpoints.

Checklist

  • My commits are signed off per the DCO (git commit -s) — see CONTRIBUTING.md
  • I have read the contributing guidelines
  • The code follows the project's style, and lint and format checks pass locally
  • I added or updated tests covering this change
  • All tests pass locally
  • I added or updated documentation (README, docstrings, docs pages, examples)
  • I updated the version, changelog, or migration notes if this change requires it
  • If this touches components that are optional to install, the imports are guarded
    so the package still works without them (reviewers: please verify this)
  • No secrets, credentials, proprietary data, or customer data are included in this PR
  • I understand this contribution is licensed under the repository's license, and that
    my commit author name and email become permanently public once merged

Notes for reviewers

  • Stacked on [TAO-2175][Feature] core/evaluation unit tests (26 tests) #80 (feature/core-eval-tests); retarget to main after the stack merges. Diff vs stack top: 9 files, +239/−3.
  • KNN validation ran with use_faiss=False — faiss-cpu 1.13.2 crashes in IndexFlatIP.search at scale (TAO-2387).
  • ADE20K seg validation is TAO-2384 (blocked: dataset not on the devbox).
  • One rebase conflict was resolved when moving to GitHub main (config/dinov3/default_config.py): kept main's newer DINOv3ExportExpConfig and added the evaluate field — worth a second look.

Migrated from GitLab MR nvidia-tao-toolkit/tao-pytorch!645, rebased onto GitHub main.

@vpraveen-nv
vpraveen-nv requested a review from a team as a code owner August 5, 2026 23:04
@tao-pr-bot

tao-pr-bot Bot commented Aug 5, 2026

Copy link
Copy Markdown

For security reasons, CI does not run automatically on NVIDIA's runners — it must be triggered per commit.

  • Comment /build to run CI on your latest commit, and make sure blossom-ci is green before merging.
  • Re-run /build after each new push — CI is pinned to the head commit, so a stale run won't count.

Tip

Should this fix also ship in a release? Add a release/X.Y.Z label (matching the target release branch) and tao-cherry-pick-bot will backport it after merge.

Important

Before merging: every file needs a license header and every commit must be DCO signed-off — see the 📖 Contribution guide for details.

vpraveen-nv and others added 3 commits August 7, 2026 16:39
…esent but None

getattr(cfg, 'cache_tag', ctx.network) never falls back when the schema
defines cache_tag: Optional[str] = None — the attribute exists with value
None, so embedding_cache_path received model_tag=None and crashed with
AttributeError whenever evaluate.cache_dir was set. Found by the dinov3
evaluate smoke on real ImageNet.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: vpraveen <vpraveen@nvidia.com>
…n suite)

Mirror the nvdinov2 evaluate action for the DINOv3 SSL family (Epic G,
TAO-2381):

- Register 'dinov3' in core/evaluation ADAPTER_REGISTRY reusing DinoV2Adapter
  (DinoV3PlModel(DinoV2PlModel) exposes the same teacher.backbone token dict;
  embed_dim/patch_size are passed from the pl_model, no subclass needed).
- ssl/dinov3/scripts/evaluate.py: load checkpoint via restore_pretrained_weights
  (stripped teacher/student .pth or timm-format .pth/.safetensors) -> adapter ->
  EvalContext -> build_enabled_evaluators -> results.json. Auto-registers as
  'dinov3 evaluate'. Keeps xformers attention under bf16 autocast (amp=True),
  same arrangement as nvdinov2.
- config/dinov3: DINOv3EvaluateExpConfig(EvaluateConfig, EvalSuiteConfig) +
  evaluate block; experiment_specs/evaluate_spec.yaml (KNN-only, ImageNet
  normalization matching the DINOv3 train/inference pipeline).
- Tests: dinov3 adapter contract + evaluate-block config defaults; README
  command table regenerated.

Validated on a4u8g-0146 (single A100): published DINOv3 ViT-B/16 timm weights
remap 162/162 tensors; ImageNet KNN via 'dinov3 evaluate' 48.36% top-1 with a
2k-image smoke index and 78.79% with a 100k index (full-index run in flight).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: vpraveen <vpraveen@nvidia.com>
…n dataset

A truncated JPEG in the ImageNet-1k train split raised OSError in a
DataLoader worker and killed the full-index KNN extraction hours in. Set
PIL ImageFile.LOAD_TRUNCATED_IMAGES = True — the same policy as c-radiov4
eval_cls.py and the mae / classification_pyt / radio dataloaders.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Signed-off-by: vpraveen <vpraveen@nvidia.com>
@vpraveen-nv
vpraveen-nv force-pushed the feature/core-eval-tests branch from b6bf6aa to be6195e Compare August 7, 2026 16:39
@vpraveen-nv
vpraveen-nv force-pushed the feature/dinov3-eval branch from f17fd64 to fdaa0c9 Compare August 7, 2026 16:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant