[TAO-2382][Feature] Add dinov3 evaluate action (shared core/evaluation suite) - #81
Open
vpraveen-nv wants to merge 3 commits into
Open
[TAO-2382][Feature] Add dinov3 evaluate action (shared core/evaluation suite)#81vpraveen-nv wants to merge 3 commits into
vpraveen-nv wants to merge 3 commits into
Conversation
|
For security reasons, CI does not run automatically on NVIDIA's runners — it must be triggered per commit.
Tip Should this fix also ship in a release? Add a Important Before merging: every file needs a license header and every commit must be DCO signed-off — see the 📖 Contribution guide for details. |
vpraveen-nv
force-pushed
the
feature/core-eval-tests
branch
from
August 6, 2026 06:28
f40bcd3 to
c1185d5
Compare
vpraveen-nv
force-pushed
the
feature/dinov3-eval
branch
from
August 6, 2026 06:28
ffd7718 to
949c9ff
Compare
vpraveen-nv
force-pushed
the
feature/core-eval-tests
branch
from
August 6, 2026 06:40
c1185d5 to
b6bf6aa
Compare
vpraveen-nv
force-pushed
the
feature/dinov3-eval
branch
from
August 6, 2026 06:40
949c9ff to
f17fd64
Compare
…esent but None getattr(cfg, 'cache_tag', ctx.network) never falls back when the schema defines cache_tag: Optional[str] = None — the attribute exists with value None, so embedding_cache_path received model_tag=None and crashed with AttributeError whenever evaluate.cache_dir was set. Found by the dinov3 evaluate smoke on real ImageNet. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vpraveen <vpraveen@nvidia.com>
…n suite) Mirror the nvdinov2 evaluate action for the DINOv3 SSL family (Epic G, TAO-2381): - Register 'dinov3' in core/evaluation ADAPTER_REGISTRY reusing DinoV2Adapter (DinoV3PlModel(DinoV2PlModel) exposes the same teacher.backbone token dict; embed_dim/patch_size are passed from the pl_model, no subclass needed). - ssl/dinov3/scripts/evaluate.py: load checkpoint via restore_pretrained_weights (stripped teacher/student .pth or timm-format .pth/.safetensors) -> adapter -> EvalContext -> build_enabled_evaluators -> results.json. Auto-registers as 'dinov3 evaluate'. Keeps xformers attention under bf16 autocast (amp=True), same arrangement as nvdinov2. - config/dinov3: DINOv3EvaluateExpConfig(EvaluateConfig, EvalSuiteConfig) + evaluate block; experiment_specs/evaluate_spec.yaml (KNN-only, ImageNet normalization matching the DINOv3 train/inference pipeline). - Tests: dinov3 adapter contract + evaluate-block config defaults; README command table regenerated. Validated on a4u8g-0146 (single A100): published DINOv3 ViT-B/16 timm weights remap 162/162 tensors; ImageNet KNN via 'dinov3 evaluate' 48.36% top-1 with a 2k-image smoke index and 78.79% with a 100k index (full-index run in flight). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vpraveen <vpraveen@nvidia.com>
…n dataset A truncated JPEG in the ImageNet-1k train split raised OSError in a DataLoader worker and killed the full-index KNN extraction hours in. Set PIL ImageFile.LOAD_TRUNCATED_IMAGES = True — the same policy as c-radiov4 eval_cls.py and the mae / classification_pyt / radio dataloaders. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: vpraveen <vpraveen@nvidia.com>
vpraveen-nv
force-pushed
the
feature/core-eval-tests
branch
from
August 7, 2026 16:39
b6bf6aa to
be6195e
Compare
vpraveen-nv
force-pushed
the
feature/dinov3-eval
branch
from
August 7, 2026 16:40
f17fd64 to
fdaa0c9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Mirrors the nvdinov2 evaluate action for the DINOv3 SSL family (Epic G), plus two small core/evaluation robustness fixes found during validation:
dinov3 evaluate: registersdinov3inADAPTER_REGISTRY(reusesDinoV2Adapter;DinoV3PlModel(DinoV2PlModel)exposes the sameteacher.backbonetoken dict); addsssl/dinov3/scripts/evaluate.py(checkpoint viarestore_pretrained_weights, accepts stripped teacher/student.pthor timm-format.pth/.safetensors);DINOv3EvaluateExpConfig(EvaluateConfig, EvalSuiteConfig)+evaluate_spec.yaml(KNN-only, ImageNet normalization); README table regen.cache_tagfallback when the config field is present butNone(crashed any run withevaluate.cache_dirset).ImageFile.LOAD_TRUNCATED_IMAGES = True, same policy as the mae/classification_pyt/radio dataloaders).Why are the changes needed?
Completes embedding-quality evaluation coverage for the DINOv3 family through the shared
core/evaluationsuite, and fixes two crashes hit during full-scale validation (acache_tagcrash withcache_dirset, and a truncated ImageNet JPEG killing full-split extraction).Related issues
JIRA: TAO-2381 / TAO-2382 / TAO-2383 / TAO-2386. N/A for GitHub issues.
Does this PR introduce any user-facing change?
Yes — new
dinov3 evaluateaction. Also: runs withevaluate.cache_dirset no longer crash, and truncated JPEGs no longer abort evaluation dataset extraction.How was this patch tested?
Validation on a4u8g-0146 (single A100):
tests/core/evaluation+ dinov3 config)dinov3 evaluateImageNet KNN, full 1.28M index @224pylint 10.00/10 on touched files.
Was this patch authored or co-authored using generative AI tooling?
Yes — portions were co-authored with an AI coding assistant.
Release note
Checklist
git commit -s) — seeCONTRIBUTING.mdso the package still works without them (reviewers: please verify this)
my commit author name and email become permanently public once merged
Notes for reviewers
feature/core-eval-tests); retarget tomainafter the stack merges. Diff vs stack top: 9 files, +239/−3.use_faiss=False— faiss-cpu 1.13.2 crashes inIndexFlatIP.searchat scale (TAO-2387).main(config/dinov3/default_config.py): kept main's newerDINOv3ExportExpConfigand added theevaluatefield — worth a second look.Migrated from GitLab MR nvidia-tao-toolkit/tao-pytorch!645, rebased onto GitHub
main.