[TAO-2149][Feature] SSL evaluate actions for NV-DINOv2 and MAE (embedding suite) - #79
Open
vpraveen-nv wants to merge 4 commits into
Open
[TAO-2149][Feature] SSL evaluate actions for NV-DINOv2 and MAE (embedding suite)#79vpraveen-nv wants to merge 4 commits into
vpraveen-nv wants to merge 4 commits into
Conversation
|
For security reasons, CI does not run automatically on NVIDIA's runners — it must be triggered per commit.
Tip Should this fix also ship in a release? Add a Important Before merging: every file needs a license header and every commit must be DCO signed-off — see the 📖 Contribution guide for details. |
This was referenced Aug 5, 2026
vpraveen-nv
force-pushed
the
feature/core-eval-d
branch
2 times, most recently
from
August 7, 2026 16:39
048c15f to
85e4a2f
Compare
…t shim Dependency inversion: radio/distillation/knn_classification.py becomes a thin re-export shim onto nvidia_tao_pytorch.core.evaluation.knn. The KNN vote math (distributed_topk, knn_top1_accuracy, _get_vote_cls) is now owned by the shared suite and merely relocated here, so existing imports keep working and the distillation KNN-top1 numbers are unchanged (validate.py + the distiller resolve to the same core objects). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Varun Praveen <vpraveen@nvidia.com>
Nested single quotes inside f-string replacement fields are only valid from Python 3.12 (PEP 701). CI lints on Python 3.11, so pylint/flake8 report a syntax error for any change set that touches or imports this module. Flip the outer quotes to double quotes; no behavior change. Signed-off-by: Varun Praveen <vpraveen@nvidia.com>
…ding suite)
Front-ends onto the shared core/evaluation suite. The evaluate action loads a
checkpoint, wraps the backbone in the (summary, features) adapter, runs the enabled
evaluators (build_enabled_evaluators), and writes results.json.
- config/ssl_evaluation is mixed into each network's EvaluateConfig:
nvdinov2 gets NVDINOv2EvaluateExpConfig(EvaluateConfig, EvalSuiteConfig);
mae gets MAEEvaluateExpConfig(EvaluateConfig, EvalSuiteConfig).
- nvdinov2/scripts/evaluate.py: new hydra @hydra_runner/@monitor_status action
(auto-registered subtask); disables the fp16 memory-efficient attention path so
fp32 eval matches the linear weights. experiment_specs/evaluate_spec.yaml added.
- mae/scripts/evaluate.py: additive dispatch — runs the embedding suite when any
evaluate.{knn,segmentation,retrieval}.enabled, else the existing classification
test (unchanged). Uses the MAE ViT adapter.
- README command table regenerated (nvdinov2 now lists `evaluate`).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Varun Praveen <vpraveen@nvidia.com>
The nested double quotes inside the f-string replacement field are only valid from Python 3.12 (PEP 701). CI lints on Python 3.11, so pylint reports a syntax error (E0001) for any change set that imports nvidia_tao_pytorch.ssl.mae.model.pl_model. Switch the inner quotes to single quotes. Signed-off-by: Varun Praveen <vpraveen@nvidia.com>
vpraveen-nv
force-pushed
the
feature/core-eval-d
branch
from
August 10, 2026 19:26
85e4a2f to
ef1ecc9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes are proposed in this pull request?
Front-ends onto the shared embedding-eval suite: nvdinov2 gains a new
evaluateaction (auto-registered subtask); MAE'sevaluategets additive dispatch — embedding suite when enabled, else the existing classification test.EvalSuiteConfigis mixed into each network's EvaluateConfig; README command table regenerated.Why are the changes needed?
Part 4/5 of the shared
core/evaluationembedding-eval suite (epic TAO-2181): exposes the shared KNN/embedding evaluation to the NV-DINOv2 and MAE model families.Related issues
JIRA: TAO-2149 (parent epic TAO-2181). N/A for GitHub issues.
Does this PR introduce any user-facing change?
Yes — new
nvdinov2 evaluateaction;mae evaluategains embedding-suite dispatch when enabled (existing behavior unchanged when disabled).How was this patch tested?
Verified in-container:
nvdinov2 evaluateruns end-to-end. Config-compose unit tests for nvdinov2 + mae land in #80.Was this patch authored or co-authored using generative AI tooling?
Yes — portions were co-authored with an AI coding assistant.
Release note
Checklist
git commit -s) — seeCONTRIBUTING.mdso the package still works without them (reviewers: please verify this)
my commit author name and email become permanently public once merged
Notes for reviewers
Review the core-eval stack in order: #78 → #79 → #80 → #81 (this is part 4/5).
Migrated from GitLab MR nvidia-tao-toolkit/tao-pytorch!617, rebased onto GitHub
main.