[TAO-1796][Feature] video_clip: add InternVideo2-CLIP TensorRT deploy module - #36
Conversation
…ploy module Add the deploy-side counterpart of the tao-pytorch video_clip task: build a TensorRT engine from the exported InternVideo2-CLIP ONNX, then run retrieval inference and evaluation against it. - gen_trt_engine, inference and evaluate subtasks behind a video_clip entrypoint, with the shipped experiment spec and config dataclasses. - A dataloader that mirrors the training-time frame sampling, ImageNet normalization and optional text canonicalization, so deploy-time preprocessing matches training exactly. - Retrieval metrics (mAP, Recall@K, median/mean rank) over the combined video and text embeddings. - EngineBuilder gains an overridable _extra_network_flags() hook, defaulting to 0, so this module can request a strongly-typed network. That is what keeps the AutoCast mixed-precision ONNX from being re-interpreted by the weakly-typed fp16 path. Signed-off-by: Alice Li <alicli@nvidia.com>
Register the video_clip console entrypoint and regenerate docs/supported_commands.md from it. Signed-off-by: Alice Li <alicli@nvidia.com>
Cover the config dataclasses, the dataloader preprocessing, the engine builder, the entrypoint wiring, the retrieval metrics and the inferencer. Signed-off-by: Alice Li <alicli@nvidia.com>
|
For security reasons, CI does not run automatically on NVIDIA's runners — it must be triggered per commit.
Tip Should this fix also ship in a release? Add a Important Before merging: every file needs a license header and every commit must be DCO signed-off — see the 📖 Contribution guide for details. |
|
Migrated from GitLab as part of the TAO GitHub-first cutover. This change lands across three repositories — the set is:
The tao-pytorch PR is self-contained and can merge independently: the module does not import |
|
/build |
1 similar comment
|
/build |
|
Hi @lianqiann 👋💛 🤖✨ AIRE (NVIDIA's automated CI agent) here! ✨🤖 🔍🚨 I detected that the CI pipeline on your PR #36 has failed — but no worries, I'm already on it! 💪🔧 🛠️🔬 I'm automatically diagnosing the root cause right now and preparing a fix for you. ⏳✅ 🔔💬 I'll ping you the moment the fix is ready to review — nothing you need to do for now. Sit back and relax! 🌿😌 💙🙏 Thanks for your patience — AIRE's got your back! 🚀🌟 |
|
Hi @lianqiann 👋🎉 🤖✨ AIRE (NVIDIA's automated CI agent) here with great news! ✨🤖 ✅🛠️ I've finished diagnosing and automatically fixed the CI failure on your PR #36! 🎯💚 What was wrong: The fix: add the missing Verified green: the fix branch now collects 316 items and reports 296 passed, 20 skipped, 0 failed ✅ 🔗📦 The fix is ready in a dedicated PR → #38 🙏💪 Please take a quick look and merge it whenever you can, so the fix lands and your CI goes green again. ✅🌟 💙🎉 Thank you so much — merging AIRE's fix keeps CI healthy for the whole team! 🚀 |
vpraveen-nv
left a comment
vpraveen-nv
left a comment
There was a problem hiding this comment.
Thanks Alice — clean deploy module, and the preprocessing-contract documentation in the dataloader is exactly the right emphasis. I checked GitLab !205 (no prior review comments there) so nothing here is a repeat. One real bug in the AUC computation (inherited from the tao-pytorch twin — worth fixing in both), plus a few smaller items inline.
tests/video_clip and tests/clip share test-module basenames; without __init__.py pytest imports both under bare module names and aborts collection with import file mismatch. Signed-off-by: Alice Li <alicli@nvidia.com>
Fix the AUC computation and three review items from vpraveen-nv: - retrieval.py: compute_auc() sorts its `scores` argument internally and indexes `labels` with the result, so both arrays must be in the same order. The call passed gallery-order `sims` with rank-order `sorted_labels`, pairing each score with the wrong item relevance. Pass `sims[sorted_idx]` so the two align. The reported AUC was not merely noisy but inverted: a query whose only relevant clip ranks last scored 1.0 instead of 0.0. Covered by two new tests -- the AUC path had no coverage at all, since every existing case set compute_auc=False. - dataloader.py: drop the hardcoded /media/wbf/ prefix remap. It is a leftover from the original dataset author machine and is already expressible via path_prefix_mapping, which is checked first. This matches the same removal in the tao-pytorch twin (PR #94). - scripts/evaluate.py: warn when no *_config.yaml is found next to the engine instead of silently assuming canonicalize_text=False. A model exported with canonicalize_text: true would be queried with mismatched tokens and just score worse, with nothing in the log to say so. - config/.../default_config.py: route path_prefix_mapping through DICT_FIELD like every other key in the schema, so its description and display name reach default_specs. Signed-off-by: Alice Li <alicli@nvidia.com>
|
/build |
5 participants
Summary
Adds the deploy-side counterpart of the tao-pytorch
video_cliptask: build a TensorRT engine from the exported InternVideo2-CLIP ONNX, then run retrieval inference and evaluation against it.Changes
video_clipentrypoint withgen_trt_engine,inferenceandevaluatesubtasks, the shipped experiment spec, and the config dataclasses underconfig/multimodal/video_clip/.EngineBuilder._extra_network_flags()— a new overridable hook on the shared builder, defaulting to0, OR-ed into thecreate_networkflags.video_clipoverrides it to request a strongly-typed network, which is what stops TensorRT's weakly-typed fp16 path from re-interpreting the AutoCast mixed-precision ONNX. No behaviour change for any existing network.Testing
pre-commit(docs supported-commands, license header, pylint, pydocstyle, flake8) clean over the full diff range.Migration provenance
Manual migration of GitLab MR !205 (branch
alicli/iv2clip). The GitHub and GitLab repositories share no git history, so the branch's net diff was replayed ontomainas three logical commits. Two notes:docs/supported_commands.mdis regenerated by the repo's ownupdate-docs-supported-commandshook and is included here.Depends on the tao-pytorch PR for the exported model this module consumes.