[TAO-1796][Feature] video_clip: add config dataclasses and schema - #30
Open
lianqiann wants to merge 1 commit into
Open
[TAO-1796][Feature] video_clip: add config dataclasses and schema#30lianqiann wants to merge 1 commit into
lianqiann wants to merge 1 commit into
Conversation
Mirror the tao-pytorch video_clip experiment config into tao-core so the TAO API, the microservices handlers and the generated skill schemas can validate video_clip jobs. - nvidia_tao_core/config/video_clip/ is a verbatim port of the canonical nvidia_tao_pytorch/config/video_clip/default_config.py -- the two files now differ only in the two import prefixes, so the schema cannot drift from the implementation. - Register video_clip in generate_schema.py, which needs the explicit VideoCLIPExperimentConfig branch because the config is not named ExperimentConfig. - Add the video_clip microservices network config. tao-pytorch is the source of truth for these fields; this change only brings the schema side into parity with it. Signed-off-by: Alice Li <alicli@nvidia.com>
|
For security reasons, CI does not run automatically on NVIDIA's runners — it must be triggered per commit.
Tip Should this fix also ship in a release? Add a Important Before merging: every file needs a license header and every commit must be DCO signed-off — see the 📖 Contribution guide for details. |
Author
|
Migrated from GitLab as part of the TAO GitHub-first cutover. This change lands across three repositories — the set is:
The tao-pytorch PR is self-contained and can merge independently: the module does not import |
Author
|
/build |
1 similar comment
Contributor
|
/build |
lianqiann
added a commit
to NVIDIA-TAO/tao-pytorch
that referenced
this pull request
Aug 12, 2026
…re lacks it TestConfigCopiesIdentical compares the video_clip config dataclass in this repo against its duplicate in tao-core, so it is a cross-package contract test by construction. tao-core only grows config.video_clip when NVIDIA-TAO/tao-core#30 lands, and until then the bare import raised ModuleNotFoundError: No module named `nvidia_tao_core.config.video_clip` which reports "the contract cannot be evaluated" as if it were "the two schemas have drifted". Those are different conditions and deserve different outcomes. Guard only the not-yet-shipped module with pytest.importorskip, the idiom already used in this suite for onnxscript, cv2 and timm. Note the rest of the repo is right to import nvidia_tao_core.api_utils unguarded -- that ships today; config.video_clip does not yet. Real drift still fails: the field-set assertions are unchanged, and the test un-skips itself as soon as tao-core ships the module. Verified both paths in the container: with tao-core config.video_clip present the test runs and passes; with it moved aside the test skips with the stated reason instead of erroring. Signed-off-by: Alice Li <alicli@nvidia.com>
hqtran-nv
pushed a commit
to NVIDIA-TAO/tao-pytorch
that referenced
this pull request
Aug 18, 2026
…ndored arch, LoRA PEFT, N-to-N/classification eval, ONNX export) (#94) * [TAO-1796][Feature] video_clip: vendor the InternVideo2-CLIP L14 architecture Vendor the InternVideo2 vision tower and the MobileCLIP text tower into the video_clip module so the model imports normally, without the INTERNVIDEO2_ROOT environment variable or the sys.path / os.chdir import hacks the upstream package requires. Adapted from OpenGVLab InternVideo2 (multi_modality), Apache-2.0. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: LoRA PEFT, task-aware eval and ONNX export Add the video_clip task on top of the vendored InternVideo2-CLIP L14 backbone: - LoRA adapters on the vision and text towers with frozen-teacher preservation (regularization) losses. LoRA is merged back into the base weights on export, so the exported model carries no adapter overhead. - A task-aware video_text dataloader plus N-to-N retrieval and classification evaluation. Validation is sharded across ranks and the eval path is selected by task_type / dataset.metrics.mode. - ONNX export for the InternVideo2-CLIP video model, defaulting to opset 23 with fused RMSNormalization and a dynamic batch dimension. - The video_clip experiment config, accurate trainable-parameter reporting and an eval-time precision fix. The CLIP LoRA PEFT and preservation-loss primitives under multimodal/clip are shared with this task and land with it. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: register the entrypoint, package data and dependencies - Register the video_clip console entrypoint and ship the vendored MobileCLIP configs as package data. The catch-all package_data patterns are not recursive, so the nested configs/ directory needs its own entry. - Declare decord==0.6.0 as the video decoder and onnxscript==0.7.1 for the opset-23 RMSNormalization export. - List video_clip in the README supported-commands tables. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: unit and integration tests Cover the vendored architecture, the LoRA/PEFT wiring, the trainable-parameter report, the video_text dataloader, the config, the subtask scripts and ONNX export parity -- including the MultiheadAttention swap and the dynamic batch dimension. Add the CLIP LoRA training integration test and the InternVideo2 real-pipeline smoke test. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: address PR #94 review feedback Resolves the eight review comments on #94. Loader - Drop the hardcoded /media/wbf/ path remap in _resolve_video_path. It was a leftover from the original dataset author's machine and path_prefix_mapping (checked first) already expresses it. The unit-test fixtures that baked in those paths now go through path_prefix_mapping instead. - Stop reseeding the global random / torch / CUDA RNGs inside get_video_text_dataloader. It ran once per loader, mid datamodule setup, resetting whatever seed_everything() had established. Randomness is now local: a torch.Generator drives RandomSampler and the DataLoader, and dataset.seed is threaded into DistributedSampler, which previously fell back to torch's default seed=0 and so ignored the configured seed on the DDP path. Preservation regularization - Keep the frozen teacher out of Lightning checkpoints. PreservationLoss.state_dict drops teacher.*, and on_load_checkpoint reconciles either shape of checkpoint against the running config, so old (teacher-bearing) and new checkpoints both load strictly with regularization enabled or disabled. Measured on a real InternVideo2-CLIP L14 run: 1.49 GB instead of 2.98 GB. - Skip building the teacher on the restore paths (evaluate / inference / export). It is training-only, and load_model_from_checkpoint nulls every weight source first, so a teacher built there would have held random weights. - Expose PreservationLoss.teacher_forward instead of having pl_clip_model reach into the private _teacher_forward for the val drift metric. PEFT - Raise when PEFT is enabled for a tower but target_modules matches no nn.Linear. inject_lora freezes the backbone first, so this silently left training with nothing trainable. The error names the tower, the requested targets and the nn.Linear leaves actually present, which covers the OpenCLIP case where the fused in_proj is a Parameter on nn.MultiheadAttention rather than a submodule. Supply chain - Pass weights_only=True to the three vendored InternVideo2 torch.load calls, which deserialize artifacts downloaded from an external HF repo. - Pin the default InternVideo2-CLIP HF repo to commit 449f7ea1, so a repo-side change cannot silently alter the weights we train and ship on. A user-supplied repo id is left unpinned. No config-schema change, so tao-core stays in parity. Docs - entrypoint docstring said "CLIP task"; it is the video_clip task. Verified on an A100 against the real 32-clip smoke pack and the real InternVideo2-CLIP L14 / MobileCLIP weights: train, evaluate (new and legacy checkpoints x regularization on/off, all four identical at 0.2221 mAP), and ONNX export with 6 LoRA modules merged and no LoRA nodes left in the graph. Unit suite 464 passed, with the 6 failures that already existed on this branch unchanged. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: document the InternVideo2-CLIP L14-only scope Follow-up to the PR #94 review fixes. The pinned HF repo (OpenGVLab/InternVideo2_distillation_models) also ships the S14 and B14 distillations, so the single pinned revision and the L14 filenames read as an oversight rather than a deliberate boundary. Make the boundary explicit. - internvideo2_assets.py: state that support is L14 only, that the filenames and the architecture in build_internvideo2_l14_config (patch 14, embed_dim 1024, depth 24, num_heads 16, clip_embed_dim 768, align_dim 512) are L14-specific, and that upstream publishes no config for the distilled sizes -- so adding a size means deriving its architecture from its checkpoint, adding its filenames and pinning its own revision. - model_configs.py: note that internvideo2clip_model_configs is intentionally single-entry, unlike the multi-version siglip2/radio/openclip tables beside it. - config/video_clip/default_config.py: say outright in model.type that internvideo2-clip-l14 is the only supported InternVideo2 size. Comments and one description string only; no executable line changes. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: decode with PyAV instead of decord decord's manylinux wheel bundles its own libav* and libx264 under decord.libs/, which the OpenCV leak check does not scan -- the tree already documented this in docker/requirements-pip.txt. The image builds a restricted LGPL-only FFmpeg (docker/Dockerfile) but nothing in Python could use it, so decord was the only working decoder and it smuggled the codecs back in. Following the cosmos-rl pattern, PyAV is built with --no-binary=av against that FFmpeg instead, so it inherits the codec allow-list rather than bundling its own. Loader - _load_with_pyav replaces _load_with_decord, reproducing its contract exactly: the same frame count, the same order, and the same duplicate padding when a clip is shorter than num_frames. - Seeks to the first wanted frame rather than scanning from zero, and seeks per frame on intra-only streams. Without the latter, MJPEG measured 2.57x slower than decord because every frame is a keyframe and decord jumps straight to the ones it needs. - The decode chain is now pyav -> ffmpeg CLI -> OpenCV, and the winning backend is logged once per process. A silent fall back to the ffmpeg CLI is ~10x slower but otherwise invisible, which makes any timing unattributable. Image - --enable-demuxer=avi, with a matching build-time assertion. The restricted build enabled only mov/matroska/image2, so .avi failed at av.open() with InvalidDataError before decoding was attempted. A third of the KPI corpus is .avi (221/665 train, 276/831 eval, MJPEG from RWF-2000) and loads today only because decord ships its own FFmpeg. AVI is a container, not a codec: it carries none of the H.264/HEVC/AAC exposure this build exists to avoid. NOTE: this is a deliberate divergence from cosmos-rl, which has the same three demuxers and would break training here if copied verbatim. - Rebuild PyAV from source after the requirements layer, then fail the build if any av.libs/decord.libs/libx264/libx265/libopenh264 survives or decord is importable. - requirements-pip.txt: decord==0.6.0 -> av==17.1.0, with the wheel explicitly called out as not what ships. Added the SPDX header the file was missing. Consequence: H.264 decode is NVDEC-only (h264_cuvid), so it needs a GPU and NVIDIA_DRIVER_CAPABILITIES including 'video'. That is the point of shipping no software H.264 codec. Verified on dev-box-rtx-pro (RTX PRO 6000 Blackwell) inside tao-toolkit-pyt:v7.0.1-pyt2.1.0-py3-03, whose ffmpeg has h264_cuvid as its only H.264 decoder - PyAV built with --no-binary=av resolves libav* to /usr/local/lib with no av.libs/, and av.Codec("h264","r") resolves to h264_cuvid. - Rebuilt FFmpeg 8.1 with the amended configure line: gpl/nonfree still 0, h264 still h264_cuvid-only, no forbidden encoders, and the MJPEG/AVI clips decode. - Frame equivalence vs decord over 32 real clips x 5 sampling cases: worst per-pixel difference 0. - Throughput vs decord: 1.00x on h264/mp4, 0.81x on mjpeg/avi. - Smoke train A/B on the same GPU, decord vs PyAV: identical retrieval (image_to_text mAP 0.2240, text_to_image 0.1836), 0 decode failures, byte- identical checkpoints, 37s vs 44s wall-clock. - 534 unit tests pass; flake8 / pydocstyle / pylint 10.00 / license headers clean. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Refactor] video_clip: move the decode backends into video_decode video_text_loader.py had grown to 1449 lines covering metadata parsing, video decoding, tensor stacking, the Dataset and the DataLoader factory. The PyAV switch added eight more private functions to it, and decoding is the concern least related to "video-text dataset support". Move the decode chain to dataloader/video_decode.py with load_video_frames as its only public symbol: the PyAV backend and its helpers, the ffmpeg CLI backend and its ffprobe helper, the OpenCV backend, the shared _linspace_indices/_clip_frame_range range helpers, and the backend logger. Pure move -- no function body is edited. video_text_loader re-exports load_video_frames, so scripts/inference.py and the test import block are unchanged, and monkeypatching video_text_loader.load_video_frames still works because "from X import Y" binds the name locally. The only orphan the move left behind, an unused PIL.Image import, is dropped. video_text_loader.py 1449 -> 996 lines; video_decode.py 482. Verified: flake8 (CI flags) clean, pylint 10.00, pydocstyle clean, 86 passed / 1 skipped in tests/multimodal_unit_test/video_clip/, and both import paths resolve to the same function object. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Refactor] video_clip: de-duplicate the PyAV decode paths The PyAV switch landed three variants of one loop -- seek, walk the stream, map each frame to an index, collect the wanted ones, stop past the last -- and re-derived a seek timestamp that a helper twenty lines above already computed. - _pyav_decode_indices becomes a dispatcher over two decoders: seek-to-each (intra-only streams) and a single _pyav_decode_walk that takes where to seek and whether to index by timestamp or by decode order. _pyav_decode_indices_by_scan is that walk with rate=None and seek_to=0, so it is gone, and with it the return issued from inside a live container.decode() iteration. - Both seek sites now call _pyav_seek_to_index instead of open-coding origin + int(index / rate / time_base) a second time. - _load_with_pyav resolves its frame range with _clip_frame_range, the helper the ffmpeg and OpenCV backends already share, instead of its own copy. - load_video_frames iterates a _DECODE_BACKENDS table instead of three copy-pasted try/except blocks that each repeat the same six arguments. - The intra-path fall-through logs at debug like every other backend failure here. It was a bare `pass`, so MJPEG could silently take the 2.57x-slower walk -- the same invisibility this feature's logging exists to prevent. - Dropped .convert("RGB") on PyAV frames: to_rgb().to_image() already returns mode RGB (checked for rgb24, yuv420p and yuvj420p), so it only bought a full image copy per frame per sample. The ffmpeg/OpenCV paths keep theirs, where it follows Image.fromarray and is load-bearing. Two deliberate behaviour changes, both making PyAV agree with the other backends: - An inverted or out-of-range annotation now clamps to a single frame rather than raising. The raise protected nothing: it dropped through to the ffmpeg CLI, which clamped and succeeded ~10x slower and silently. A parametrized case pins the new semantics. - When timestamps turn out to be unusable, the retry indexes every frame by decode order. Previously a mid-stream pts of None either restarted the scan or, if no seek had happened, mixed counter-derived indices in among pts-derived ones. video_decode.py 482 -> 452 lines; the two PyAV tests collapse to one parametrized case over a shared fake_pyav fixture, 100 -> 63 lines including the new clamp case. Verified on rtdetr-pytorch2 against the 32-clip smoke pack (24 mp4/H.264, 8 avi/MJPEG), _load_with_pyav before (66313d1) vs after over 5 sampling cases -- full clip, frame range, time range, short-clip padding, single frame: - 160/160 comparisons, worst per-pixel difference 0, no divergent raises. - Path coverage confirmed by instrumentation: .avi takes the intra seek-to-each path, .mp4 takes the timed walk. - flake8 (CI flags) clean, pylint 10.00, pydocstyle clean, 87 passed / 1 skipped in tests/multimodal_unit_test/video_clip/. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Refactor] video_clip: correct the onnxscript pin note, drop dead export code The onnxscript pin's comment named only the opset-23 RMSNormalization fusion, which invites deleting the fusion and the pin together. Two things are wrong with that reading. onnxscript is required twice over. torch imports it directly for the dynamo ONNX exporter (torch/onnx/_internal/exporter/_core.py), which export.py selects whenever opset > 20 -- and the default opset is 23, so every default export needs it. Measured: a bare nn.Linear at opset 23 raises ModuleNotFoundError without onnxscript, while the same model at opset 17 exports fine. torch declares no dependency on onnxscript, so nothing installs it unless this pin does. It is also required by the shipped fp16 TensorRT path, which consumes the opset-23 graph: ModelOpt AutoCast runs --op_types_to_exclude RMSNormalization, and those nodes exist only because _fuse_rms_normalization emits them. That path measured mAP 0.5450 against an fp32 control of 0.5447, at 21.8 ms (3.84x fp32), with no manual layer pin. The opset-17 alternative is not interchangeable: under opset 23 TensorRT auto-names the decomposed layers, so the older named-layer fp32 pin matches 0 of them. Worth stating plainly, because the record reads the other way at a glance: "opset-23 ruled out" refers to relying on stash_type alone in a weakly-typed TRT build, which collapses to mAP 0.1456. Opset 23 itself is required -- it is the substrate the shipped solution stands on. The pin stays exact rather than >=, because export.py imports the private onnxscript.rewriter._ir_utils and _fusion_utils, which carry no API-stability guarantee. Separately, this file was seeded from multimodal/clip/scripts/export.py and inherited two blocks that cannot execute for InternVideo2 -- the comment still named SigLIP2 NaFlex: - the aten::_upsample_bilinear2d_aa custom symbolic, which needs antialias=True to be reachable (nothing in video_clip passes it) and is registered through a legacy-exporter-only API the dynamo path ignores. It also mutated torch's global symbolic registry at import time. - ExportFriendlyMHA and _replace_mha_for_export, which look for nn.MultiheadAttention. Counted on the real loaded model: 0 of 752 modules. MobileCLIP's attention is a custom MultiHeadAttention class, not the torch one, which is why the swap never matched. test_export_mha_swap passed only because it built synthetic nn.MultiheadAttention models. multimodal/clip/scripts/export.py is untouched -- both blocks are live there for SigLIP2. export.py 1265 -> 1102 lines, plus the two imports (math, torch.nn.functional) that only the removed MHA decomposition used. Verified on rtdetr-pytorch2 in venv tao-cli-github: - Combined-encoder export at the default opset 23, before vs after, from the same checkpoint: graphs IDENTICAL -- 1521 nodes, 527 initializers, 96 RMSNormalization, 23 op types, no op-histogram difference, same inputs and outputs. - "Fused 96 decomposed RMSNorm subgraph(s)" still logged, matching the count on record for this architecture. - flake8 (CI flags) clean, pylint 10.00, pydocstyle clean. - tests/multimodal_unit_test/video_clip 83 passed / 1 skipped; the one failure is the pre-existing environment-only tao-core #30 gap (nvidia_tao_core.config.video_clip not installed), untouched by this change. - tests/multimodal_unit_test/clip 456 passed, confirming the sibling exporter is unaffected. Requirements change is comment-only, so the built image layer is unchanged. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Test] video_clip: guard the opset-23 RMSNormalization fusion _fuse_rms_normalization is built on onnxscript.rewriter._ir_utils and _fusion_utils. Both are private and carry no API-stability guarantee, so an onnxscript bump can make the rewrite match nothing. Nothing caught that: the helper would return 0, the export would still succeed, CI would stay green, and the damage would surface only later as an fp16 accuracy collapse -- the deploy path keeps RMSNorm in fp32 via ModelOpt AutoCast's --op_types_to_exclude RMSNormalization, which needs the fused node to exist. Two tests over a stack of longhand RMSNorms exported at opset 23: - every decomposed subgraph is rewritten (count matches, and the Pow/ReduceMean decomposition is gone rather than shadowed), plus an assertion that torch did not already emit a fused node on its own -- if a future torch does, the helper has become unnecessary and this fails to say so. - the fused node carries stash_type fp32 and axis -1, which is the property the deploy side actually depends on. Guarded with importorskip so environments without onnxscript skip rather than error; CI has it via docker/requirements-pip.txt. Verified on rtdetr-pytorch2 in venv tao-cli-github: both pass, and both fail when the rewrite pattern is deliberately broken (Pow exponent 2.0 -> 3.0), so the tests are load-bearing rather than decorative. flake8 (CI flags) clean, pylint 10.00, pydocstyle clean. Full video_clip suite 85 passed / 1 skipped; the single failure is the pre-existing environment-only tao-core #30 gap (nvidia_tao_core.config.video_clip not installed). Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Docs] video_clip: shorten the onnxscript pin note Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Docs] video_clip: trim the av pin note Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: fix retrieval AUC label alignment compute_auc() sorts its `scores` argument internally and indexes `labels` with the result, so both arrays must be in the same order. The call passed gallery-order `sims` with rank-order `sorted_labels`, pairing each score with the wrong item relevance. Pass `sims[sorted_idx]` so the two align. The reported AUC was not merely noisy but inverted: a query whose only relevant clip ranks last scored 1.0 instead of 0.0, and vice versa. Adds tests/multimodal_unit_test/video_clip/test_retrieval_auc.py -- the AUC path in this evaluator had no coverage, so nothing exercised the line. The gallery is the identity basis and the query is chosen so the ranking order differs from gallery order, which is what makes the test able to tell the two orderings apart. Mirrors the same fix in tao-deploy PR #36, keeping the deploy copy of this evaluator identical to this one. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: assert RMSNorm fusion end state, not who fused torch 2.11 / onnxscript 0.6 fuse RMSNorm in their own ONNX optimization pass, so `_fuse_rms_normalization` finds nothing left to match and the tests failed on a stale precondition: test_fuse_rms_normalization_rewrites_every_subgraph AssertionError: torch exported a fused RMSNormalization on its own test_fused_node_stashes_in_fp32 AssertionError: assert 0 == 1 Verified this is a test assumption and not a deploy regression: the natively fused node carries axis=-1 and stash_type=1 (TensorProto.FLOAT), exactly what the helper emitted, so AutoCast still has the fp32-marked RMSNormalization it needs to exclude from fp16. Both tests now assert the end state -- every decomposed subgraph ends up fused, with the decomposition gone -- and count `pre_fused + fused`, so they hold whichever component does the work. This keeps the loud failure the file exists for. Checked against a deliberately unfusable export (opset 17, which has no RMSNormalization): pre_fused + fused = 0 != 3 and the Pow/ReduceMean chain survives, so a silently non-matching onnxscript rewrite still fails the test. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: skip tao-core parity test when tao-core lacks it TestConfigCopiesIdentical compares the video_clip config dataclass in this repo against its duplicate in tao-core, so it is a cross-package contract test by construction. tao-core only grows config.video_clip when NVIDIA-TAO/tao-core#30 lands, and until then the bare import raised ModuleNotFoundError: No module named `nvidia_tao_core.config.video_clip` which reports "the contract cannot be evaluated" as if it were "the two schemas have drifted". Those are different conditions and deserve different outcomes. Guard only the not-yet-shipped module with pytest.importorskip, the idiom already used in this suite for onnxscript, cv2 and timm. Note the rest of the repo is right to import nvidia_tao_core.api_utils unguarded -- that ships today; config.video_clip does not yet. Real drift still fails: the field-set assertions are unchanged, and the test un-skips itself as soon as tao-core ships the module. Verified both paths in the container: with tao-core config.video_clip present the test runs and passes; with it moved aside the test skips with the stated reason instead of erroring. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: ship experiment specs in the wheel experiment_specs/ is not a Python package, so find_packages() skips it and the non-recursive catch-all package_data globs never reach its YAML. Add a package-scoped key on nvidia_tao_pytorch.multimodal.video_clip, matching the vendored MobileCLIP configs entry directly above it. Signed-off-by: Alice Li <alicli@nvidia.com> * [TAO-1796][Feature] video_clip: reject image-only model types The video dataloader emits [B, T, C, H, W] and only the InternVideo2-CLIP adapter consumes 5D input, but build_model still dispatched to the C-RADIO / SigLIP2 / OpenCLIP builders and an open_clip catch-all, which built fine and then failed on the first forward pass. Reject them up front and trim the model.type description, which was inherited from the image-CLIP task. Signed-off-by: Alice Li <alicli@nvidia.com> --------- Signed-off-by: Alice Li <alicli@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Mirrors the tao-pytorch
video_clipexperiment config into tao-core so the TAO API, the microservices handlers and the generated skill schemas can validatevideo_clipjobs.Changes
nvidia_tao_core/config/video_clip/— a verbatim port of the canonicalnvidia_tao_pytorch/config/video_clip/default_config.py. The two files now differ only in the two import prefixes, so the schema cannot drift from the implementation.generate_schema.py— registervideo_clip. It needs an explicitVideoCLIPExperimentConfigbranch because the config class is not namedExperimentConfig.video_clip.config.json— the microservices network config.Testing
VideoCLIPExperimentConfig()instantiates with the expected top level (dataset,model,peft,regularization,train,evaluate,inference,export,gen_trt_engine,wandb, …).generate_schema("video_clip")produces a valid schema (11 properties) intao-toolkit-pyt:v7.0.1-pyt2.1.0.pre-commit(license header, pylint, pydocstyle, flake8) clean over the full diff range.Notes for review
alicli/video_clip_lora_taocore) had drifted 202 lines from the canonical copy — it still advertised the WebDataset/custom dataset schema that tao-pytorch has since removed, and lackedcaption_mode,opt_batch_sizeandrelevance_file. Since parity is the point of this change, the canonical file was ported directly rather than replaying that branch.mainalready shipspeft/lora/regularizationinconfig/clip/default_config.py, but tao-coremainhas none of it — a pre-existing schema gap that predates this work and deserves its own PR.Migration provenance
Companion to the manual GitLab-to-GitHub migration of tao-pytorch MR !620. No GitLab MR was ever opened for the tao-core half.