Training data format and info clarification

[Vibhuvan]

Hey, I dont understand how the training data format should since you have not mentioned it properly.

You have written the train_scripts/run_sft_qwen3_8b.sh script in this manner

model_path="/path/to/Qwen3-VL-8B-Instruct"
datasets="gemini_refined_data"
model_id="qwen3-vl-8b"
min_tokens=64
total_tokens=14336
fps=2
fps_max_frames=""
seed=42

But, here you haven't mentioned how the training data gemini_refined_data should be and how the structure should be. Can you please clarify on this?

[jonasjunzhang]

Thanks for pointing this out — you are right that this was not clearly explained enough in the current documentation.

gemini_refined_data in the SFT script is not a JSON file path. It is an internal dataset name recognized by our training data loader.

• SFT script: train_scripts/run_sft_qwen3_8b.sh
• Dataset dispatch logic: training/data/hybrid.py
• Dataset loading logic: training/data/grounding.py
• Actual annotation/video paths: timelens/dataset/timelens_data.py

More specifically, the loading chain is:

1. In the SFT script, we pass:

datasets="gemini_refined_data"

2. That string is interpreted as one of the supported dataset names here:

if name in ("gemini_refined_data", "timelens-100k"):
    filter_args = _build_default_filter_args(data_args.target_size)
    dataset_name = name

Source: training/data/hybrid.py

3. Then the loader resolves gemini_refined_data like this:

if dataset_name in ("gemini_refined_data", "timelens-100k"):
    base_annos = TimeLens100KDataset.load_annos(split="train")
    if dataset_name == "gemini_refined_data":
        raw_annos = [
            anno
            for anno in base_annos
            if not _is_audio_related_query(anno["query"])
        ]
    else:
        raw_annos = base_annos

Source: training/data/grounding.py

So gemini_refined_data is not a separate annotation file you need to prepare for SFT. It means:

• first load the official TimeLens-100K training annotations
• then filter out audio-related queries

The actual paths are hardcoded here:

class TimeLens100KDataset:
    ANNO_PATH_TRAIN = "data/TimeLens-100K/timelens-100k.jsonl"
    VIDEO_ROOT = "data/TimeLens-100K/videos"

Source: timelens/dataset/timelens_data.py

So for SFT, the code actually reads:

• annotation file: data/TimeLens-100K/timelens-100k.jsonl
• video root: data/TimeLens-100K/videos

If you want to prepare/download the data correctly, please follow these README sections:

• Download TimeLens-100K

In other words, if you use our codebase as-is, you should download the official dataset and place it under:

data/TimeLens-100K/
├── timelens-100k.jsonl
└── videos/

For the annotation format, you can refer to the loader here:

• timelens/dataset/timelens_data.py

From the loader, each line in timelens-100k.jsonl is a video-level JSON object containing at least:

• source
• video_path
• duration
• events

and each item in events should contain at least:

• query
• span

Conceptually, the format looks like this:

{
  "source": "...",
  "video_path": "some_subdir/video.mp4",
  "duration": 123.4,
  "events": [
    {"query": "...", "span": [start, end]},
    {"query": "...", "span": [start, end]}
  ]
}