Skip to content

Update data download for gated test and train fallback - #30

Open
AnkitaNaik wants to merge 8 commits into
mainfrom
dev/gated-test
Open

AnkitaNaik wants to merge 8 commits into
mainfrom
dev/gated-test

Conversation

@AnkitaNaik

@AnkitaNaik AnkitaNaik commented Sep 14, 2026

Copy link
Copy Markdown
Collaborator

Summary

This PR updates the VAKRA dataset download flow for the new Hugging Face dataset layout:

  • ibm-research/VAKRA-GatedTest contains the gated test split.
  • ibm-research/VAKRA contains the public train split.

The download flow now preserves anonymous public access while supporting gated test access when Hugging Face authentication is available.

Behavior

make download

make download now:

  1. Creates both data/test/ and data/train/.
  2. Looks for Hugging Face auth from:
    • HF_TOKEN
    • HUGGING_FACE_HUB_TOKEN
    • huggingface-cli login
  3. If auth is available, attempts to download the gated test set into data/test/.
  4. If no auth is available, or the token does not have gated test access, downloads the public train split into data/train/.
  5. Prints a final status message explaining which split was downloaded and how to request gated test access.

benchmark_runner.py

The benchmark runner now:

  1. Checks data/test/capability_*/input/*.json first.
  2. Falls back to data/train/capability_*/input/*.json when the test split is missing or empty.
  3. Logs when it falls back to the train split.

Explicit CAPABILITY_1_DIR, CAPABILITY_2_DIR, CAPABILITY_3_DIR, and CAPABILITY_4_DIR overrides still work as direct user-provided paths.

Example Commands and Output

Anonymous public download

If no Hugging Face auth is configured:

unset HF_TOKEN
unset HUGGING_FACE_HUB_TOKEN
make download

Output

========================================================================
Data sync complete.  GATED TEST SET NOT DOWNLOADED. No Hugging Face auth was found. 
Public train split was downloaded anonymously to data/train instead.
Request gated test access by opening a GitHub issue: [https://github.com/IBM/vakra/issues/new?template=gated_test_access.yml](https://github.com/IBM/vakra/issues/new?template=gated_test_access.yml%22)
========================================================================

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant