Skip to content

Add an MS MARCO dataset importer #8

Description

@AshwinUgale

The problem / use case

Context: cli.py has import-squad and import-beir that emit docs.jsonl +
queries.jsonl with source-span gold. MS MARCO is a natural third target.

What: Add an import-msmarco subcommand that converts an MS MARCO passage/qrels
export into the same format.

Acceptance:

  • retrieval-lab import-msmarco … --out-dir … produces valid docs.jsonl + queries.jsonl
  • gold is documented as passage-level (coarser/stricter than answer-span), matching how
    BEIR import already frames it
  • a small fixture test in tests/test_datasets.py; CHANGELOG.md updated

Pointers: mirror import-beir (passage-level relevance → span over the whole passage).

Proposed idea

Add import-msmarco alongside the SQuAD/BEIR importers so users can benchmark on a well-known passage-ranking dataset; its passage-level relevance imports as coarser (passage-span) gold, stated honestly.

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions