Skip to content

[BUG] Handle BERTScore baseline files #995

@amstu2

Description

@amstu2

Describe the bug

The default BERTScorer has a rescale_with_baseline argument that is set to True. When this flag is set, the BERTScorer class requires a baseline_path to be set, otherwise it will attempt to load TSV files that are expected to be in the src/lighteval/metrics/imports directory.

To Reproduce

Requires #990 to be applied to fix overflow error.

from lighteval.models.dummy.dummy_model import DummyModelConfig
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters


def main():
    task = "helm|summarization:xsum|0"

    model_config = DummyModelConfig(
        model_name="dummy",
        seed=1234,
    )

    pipeline_params = PipelineParameters(
        launcher_type=ParallelismManager.NONE,
        max_samples=1
    )

    evaluation_tracker = EvaluationTracker(
        output_dir="./results",
        save_details=True,
        push_to_hub=False,
    )

    pipeline = Pipeline(
        tasks=task,
        pipeline_parameters=pipeline_params,
        evaluation_tracker=evaluation_tracker,
        model_config=model_config,
    )
    pipeline.evaluate()

if __name__ == "__main__":
    main()

Expected behavior

Either a warning is displayed to the user to let them know that the baselines need to be manually downloaded and where they can be acquired, or lighteval will handle the management of loading the baseline files. It looks like the files are downloaded from here.

I'm happy to create a PR to fix this, but need to know the preferred method.

Version info

main on c7a063aee58dc14f51841a6b4f977cd914e10a8f

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions