-
Notifications
You must be signed in to change notification settings - Fork 364
Open
Labels
Description
Describe the bug
The default BERTScorer has a rescale_with_baseline
argument that is set to True
. When this flag is set, the BERTScorer
class requires a baseline_path
to be set, otherwise it will attempt to load TSV files that are expected to be in the src/lighteval/metrics/imports
directory.
To Reproduce
Requires #990 to be applied to fix overflow error.
from lighteval.models.dummy.dummy_model import DummyModelConfig
from lighteval.logging.evaluation_tracker import EvaluationTracker
from lighteval.pipeline import ParallelismManager, Pipeline, PipelineParameters
def main():
task = "helm|summarization:xsum|0"
model_config = DummyModelConfig(
model_name="dummy",
seed=1234,
)
pipeline_params = PipelineParameters(
launcher_type=ParallelismManager.NONE,
max_samples=1
)
evaluation_tracker = EvaluationTracker(
output_dir="./results",
save_details=True,
push_to_hub=False,
)
pipeline = Pipeline(
tasks=task,
pipeline_parameters=pipeline_params,
evaluation_tracker=evaluation_tracker,
model_config=model_config,
)
pipeline.evaluate()
if __name__ == "__main__":
main()
Expected behavior
Either a warning is displayed to the user to let them know that the baselines need to be manually downloaded and where they can be acquired, or lighteval will handle the management of loading the baseline files. It looks like the files are downloaded from here.
I'm happy to create a PR to fix this, but need to know the preferred method.
Version info
main
on c7a063aee58dc14f51841a6b4f977cd914e10a8f