Option to disable BERTScore in QAAccuracy

[athewsey]

The addition of BERTScore to the QAAccuracy in version 1.2.0 has massively slowed down evaluation: I've seen interactive evaluations (with our custom Streamlit solution here) change from ~30 seconds to 3 minutes on upgrading; not to mention some AWS Console created jobs going from less than one hour to ~14 hours or more.

Since this scoring method appears very compute intensive compared to the others, can we have a configuration option in QAAccuracyConfig to disable it?

[hellosamstuart]

Adding a +1 for this issue.

While useful for smaller datasets, this addition broke the evaluation pipeline due to timeouts for my customer who is using FMEval QA Accuracy / Factual Knowledge (1600 records)

We will need to roll back the FMEval version in the image to address this limitation however an option to toggle it on/off would solve the problem.

[Zhenshan-Jin]

@athewsey @hellosamstuart thank you for your interest in fmeval. Your request has been shared with the team and will keep you posted about the next step plan.

[hellosamstuart]

Any update on this feature @Zhenshan-Jin?