I'm integrating fmeval with experiments tracking solutions (MLflow for now), and the lack of callback mechanisms means that the tracing can only happen after an evaluation is completed.
Drawbacks:
The suggested solution is to implement a callback mechanism to be able to tracks results as they're generated, simplifying integration with experiment tracking solutions.