[BUG][RAG][v0.12.6][RUN_EVALS] Online aggregation updates at shard boundaries only

[anay-rfai]

Bug Description

When metrics are being evaluated using online aggregation, it is desirable to see real-time updates to the estimates and confidence intervals. Right now it only updates at shard boundaries. That means, if I have num_shards=4, I have to wait for 25% (1/4) of the data to be processed before I can see the first estimate.

To Reproduce

1. Set online aggregation in the config.
   "online_strategy_kwargs": {
   "strategy_name": "normal",
   "confidence_level": 0.95,
   "use_fpc": True,
   }
2. Run any of the context engineering notebooks -- I'm running GSM8K notebook.

Expected Behavior

Ideally it should update after every mini-batch of inference. But if not, at least update with a reasonable wall clock frequency such as every 30/60 sec

Screenshots

Environment

(infer) palebluedot@palebluedotcloud:~/infer$ rapidfireai doctor
🔍 RapidFire AI System Diagnostics
==================================================

🐍 Python Environment:
------------------------------
Version: 3.12.12
Implementation: CPython
Executable: /home/palebluedot/miniconda3/envs/infer/bin/python3.12
Conda Environment: infer
Virtual Environment: no

📦 Installed Packages:
------------------------------
bitsandbytes                       0.48.2
evaluate                           0.4.6
flashinfer-python                  0.2.5+cu124torch2.5
Flask                              3.1.2
flask-cors                         6.0.1
gunicorn                           23.0.0
mlflow                             3.6.0
mlflow-skinny                      3.6.0
mlflow-tracing                     3.6.0
nltk                               3.9.2
peft                               0.18.0
rapidfireai                        0.12.6rc1
sentence-transformers              5.1.0
sentencepiece                      0.2.1
torch                              2.5.1+cu124
torchaudio                         2.5.1+cu124
torchvision                        0.20.1+cu124
transformers                       4.56.1
trl                                0.21.0
... (showing only relevant packages)

🚀 GPU & CUDA Information:
------------------------------
nvidia-smi: found
Driver Version: 575.64.03
CUDA Runtime: 12.9
GPU Count: 1
GPU Model: NVIDIA A100-SXM4-80GB
Total VRAM: 80.0 GB
nvcc: not found
CUDA Installation: present
CUDA on PATH: no

💻 System Information:
------------------------------
Platform: Linux-6.8.0-64-generic-x86_64-with-glibc2.35
Architecture: x86_64
Processor: x86_64

🔧 Environment Variables:
------------------------------
CUDA_HOME: not set
CUDA_PATH: not set
LD_LIBRARY_PATH: not set
PATH: /home/palebluedot/miniconda3/envs/infer/bin:/home/palebluedot/miniconda3/condabin:/home/palebluedot/...

✅ Diagnostics complete!

Additional Context

Add any other context about the problem here.

Error Logs

If applicable, paste any error logs or stack traces here.

Paste error logs here