Smoke Tests Fail due to tolerances on my machine

[scarere]

For some reason smoke tests fail locally due to results being outside tolerances. Tried starting from a fresh environment and the issue persists. Not urgent but worth looking into eventually. Creating an issue so that I don't forget about it.

System Info

13th Gen Intel® Core™ i9-13900K × 32
Ubuntu 22.04.4 LTS
NVIDIA GeForce RTX 4090
Driver Version: 555.42.06
Cuda Version: 12.1
Python Version: 3.9.19

Steps to recreate

python3.9 -m venv test_env
source test_env/bin/activate
cd ../../FL4Health
git checkout main
git pull
pip install --upgrade pip poetry
poetry install --with "codestyle, dev, dev-local, test, picai"
python -m tests.smoke_tests.run_smoke_test

Output

$ python -m tests.smoke_tests.run_smoke_test
2024-08-08 15:59:27 INFO     Running smoke tests with parameters:
2024-08-08 15:59:27 INFO     	Server : examples.fedprox_example.server
2024-08-08 15:59:27 INFO     	Client : examples.fedprox_example.client
2024-08-08 15:59:27 INFO     	Config : tests/smoke_tests/fedprox_config.yaml
2024-08-08 15:59:27 INFO     	Dataset: examples/datasets/mnist_data/
2024-08-08 15:59:27 INFO     Preloading MNIST dataset...
2024-08-08 15:59:27 INFO     Preloading MNIST dataset...
INFO :      Data directory: examples/datasets/mnist_data
2024-08-08 15:59:27 INFO     Data directory: examples/datasets/mnist_data
Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden

Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/train-images-idx3-ubyte.gz
100%|████████████████████████████████████████████████████| 9912422/9912422 [00:00<00:00, 22336309.08it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/train-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden

Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/train-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/train-labels-idx1-ubyte.gz
100%|█████████████████████████████████████████████████████████| 28881/28881 [00:00<00:00, 1481673.44it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/train-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden

Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/t10k-images-idx3-ubyte.gz
100%|████████████████████████████████████████████████████| 1648877/1648877 [00:00<00:00, 13375956.93it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/t10k-images-idx3-ubyte.gz to examples/datasets/mnist_data/MNIST/raw

Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz
Failed to download (trying next):
HTTP Error 403: Forbidden

Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz
Downloading https://ossci-datasets.s3.amazonaws.com/mnist/t10k-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw/t10k-labels-idx1-ubyte.gz
100%|██████████████████████████████████████████████████████████| 4542/4542 [00:00<00:00, 12683441.26it/s]
Extracting examples/datasets/mnist_data/MNIST/raw/t10k-labels-idx1-ubyte.gz to examples/datasets/mnist_data/MNIST/raw

2024-08-08 15:59:33 INFO     Finished preloading MNIST dataset
2024-08-08 15:59:33 INFO     Starting server...
2024-08-08 15:59:36 INFO     Server started
2024-08-08 15:59:36 INFO     Starting client 0
2024-08-08 15:59:36 INFO     Starting client 1
2024-08-08 15:59:36 INFO     Collecting output for Client 0...
2024-08-08 15:59:46 INFO     Output collected for Client 0
2024-08-08 15:59:46 INFO     Collecting output for Client 1...
2024-08-08 15:59:46 INFO     Output collected for Client 1
2024-08-08 15:59:46 INFO     All clients finished execution
2024-08-08 15:59:46 INFO     Collecting output for Server...
2024-08-08 15:59:49 INFO     Output collected for Server
2024-08-08 15:59:49 INFO     Server has finished execution
Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/home/shawn/Code/FL4Health/tests/smoke_tests/run_smoke_test.py", line 462, in <module>
    loop.run_until_complete(
  File "/usr/lib/python3.9/asyncio/base_events.py", line 647, in run_until_complete
    return future.result()
  File "/home/shawn/Code/FL4Health/tests/smoke_tests/run_smoke_test.py", line 260, in run_smoke_test
    assert len(server_errors) == 0, f"Server metrics check failed. Errors: {server_errors}"
AssertionError: Server metrics check failed. Errors: ["Saved value for metric 'val - prediction - accuracy' (0.4656666666666667) does not match the requested value (0.4627) within requested tolerance (0.0005).", "Saved value for metric 'loss_aggregated' (2.022878646850586) does not match the requested value (2.0268) within requested tolerance (0.0005).", "Saved value for metric 'val - prediction - accuracy' (0.43377777777777776) does not match the requested value (0.4323) within requested tolerance (0.0005).", "Saved value for metric 'loss_aggregated' (1.7308796644210815) does not match the requested value (1.7356) within requested tolerance (0.0005).", "Saved value for metric 'val - prediction - accuracy' (0.5417777777777778) does not match the requested value (0.5362) within requested tolerance (0.0005).", "Saved value for metric 'loss_aggregated' (1.4460455179214478) does not match the requested value (1.4518) within requested tolerance (0.0005)."]