from-307-dockerize-the-best-run-from-HPO-study

[david-thrower]

TLDR Package the settings from best run in Docker

Issue

We have found the optimal model on the HPO srudy done on branch 307. We need to package this run in a suitable GPU enabled container to scale it up and control the dependencies.

Task

Make this run in tensorflow/tensorflow:2.20.0-gpu

• Dockerize the script
• Set up a volume mount for artifacts
• Clean up dependencies
• Parameterize the script (Dataset, number of samples, sample expansion, ...)

To DO:

• Replace the prompt samples with the ones from the data set.
• Set the stage 1-a and 1-b model checkpoints to save in the artifacts folder.
• Make sure model serialization still works on the mounted volume
• Add conditional MlFlow logging of params (log if MLFLOW_PORT != 0).
• Set MlFlow up on SQLite on the monted volume.
• Add MlFlow system metrics.
• make sure the experiment name is unique automatically. (Generate a unique name to avoid naming collisions.)
• Rebuild and push the latest container.

[david-thrower]

Final smoke tests running at:

https://github.com/david-thrower/cerebros-core-algorithm-alpha/actions/runs/21200746037/job/60985758425

[david-thrower]

The run listed above exhausted the Github Actions workflows (4 CPU 16 GB) resources. I scaled down and this one passed all tests:

https://github.com/david-thrower/cerebros-core-algorithm-alpha/actions/runs/21214578829/job/61031958622#step:4:5439

This should be the initial release of the container.

[david-thrower]

Moved the mlflow.enable_system_metrics_logging() to the correct conditional block. This should be the final run before release (a few small details ironed out).

https://github.com/david-thrower/cerebros-core-algorithm-alpha/actions/runs/21222228108/job/61059727431