Merged
Conversation
Co-authored-by: Tanny Chavez Esparza <tanchavez@lbl.gov>
…aoyachong/mlex_tomo_framework into xiaoya-update-prefect3
…aoyachong/mlex_tomo_framework into xiaoya-update-prefect3
…aoyachong/mlex_tomo_framework into xiaoya-update-prefect3
Wiebke
approved these changes
Dec 4, 2025
Member
Wiebke
left a comment
Wiebke
left a comment
There was a problem hiding this comment.
This all works well and the result-tracking related updates are a great simplification!
Note: The online PR view seems to include some of the changes already included in #15
Copying the diff calculated locally below to track changes reviewed.
diff --git a/.env.example b/.env.example
index f96c2f5..e383809 100644
--- a/.env.example
+++ b/.env.example
@@ -25,18 +25,6 @@ TILED_INGEST_RMQ_PW=guest
FLOW_NAME="Parent flow/launch_parent_flow"
TIMEZONE="US/Pacific"
-RESULTS_DIR="/app/work/results" #for docker flow type
-MOUNT_RESULTS_DIR="${PWD}/data/results"
-# Define flow type
-FLOW_TYPE="docker"
-
-# Environment variables for docker-based flows
-IMAGE_NAME="ghcr.io/mlexchange/mlex_dlsia_segmentation_prototype"
-IMAGE_TAG="main"
-CONTAINER_NETWORK="mlex_tomo_framework_mle_net"
-
-TRAIN_SCRIPT_PATH="src/train.py"
-SEGMENT_SCRIPT_PATH="src/segment.py"
#mlflow_db
MLFLOW_DB_PW=<unique password>
diff --git a/README.md b/README.md
index 4ab4b41..860277b 100644
--- a/README.md
+++ b/README.md
@@ -89,5 +89,15 @@ Folder where Tiled reads and writes data. It has two subdirectories:
- `recons` this folder contains sample reconstruction data, and is NOT included in `.gitignore`. It exists to easily deliver some synthetic data for developers to quickly test with. It is also a test base for the `tiled_ingest` project, which requires data to already exist in Tiled's file system.
- `writable` this folder is intended for developers to write data into Tiled with, and is included in `.gitignore`
-## MLFlow Configuration in .env
-You only need to set `MLFLOW_TRACKING_USERNAME` and `MLFLOW_TRACKING_PASSWORD` in the `.env` file. These values will be used to update the `admin_username` and `admin_password` fields in `basic_auth.ini.example` and automatically generate `basic_auth.ini` in the container when the service starts. There’s no need to modify `basic_auth.ini.example` or create `basic_auth.ini` manually.
\ No newline at end of file
+## MLFlow Configuration
+You only need to set `MLFLOW_TRACKING_USERNAME` and `MLFLOW_TRACKING_PASSWORD` in the `.env` file. These values will be used to update the `admin_username` and `admin_password` fields in `basic_auth.ini.example` and automatically generate `basic_auth.ini` in the container when the service starts. There’s no need to modify `basic_auth.ini.example` or create `basic_auth.ini` manually.
+
+If you are running MLflow locally, add a hostname mapping so that `MLflow` resolves to your local machine:
+```bash
+sudo vim /etc/hosts
+```
+
+Then add the following line:
+```
+127.0.0.1 mlflow
+```
\ No newline at end of file
diff --git a/docker-compose.yml b/docker-compose.yml
index cc593b4..e0499d4 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -1,7 +1,7 @@
version: "3.3"
services:
prefect:
- image: prefecthq/prefect:2.14-python3.11
+ image: prefecthq/prefect:3.4.2-python3.11
command: prefect server start
environment:
- PREFECT_SERVER_API_HOST=0.0.0.0
@@ -128,6 +128,7 @@ services:
mlex_segmentation:
image: ghcr.io/mlexchange/mlex_highres_segmentation:main
+ command: sh -c "python scripts/save_mlflow_algorithm.py || true && gunicorn -b 0.0.0.0:8075 --reload app:server"
environment:
- DATA_TILED_URI=http://tiled:8000/api/v1/metadata/recons
- DATA_TILED_API_KEY=${TILED_API_KEY}
@@ -137,25 +138,18 @@ services:
- SEG_TILED_API_KEY=${TILED_API_KEY}
- USER_NAME=${MLEX_SEGM_USER}
- USER_PASSWORD=${MLEX_SEGM_PW}
- - RESULTS_DIR=${RESULTS_DIR}
- PREFECT_API_URL=http://prefect:4200/api
- FLOW_NAME=${FLOW_NAME}
- TIMEZONE=${TIMEZONE}
- - FLOW_TYPE=${FLOW_TYPE}
- - IMAGE_NAME=${IMAGE_NAME}
- - IMAGE_TAG=${IMAGE_TAG}
- - CONTAINER_NETWORK=${CONTAINER_NETWORK}
- - MOUNT_RESULTS_DIR=${MOUNT_RESULTS_DIR}
- - TRAIN_SCRIPT_PATH=${TRAIN_SCRIPT_PATH}
- - SEGMENT_SCRIPT_PATH=${SEGMENT_SCRIPT_PATH}
# MLflow
- MLFLOW_TRACKING_URI=${MLFLOW_TRACKING_URI}
- MLFLOW_TRACKING_USERNAME=${MLFLOW_TRACKING_USERNAME}
- MLFLOW_TRACKING_PASSWORD=${MLFLOW_TRACKING_PASSWORD}
+ # MLflow algorithm registration
+ - MLFLOW_TRACKING_URI_OUTSIDE=${MLFLOW_TRACKING_URI}
+ - ALGORITHM_JSON_PATH=/app/assets/models.json
ports:
- 8075:8075
- volumes:
- - ${MOUNT_RESULTS_DIR}:/app/work/results
# depends_on:
# - tiled
# Intermediate solution restarting frontend on failure due to connection issues
@@ -166,4 +160,5 @@ services:
networks:
mle_net:
+ name: mle_net
driver: bridge
Contributor
Author
You are right. I merged #15 into this PR before #15 was merged into the main branch. As a result, the online PR already includes some of the changes from #15.
You are right. I merged #15 into this PR before #15 was merged into the main branch. As a result, the online PR already includes some of the changes from #15. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR upgrades to
Prefect3.4.2, aligning with the newmlex_utilsandmlex_prefect_workerversions.It also removes unnecessary environment-related variables, since the new
Prefectworker now loads environment details from aconfig.ymlfile instead of receiving them from the application side.Companion PRs:
highres_segmentation: mlexchange/mlex_highres_segmentation#201
prefect worker: mlexchange/mlex_prefect_worker#26
mlex_utils: mlexchange/mlex_utils#5
dlsia_proto: mlexchange/mlex_dlsia_segmentation_prototype#38