Skip to content

Kate/bl7012 datamovement #100

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
davramov merged 4 commits into from
Dec 12, 2025
Merged

Kate/bl7012 datamovement #100

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
f7a5d43
updated endpoint ID for data7012 to cosmic phasis share collection id
kkamdin Dec 11, 2025
77c8d89
update the nersc7012 uuid to the correct nersc share collection ID
kkamdin Dec 11, 2025
b44e3af
comment out unused import
kkamdin Dec 11, 2025
08eeef4
updated deployment script
kkamdin Dec 11, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -93,13 +93,13 @@ globus:
nersc7012:
root_path: /global/cfs/cdirs/als/gsharing/data_mover/7012
uri: nersc.gov
uuid: df82346e-9a15-11ea-b3c4-0ae144191ee3
uuid: d40248e6-d874-4f7b-badd-2c06c16f1a58
name: nersc7012

data7012:
root_path: /
uri: hpc.lbl.gov
uuid: 639c49be-604f-423c-9c5d-82a53afe1bf1
uuid: 741b96e1-1b98-42a8-918d-daacc24c145f
name: data7012

globus_apps:
Expand Down
72 changes: 68 additions & 4 deletions create_deployments_7012.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,72 @@
export $(grep -v '^#' .env | xargs)
#!/bin/bash
# Create BL7012 deployment
# Works with both local Prefect server and remote deployments

set -e

echo "Creating BL7012 deployment..."
echo ""

# Load environment variables from .env
if [ ! -f .env ]; then
echo "Error: .env file not found"
echo "Please create a .env file with required variables"
exit 1
fi

export $(grep -v '^#' .env | xargs) 2>/dev/null || true
echo "✓ Loaded .env"

# Use PREFECT_API_URL from .env if set, otherwise default to local
PREFECT_API_URL=${PREFECT_API_URL:-http://127.0.0.1:4200/api}
export PREFECT_API_URL

# Use PREFECT_API_KEY from .env if set (not needed for local server)
if [ -n "$PREFECT_API_KEY" ]; then
export PREFECT_API_KEY
echo "✓ Using API key for authentication"
fi

echo "Target: $PREFECT_API_URL"
echo ""

# Check if server is running
if ! curl -s -f "$PREFECT_API_URL/health" > /dev/null 2>&1; then
echo "Error: Cannot connect to Prefect server at $PREFECT_API_URL"
if [ "$PREFECT_API_URL" = "http://127.0.0.1:4200/api" ]; then
echo "Start local server with: prefect server start"
else
echo "Check that the remote server is accessible"
fi
exit 1
fi

echo "✓ Server is running"
echo ""

# Deploy using modern command
# Work pool: process_newdata7012_ptycho4 (production)
# Work queue: bl7012_ptycho4
prefect deploy \
--name bl7012-process-new-file \
--pool process_newdata7012_ptycho4 \
--work-queue bl7012_ptycho4 \
orchestration/flows/bl7012/move.py:process_new_file

echo ""
echo "✓ Deployment created!"
echo ""
echo "Deployment details:"
echo " Name: bl7012-process-new-file"
echo " Flow: process_newfile_7012_ptycho4"
echo " Work Pool: process_newdata7012_ptycho4"
echo " Work Queue: bl7012_ptycho4"
echo ""

# Old version (deprecated commands - kept for reference):
# prefect deployment build ./orchestration/flows/bl7012/move.py:process_new_file_ptycho4 -n 'process_newdata7012_ptycho4' -q bl7012_ptycho4
# prefect deployment apply process_new_file_ptycho4-deployment.yaml

prefect deployment build ./orchestration/flows/bl7012/move_recon.py:transfer_auto_recon -n 'transfer_auto_recon' -q transfer_auto_recon
prefect deployment apply transfer_auto_recon-deployment.yaml
#
# Note: The flow function is 'process_new_file' but the @flow decorator names it 'process_newfile_7012_ptycho4'
# The function name is what matters for 'prefect deploy'

2 changes: 1 addition & 1 deletion orchestration/flows/bl7012/config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
from globus_sdk import TransferClient

from orchestration.globus import transfer
from ptycho_nersc import NerscPtychoClient
# from ptycho_nersc import NerscPtychoClient

import os

Expand Down