Skip to content

allow compose dir override in postgres update script - #784

Open
fmigneault wants to merge 2 commits into
masterfrom
update-postres-compose-dir
Open

fmigneault wants to merge 2 commits into
masterfrom
update-postres-compose-dir

Conversation

@fmigneault

Copy link
Copy Markdown
Member

Overview

Improve postgres update.

Changes

Non-breaking changes

  • Postgres Update Script: Employ the common COMPOSE_DIR approach to allow custom overrides.

Breaking changes

  • n/a

CI Operations

birdhouse_daccs_configs_branch: master
birdhouse_skip_ci: false

@fmigneault fmigneault self-assigned this Nov 20, 2026
@fmigneault
fmigneault enabled auto-merge November 20, 2026 00:40
@github-actions github-actions Bot added documentation Improvements or additions to documentation ci/deployment Related to deployment utilities and scripts labels Nov 20, 2026
@crim-jenkins-bot

Copy link
Copy Markdown
Collaborator

E2E Test Results

DACCS-iac Pipeline Results

Build URL : http://daccs-jenkins.crim.ca:80/job/DACCS-iac-birdhouse/4608/
Result ❌ FAILURE

BIRDHOUSE_DEPLOY_BRANCH : update-postres-compose-dir
DACCS_IAC_BRANCH : master
DACCS_CONFIGS_BRANCH : master
PAVICS_E2E_WORKFLOW_TESTS_BRANCH : master
PAVICS_SDI_BRANCH : master

DESTROY_INFRA_ON_EXIT : true
PAVICS_HOST : https://host-140-216.rdext.crim.ca

PAVICS-e2e-workflow-tests Pipeline Results

Tests URL : http://daccs-jenkins.crim.ca:80/job/PAVICS-e2e-workflow-tests/job/master/1157/

NOTEBOOK TEST RESULTS
    
[2026-11-20T02:29:37.557Z] ============================= test session starts ==============================
[2026-11-20T02:29:37.557Z] platform linux -- Python 3.12.13, pytest-9.0.3, pluggy-1.6.0
[2026-11-20T02:29:37.557Z] rootdir: /home/jenkins/agent/workspace/PAVICS-e2e-workflow-tests_master
[2026-11-20T02:29:37.557Z] plugins: xdist-3.8.0, anyio-4.13.0, nbval-0.11.0, tornasync-0.6.0.post2, typeguard-4.5.1
[2026-11-20T02:29:37.557Z] collected 618 items
[2026-11-20T02:29:37.557Z] 
[2026-11-20T02:29:44.932Z] notebooks-auth/geoserver.ipynb ..................                        [  2%]
[2026-11-20T02:30:35.820Z] notebooks-auth/test_cowbird_jupyter.ipynb ..........                     [  4%]
[2026-11-20T02:30:41.691Z] notebooks-auth/test_thredds.ipynb F..........                            [  6%]
[2026-11-20T02:32:18.969Z] pavics-sdi-master/docs/source/notebooks/CaSR_basic.ipynb ......          [  7%]
[2026-11-20T02:45:24.386Z] pavics-sdi-master/docs/source/notebooks/FAQ_dask_parallel.ipynb ..s..... [  8%]
[2026-11-20T02:46:11.098Z] .                                                                        [  8%]
[2026-11-20T02:46:26.606Z] pavics-sdi-master/docs/source/notebooks/WCS_example.ipynb ......         [  9%]
[2026-11-20T02:46:33.393Z] pavics-sdi-master/docs/source/notebooks/WFS_example.ipynb .....          [ 10%]
[2026-11-20T02:56:24.990Z] pavics-sdi-master/docs/source/notebooks/climex.ipynb ...........         [ 12%]
[2026-11-20T02:56:24.990Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-climate-stations.ipynb . [ 12%]
[2026-11-20T02:56:28.211Z] ...............                                                          [ 14%]
[2026-11-20T02:56:39.452Z] pavics-sdi-master/docs/source/notebooks/eccc-geoapi-xclim.ipynb .....    [ 15%]
[2026-11-20T02:56:54.195Z] pavics-sdi-master/docs/source/notebooks/esgf-dap.ipynb ....              [ 16%]
[2026-11-20T02:57:11.812Z] pavics-sdi-master/docs/source/notebooks/forecasts.ipynb ......           [ 17%]
[2026-11-20T02:57:17.904Z] pavics-sdi-master/docs/source/notebooks/opendap.ipynb .......            [ 18%]
[2026-11-20T02:57:23.242Z] pavics-sdi-master/docs/source/notebooks/pavics_thredds.ipynb .....       [ 19%]
[2026-11-20T03:00:07.683Z] pavics-sdi-master/docs/source/notebooks/regridding.ipynb ............... [ 21%]
[2026-11-20T03:01:19.892Z] .............                                                            [ 23%]
[2026-11-20T03:01:25.500Z] pavics-sdi-master/docs/source/notebooks/rendering.ipynb ....             [ 24%]
[2026-11-20T03:01:27.404Z] pavics-sdi-master/docs/source/notebooks/subset-user-input.ipynb ........ [ 25%]
[2026-11-20T03:01:56.152Z] .................                                                        [ 28%]
[2026-11-20T03:02:05.931Z] pavics-sdi-master/docs/source/notebooks/subsetting.ipynb .....           [ 29%]
[2026-11-20T03:02:07.390Z] pavics-sdi-master/docs/source/notebook-components/weaver_example.ipynb . [ 29%]
[2026-11-20T03:02:25.716Z] ..F......                                                                [ 30%]
[2026-11-20T03:02:37.110Z] finch-main/docs/source/notebooks/dap_subset.ipynb ...........            [ 32%]
[2026-11-20T03:02:48.644Z] finch-main/docs/source/notebooks/finch-usage.ipynb ......                [ 33%]
[2026-11-20T03:02:50.027Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-1DataAccess.ipynb . [ 33%]
[2026-11-20T03:02:55.604Z] ....                                                                     [ 34%]
[2026-11-20T03:03:34.332Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-2Subsetting.ipynb . [ 34%]
[2026-11-20T03:05:11.506Z] ............                                                             [ 36%]
[2026-11-20T03:05:23.702Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-3Climate-Indicators.ipynb . [ 36%]
[2026-11-20T03:06:06.542Z] .....s                                                                   [ 37%]
[2026-11-20T03:06:14.669Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-4Ensembles.ipynb . [ 37%]
[2026-11-20T03:06:22.489Z] ..                                                                       [ 38%]
[2026-11-20T03:06:37.386Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-5Visualization.ipynb . [ 38%]
[2026-11-20T03:07:40.073Z] .........                                                                [ 39%]
[2026-11-20T03:07:52.283Z] PAVICS-landing-master/content/notebooks/climate_indicators/PAVICStutorial_ClimateDataAnalysis-6Regridding_Conversion.ipynb . [ 39%]
[2026-11-20T03:09:33.833Z] ...                                                                      [ 40%]
[2026-11-20T03:09:33.833Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-01_Intro.ipynb . [ 40%]
[2026-11-20T03:09:39.532Z] ....                                                                     [ 41%]
[2026-11-20T03:09:44.865Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-02_Calibration.ipynb . [ 41%]
[2026-11-20T03:09:55.881Z] .....                                                                    [ 42%]
[2026-11-20T03:10:00.074Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-03_Watershed_properties.ipynb . [ 42%]
[2026-11-20T03:10:10.604Z] .............                                                            [ 44%]
[2026-11-20T03:10:17.176Z] PAVICS-landing-master/content/notebooks/hydrology/PAVICStutorial_Hydrology-04_Time_series_analysis.ipynb . [ 44%]
[2026-11-20T03:10:18.529Z] ......                                                                   [ 45%]
[2026-11-20T03:10:30.931Z] raven-main/docs/source/notebooks/Region_selection.ipynb .........        [ 47%]
[2026-11-20T03:10:31.886Z] raven-main/docs/source/notebooks/Subset_climate_data_over_watershed.ipynb . [ 47%]
[2026-11-20T03:10:54.601Z] ......                                                                   [ 48%]
[2026-11-20T03:10:57.889Z] RavenPy-main/docs/notebooks/00_Introduction_to_JupyterLab.ipynb ......   [ 49%]
[2026-11-20T03:11:08.472Z] RavenPy-main/docs/notebooks/01_Getting_watershed_boundaries.ipynb ...... [ 50%]
[2026-11-20T03:11:08.472Z] ..                                                                       [ 50%]
[2026-11-20T03:11:16.593Z] RavenPy-main/docs/notebooks/02_Extract_geographical_watershed_properties.ipynb . [ 50%]
[2026-11-20T03:11:20.713Z] .............                                                            [ 52%]
[2026-11-20T03:12:26.197Z] RavenPy-main/docs/notebooks/03_Extracting_forcing_data.ipynb ........... [ 54%]
[2026-11-20T03:12:26.197Z]                                                                          [ 54%]
[2026-11-20T03:12:33.864Z] RavenPy-main/docs/notebooks/04_Emulating_hydrological_models.ipynb ..... [ 55%]
[2026-11-20T03:12:44.289Z] ...............                                                          [ 57%]
[2026-11-20T03:12:50.854Z] RavenPy-main/docs/notebooks/05_Advanced_RavenPy_configuration.ipynb .... [ 58%]
[2026-11-20T03:12:59.122Z] .........                                                                [ 59%]
[2026-11-20T03:13:13.082Z] RavenPy-main/docs/notebooks/06_Raven_calibration.ipynb ......            [ 60%]
[2026-11-20T03:13:21.775Z] RavenPy-main/docs/notebooks/07_Making_and_using_hotstart_files.ipynb ... [ 61%]
[2026-11-20T03:13:24.339Z] ...                                                                      [ 61%]
[2026-11-20T03:13:32.460Z] RavenPy-main/docs/notebooks/08_Getting_and_bias_correcting_CMIP6_data.ipynb . [ 61%]
[2026-11-20T03:14:24.534Z] ............sss                                                          [ 64%]
[2026-11-20T03:14:32.724Z] RavenPy-main/docs/notebooks/09_Hydrological_impacts_of_climate_change.ipynb . [ 64%]
[2026-11-20T03:14:38.563Z] ....                                                                     [ 65%]
[2026-11-20T03:15:22.121Z] RavenPy-main/docs/notebooks/10_Data_assimilation.ipynb ........          [ 66%]
[2026-11-20T03:15:33.806Z] RavenPy-main/docs/notebooks/11_Climatological_ESP_forecasting.ipynb .... [ 67%]
[2026-11-20T03:16:01.889Z] ....                                                                     [ 67%]
[2026-11-20T03:16:11.867Z] RavenPy-main/docs/notebooks/12_Performing_hindcasting_experiments.ipynb . [ 67%]
[2026-11-20T03:16:21.968Z] .......                                                                  [ 69%]
[2026-11-20T03:16:51.518Z] RavenPy-main/docs/notebooks/Assess_probabilistic_flood_risk.ipynb ...... [ 70%]
[2026-11-20T03:16:51.518Z] .                                                                        [ 70%]
[2026-11-20T03:17:01.509Z] RavenPy-main/docs/notebooks/Comparing_hindcasts_and_ESP_forecasts.ipynb . [ 70%]
[2026-11-20T03:17:22.539Z] .......                                                                  [ 71%]
[2026-11-20T03:17:31.192Z] RavenPy-main/docs/notebooks/Distributed_hydrological_modelling.ipynb ... [ 72%]
[2026-11-20T03:17:50.235Z] ....                                                                     [ 72%]
[2026-11-20T03:18:04.390Z] RavenPy-main/docs/notebooks/Hydrological_realtime_forecasting.ipynb .... [ 73%]
[2026-11-20T03:18:10.863Z] ..                                                                       [ 73%]
[2026-11-20T03:18:20.941Z] RavenPy-main/docs/notebooks/Managing_Jupyter_Environments.ipynb ...      [ 74%]
[2026-11-20T03:18:55.613Z] RavenPy-main/docs/notebooks/Perform_Regionalization.ipynb .......        [ 75%]
[2026-11-20T03:18:59.889Z] RavenPy-main/docs/notebooks/Running_HMETS_with_CANOPEX_dataset.ipynb ... [ 75%]
[2026-11-20T03:19:14.767Z] ..........                                                               [ 77%]
[2026-11-20T03:19:37.895Z] RavenPy-main/docs/notebooks/Sensitivity_analysis.ipynb ......            [ 78%]
[2026-11-20T03:19:46.409Z] RavenPy-main/docs/notebooks/time_series_analysis.ipynb ...........       [ 80%]
[2026-11-20T03:19:54.527Z] RavenPy-main/docs/notebooks/paper/Perform_a_climate_change_impact_study_on_a_watershed.ipynb . [ 80%]
[2026-11-20T03:24:18.259Z] .............Fxxxxxx                                                     [ 83%]
[2026-11-20T03:24:24.625Z] xhydro-main/docs/notebooks/pavics_notebooks/climate_change.ipynb ....... [ 84%]
[2026-11-20T03:24:50.465Z] .........................                                                [ 88%]
[2026-11-20T03:25:02.675Z] xhydro-main/docs/notebooks/pavics_notebooks/hydrological_modelling_raven_distributed.ipynb . [ 88%]
[2026-11-20T03:25:16.756Z] ..................FFFFFFFF                                               [ 93%]
[2026-11-20T03:26:23.819Z] xhydro-main/docs/notebooks/pavics_notebooks/pmp.ipynb .................. [ 95%]
[2026-11-20T03:26:53.644Z] ..........                                                               [ 97%]
[2026-11-20T03:26:55.842Z] notebooks/hummingbird.ipynb ..........                                   [ 99%]
[2026-11-20T03:30:14.954Z] notebooks/stress-tests.ipynb .....                                       [100%]
[2026-11-20T03:30:14.954Z] 
[2026-11-20T03:30:14.954Z] =================================== FAILURES ===================================
    
  

@mishaschwartz mishaschwartz left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't really understand what this is solving.

Especially since BIRDHOUSE_EXE is defined relative to COMPOSE_DIR, if you set COMPOSE_DIR to something unexpected then the whole script will fail since it won't be able to find the right script.

Employ the common COMPOSE_DIR approach to allow custom overrides.

What custom overrides? Do you mean to be setting BIRDHOUSE_LOCAL_ENV instead?

@fmigneault

Copy link
Copy Markdown
Member Author

We override COMPOSE_DIR initially because our env.local is symlinked to an external repository, and it incorrectly resolves THIS_DIR when following those links, depending how the scrips are invoked (via bin/birdhouse configs -c <script>, bash <script, . <script> or ./<script>).
All ${BIRDHOUSE_EXE} references within this script (and the ones nested within sub-invokes when resolved) were leading to "command not found" inconsistently, so steps were half-accomplished and corrupting results.

For the common custom override portion, what I meant is that it aligns with how other scripts define the references, eg:

THIS_FILE="$(readlink -f "$0" || realpath "$0")"
THIS_DIR="$(dirname "${THIS_FILE}")"
COMPOSE_DIR="${COMPOSE_DIR:-$(dirname "${THIS_DIR}")}"

This script was skipping over the COMPOSE_DIR resolution, so it produces inconsistent results between their invocations.

Obviously, it assumes that if one overrides COMPOSE_DIR, they do it with something sensible.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ci/deployment Related to deployment utilities and scripts documentation Improvements or additions to documentation

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants