Skip to content

Store forecasts by execution and fuse latest completed inputs - #1383

Draft
damonbayer wants to merge 22 commits into
mainfrom
issue-1315-run-based-storage
Draft

damonbayer wants to merge 22 commits into
mainfrom
issue-1315-run-based-storage

Conversation

@damonbayer

@damonbayer damonbayer commented Nov 4, 2026 •

Copy link
Copy Markdown
Collaborator

Forecast reruns can overwrite earlier fits, and fusion inputs may be produced by separate executions. Store each execution separately and resolve completed inputs across executions so reruns preserve earlier results while downstream work can use the latest available fits.

Execution directories use YYYY-MM-DD_forecasts/<run_timestamp>_<execution_id>/<location>/<disease>/<model>/. Dagster workers share a directory derived from the persisted start time in America/New_York and the run ID; standalone invocations use a UUID. Each execution saves run_config.yaml and execution.json. Fit metadata records training-window settings, and an empty _COMPLETE marker is created only after all outputs are published successfully.

  • Add find_fits(base, keys=..., keep=...) with "latest" (default), "earliest", and "all". Selection follows execution-directory names. Supplying keys limits lookups to those fit paths; single-fit selection stops once every requested key is found. Incomplete fits are ignored.
  • Make the four fusion assets ordinary Dagster assets, independently selectable with their upstream dependencies and automation. Their only configurable setting is FusionConfig.output_basedir; they do not use model resources or inherit upstream location/disease dimensions. Each discovers every location/disease pair with both required completed inputs, selects the latest fit for each input, checks forecast windows and coverage, and records input paths in its metadata. Inputs can come from different executions. Pairs with missing inputs are skipped, and fusion retains overlapping historical trajectories.
  • Remove location-specific output-directory overrides. All model locations write under the model-level output_basedir; location overrides still apply to training settings. Fusion and postprocessing discover inputs under their configured output root.
  • Postprocessing automatically includes every disease with completed, current fits in the forecast-date directory. Remove the disease-selection arguments from the Dagster launchpad and standalone interface. It selects the latest completed fits and excludes fusions whose recorded inputs no longer match the selected upstream fits. It writes collated Hubverse tables and figures to its own execution's 00_summary/<disease>/, with one shared 00_summary/sources.tsv containing location, disease, model, and path columns. The numeric prefix places collated outputs before location folders when browsing by name. A summary can include fits from multiple executions; previously written summaries remain unchanged. Postprocessing does not execute fusion.
  • Update Python/R readers, integration helpers, and documentation; remove unused storage helpers and correct the container workflow's configuration import and Dagster change-detection paths.

Validation:

  • Focused Dagster configuration and fusion/postprocessing tests: 51 passed. Tests actually materialize all four fusion assets without model resources, covering multiple locations and diseases, latest input selection across executions, incomplete newer fits, and missing pairs. Other coverage includes the shared model output root, automatic multi-disease summaries, empty selections, and TSV manifest columns and rows.
  • Verified that an empty selection writes a header-only manifest. Retained demo folders and their manifests use the final 00_summary/sources.tsv layout.
  • Pre-commit checks passed on changed files.
  • Ran two retained executions using synthetic COVID-19 surveillance data and the current checkout's R package. Execution 1 produced CA and US Fable fits and a short CA PyRenew E fit (one chain, 40 warmup iterations, 40 retained draws). Execution 2 reran CA Fable and fused it with execution 1's PyRenew fit. Both executions produced summaries.
  • Verified all five completion markers, all/earliest/latest selection, fusion input provenance, summary table row counts, and collated PDF page counts. The second summary retained US Fable and CA PyRenew from execution 1 while selecting CA Fable and fusion from execution 2.

Closes #1315.
Closes #1300

@codecov

codecov Bot commented Nov 4, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.91667% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.91%. Comparing base (03a06e2) to head (000512d).

Files with missing lines Patch % Lines
.../stf/routine/utils/postprocess_forecast_batches.py 82.85% 6 Missing ⚠️
src/cfa/stf/routine/utils/prop_utils.py 83.87% 5 Missing ⚠️
src/cfa/stf/routine/utils/collate_plots.py 60.00% 4 Missing ⚠️
src/cfa/stf/routine/run_storage.py 97.14% 2 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1383      +/-   ##
==========================================
+ Coverage   82.32%   85.91%   +3.58%     
==========================================
  Files          41       41              
  Lines        1703     1732      +29     
==========================================
+ Hits         1402     1488      +86     
+ Misses        301      244      -57     
Flag Coverage Δ
pipelines 86.02% <92.73%> (+3.65%) ⬆️
stfroutineforecasting 84.76% <100.00%> (+2.74%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@damonbayer damonbayer changed the title Store forecasts by Dagster execution and fuse latest completed fits Store forecasts by execution and fuse latest completed fits Nov 6, 2026
@damonbayer damonbayer changed the title Store forecasts by execution and fuse latest completed fits Store forecasts by execution and fuse latest completed inputs Nov 9, 2026

This branch was successfully deployed

1 active deployment
test — 000512dd Deployed Nov 9, 2026 by damonbayer via Copy image from GHCR to ACR #4197
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Workflow/Directory Structure Revamp Smarter or postponed fusion model assets

1 participant