Aerosol Intro Tutorial and debugging - #970
Merged
boulderdaze merged 6 commits intoJul 20, 2026
Merged
Conversation
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #970 +/- ##
==========================================
+ Coverage 73.46% 73.68% +0.22%
==========================================
Files 131 131
Lines 11531 11307 -224
==========================================
- Hits 8471 8332 -139
+ Misses 3060 2975 -85
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
K20shores
requested changes
Jul 20, 2026
| # | ||
| # Analytical solutions used | ||
| # ───────────────────────── | ||
| # DissolvedReaction foo → bar, rate = k·[S]/([S]+ε)·[foo] |
Collaborator
There was a problem hiding this comment.
I prefer if we remove the unicode characters and replace them with ascii
Collaborator
Author
There was a problem hiding this comment.
I think they should be removed now
boulderdaze
approved these changes
Jul 20, 2026
smeechncar
added a commit
that referenced
this pull request
Jul 20, 2026
Per K20shores' review on #960. After merging origin/main (which added "14. aerosol_chemistry_intro.ipynb" via #970 and shifted 14-17 up to 15-18), this branch's "17. miem_nox_box_model.ipynb" collided with main's renumbered "17. cam_aerosol_distributions.ipynb". Renamed to 19 (next after main's new highest, 18) and fixed the notebook's own stale "part 17" self-reference. docs/source/user_guide/python/notebooks/index.rst was already out of sync with main (missing 14, 17, 18) before this PR -- backfilled those alongside the new 19 entry so the list matches the actual tutorials/ directory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
smeechncar
added a commit
that referenced
this pull request
Jul 21, 2026
* Build miem python bindings Adds musica.miem.Emissions, wrapping musica::EmissionsModel behind pybind11: python/bindings/miem/miem.cpp (_EmissionsModel class with zero-copy numpy surface-flux access via py::capsule, mirroring TUV-x's _run_tuvx pattern rather than MICM's opaque-vector one, since EmissionsModel reassigns its internal state on every Run() call) plus the _create_emissions_from_mechanism factory. Registered as an optional backend (default OFF) mirroring MIAM/CARMA/TUV-x's common.hpp/common.cpp/CMakeLists.txt wiring, with a new backend.miem_available() check. Also fixes a real packaging gap surfaced by this work: python/bindings/CMakeLists.txt never installed libmiem alongside libmusica/libmechanism_configuration in the wheel/editable-install package directory, so the extension couldn't be imported with MIEM enabled until now (invisible previously since MIEM defaults off and #949 never built the Python extension with it on). Adds python/musica/miem/ (the CppWrapper-composed Emissions class), tests under python/test/unit/miem/ covering both a file-based configure-and-run path and an in-code construction path (both synthesize a minimal uptempo-convention NetCDF fixture at runtime, since none exists anywhere in the repo), and a new tutorial (tutorials/17. miem_nox_box_model.ipynb + python/musica/examples/ miem_nox_box_model.py) driving a synthetic NO emission flux into TS1's real, already-validated NOx chemistry in a single well-mixed box. Closes #942 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix invalid DEALLOCATE in TUVX/CARMA GetVersion, fix box model tutorial datetime64 bug While verifying #942's miem_nox_box_model.py tutorial with a working Fortran toolchain (TUVX/CARMA enabled), `import musica` segfaulted under Intel ifx. Root cause (unrelated to #942's own diff, pre-existing in this repo): internal_get_tuvx_version/internal_free_tuvx_version in src/tuvx/interface.F90 (and the identical copy-pasted internal_get_carma_version/ internal_free_carma_version in src/carma/interface.F90) allocated a local Fortran pointer, handed its address to C++ via c_loc, and later reconstructed a Fortran pointer to that same address via c_f_pointer purely to DEALLOCATE it. gfortran tolerates this; Intel's ifx runtime correctly rejects it (forrtl: severe (173): a pointer passed to DEALLOCATE points to an object that cannot be deallocated), since the descriptor rebuilt via c_f_pointer doesn't carry the original allocation's bookkeeping. Fixed by backing both version strings with a module-level SAVE'd allocatable+target buffer that is deallocated by its original name instead of through a reconstructed pointer. Also fixed a real bug in the tutorial script surfaced by actually running it end to end: np.array([t.timestamp() for t in sim_times], dtype="datetime64[s]") fails because numpy's datetime64 conversion doesn't accept raw Python floats; cast to int first. Verified: simulation now runs to completion with no solver-not-converged warnings, and the output shows the expected NOx photochemistry (NO accumulates from miem's constant emission flux, NO2 grows from NO+O3 conversion, O3 is titrated down accordingly). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Add real-fixture variant of the miem NOx box-model tutorial Adds miem_nox_box_model_real_fixture.py, which drives the same TS1 + miem box model as miem_nox_box_model.py but against the real committed CAMS-GLOB-ANT NOx fixture (configs/miem/x1.163842_2024_nox_subset.nc, copied from miem/test/data/) instead of a synthetic single-cell inventory. Exercises the real one-to-many species split (nox_anth_sum -> NO 0.9 + NO2 0.1, matching emissions_model_end_to_end_nox.cpp) and linear temporal interpolation across the fixture's 12 real monthly snapshots, using the highest-emitting cell (1602) as the box location. Verified end-to-end: NO/NO2 flux splits exactly 9:1 at the real July 2024 magnitude, and the resulting NOx-titrates-O3 chemistry response matches the synthetic-inventory version's expected shape. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Enable MUSICA_ENABLE_MIEM in pip build so miem bindings actually compile pyproject.toml set MUSICA_ENABLE_MIAM=ON for scikit-build-core but never set MUSICA_ENABLE_MIEM (which defaults OFF), so `pip install .` never compiled python/bindings/miem/miem.cpp, leaving musica._musica._miem._create_emissions_from_mechanism missing at runtime. This is what broke the run-notebooks CI job on tutorial 17. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix datetime64 ValueError in miem_nox_box_model tutorial notebook Same bug as 5903caa fixed in the .py example, but that commit missed the notebook's copy of the same line: np.array([t.timestamp() for t in sim_times], dtype="datetime64[s]") raises "Could not convert object to NumPy datetime" because numpy won't convert raw Python floats to datetime64[s] directly. Cast to int first, as already done in the .py version. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Enable MUSICA_ENABLE_MIEM in wheel-build prebuild scripts 5e47ec4 set MUSICA_ENABLE_MIEM=ON in pyproject.toml so the Python extension itself compiles python/bindings/miem/miem.cpp, but missed that the wheel workflow first builds a separate "prebuilt" musica core library (python/tools/prepare_build_environment_{linux,macos,windows}.sh, installed to /opt or /tmp musica-prebuilt) which the extension then links/includes against. That prebuild step never had MUSICA_ENABLE_MIEM set, so its installed headers never included miem/source_types.hpp, and every wheel build failed with: musica/configuration/emissions.hpp:7:10: fatal error: miem/source_types.hpp: No such file or directory Added -DMUSICA_ENABLE_MIEM=ON next to the existing -DMUSICA_ENABLE_MIAM=ON in all three platform prebuild scripts. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix Windows wheel build: hint miem's netCDF discovery at MSYS2's prefix Both Windows wheel jobs (windows-latest AMD64, windows-11-arm ARM64) failed after enabling MUSICA_ENABLE_MIEM: miem's own cmake/FindnetCDF.cmake (unlike musica's own dependencies.cmake, which uses pkg-config) only looks for an nc-config script or an NETCDF_DIR/NETCDF_ROOT-hinted install. With neither set, it can't see the pacman-installed MSYS2 netCDF package (even though musica's own pkg-config-based lookup finds it fine moments earlier in the same configure run) and falls back to FetchContent building netcdf-c v4.9.2 from source -- which doesn't compile under MinGW-UCRT: dpathmgr.c's `_wstat64(wpath, buf)` call passes a `struct stat *` where `struct _stat64 *` is expected. Exporting NETCDF_ROOT to MSYS2's install prefix ($MINGW_PREFIX, e.g. /ucrt64 or /clangarm64 on the ARM64 runner) before configuring lets miem's FindnetCDF.cmake locate the same pacman-installed netCDF musica itself already uses, avoiding the from-source fallback entirely. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Point MIEM_GIT_TAG at NCAR/miem#22 to test Windows netCDF discovery fix Temporary: pins to the fix branch commit (not yet merged) so this PR's own CI validates the fix before NCAR/miem#22 lands on miem's main. Once that PR merges, bump this back to a main commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix Windows YAML parse failure in miem file-based emissions test Embedding a raw Windows tmp_path (backslashes) inside a double-quoted YAML string made yaml-cpp choke on sequences like "\msys64" as an invalid escape character. Use as_posix() so the embedded path uses forward slashes, which both YAML and Windows file APIs accept. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Bound wheel-build job timeout and add pytest watchdog for CI hang diagnosis The windows-11-arm wheel build has hung for 6h (job timeout) three times in a row on this branch, with the pytest process still alive but silent after printing its final summary. Cap the job at 90 min so a recurrence fails fast, and route pytest through a faulthandler watchdog that dumps every live thread's stack before force-exiting if it hangs again. * Route Windows wheel test-command through the pytest watchdog The windows.test-command override in pyproject.toml has its own pytest invocation (for its Windows-specific -k filter) that bypassed the shared run_pytest_with_watchdog.py wrapper added in 22df660 — so PR #960's watchdog mitigation never actually fired on Windows. Both windows-latest and windows-11-arm wheel builds hung silently after pytest printed its summary, timing out at the 90 min cap with no diagnostic dump. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Skip interpreter shutdown in the Windows test watchdog pytest already passes and no non-main Python threads remain before the hang; the stall is inside CPython's own finalization (extension DLL unload / native static destructors) after faulthandler has already been torn down, so the watchdog can never observe or diagnose it. os._exit() bypasses that teardown path entirely since a clean shutdown isn't needed once results are already in. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Use TerminateProcess on Windows to skip DLL_PROCESS_DETACH teardown os._exit() (6be9fb8) did not fix the Windows wheel-build hang: the CRT's _exit() still routes through ExitProcess(), which still delivers DLL_PROCESS_DETACH to every loaded DLL before the process dies -- same hang, same spot, confirmed by a second identical 90-minute timeout on both windows-11-arm and windows-latest AMD64. TerminateProcess() is the only Win32 call that skips DLL_PROCESS_DETACH entirely, so call it directly via ctypes on Windows once pytest has already passed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Skip miem tests on Windows wheel builds; revert teardown hack The Windows wheel-build hang isn't a musica/miem teardown bug: netCDF's bundled AWS SDK never closes its thread on process shutdown, hanging the whole process once real netCDF I/O runs (only miem's tests do this). Root-caused by K20shores, who has a fix upstream at Unidata/netcdf-c#3400. Skip miem's tests on Windows for now, same pattern already used for tuvx. With that in place, the TerminateProcess()/os._exit() exit-mechanic hack added while chasing this as a teardown bug is no longer needed; reverted run_pytest_with_watchdog.py to a plain sys.exit(), keeping just the faulthandler watchdog for future hang diagnostics. * Re-pin MIEM_GIT_TAG to merged NCAR/miem#22 miem#22 (Windows/MSYS2 netCDF discovery fix) merged to main as 2bb1e21. Re-pin from the unmerged 59cde48c to the merged commit. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Revert TUV-x/CARMA interface.F90 pointer-lifecycle fix Per K20shores' review on #960: this PR should stay scoped to miem. The ifx DEALLOCATE fix (5903caa) is a real, pre-existing bug unrelated to miem -- reverting here and will track it in a separate issue. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix per-cell concentration update, remove watchdog script Per K20shores' review on #960: - current_concentrations[name] = [current_concentrations[name][0] + delta] collapsed the concentration array down to a single element instead of adding delta to every cell. Fixed in both miem_nox_box_model.py and miem_nox_box_model_real_fixture.py with a list comprehension over the existing array (harmless no-op today since both scripts use num_grid_cells=1, but correct regardless of cell count). - Removed python/tools/run_pytest_with_watchdog.py and its two pyproject.toml test-command references. It was built to diagnose a Windows wheel-build hang that's since been root-caused as an upstream netCDF/AWS-SDK issue (Unidata/netcdf-c#3400), unrelated to the faulthandler-based approach this script took. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Renumber miem tutorial to 19, update tutorials docs list Per K20shores' review on #960. After merging origin/main (which added "14. aerosol_chemistry_intro.ipynb" via #970 and shifted 14-17 up to 15-18), this branch's "17. miem_nox_box_model.ipynb" collided with main's renumbered "17. cam_aerosol_distributions.ipynb". Renamed to 19 (next after main's new highest, 18) and fixed the notebook's own stale "part 17" self-reference. docs/source/user_guide/python/notebooks/index.rst was already out of sync with main (missing 14, 17, 18) before this PR -- backfilled those alongside the new 19 entry so the list matches the actual tutorials/ directory. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Fix garbled section headers in miem tutorial notebook Splitting a heading and its following paragraph into separate markdown cells avoids an nbsphinx/pandoc RST-conversion bug that corrupted "4. Photolysis Rates from TUV-x", "5. Initial Conditions", and "9. Plot Results" into literal :nbsphinx-math: text in the built docs. Verified via a local sphinx+nbsphinx rebuild: zero RST warnings after the split, versus real parser errors before. * Drive NOx emissions via MICM Emission reactions instead of manual concentration bumps Per K20shores' review on #960: replace the per-step "convert flux to a concentration delta, mutate state, then solve" pattern with native MICM Emission reactions for NO (and NO2 in the real-fixture demo), driven each step by miem's computed flux set as an EMIS.<species> rate parameter. This lets the solver integrate emission and chemistry together in one ODE step instead of as two sequential operations. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> * Update miem tutorial notebook wording and sync with Emission-reaction change Tightens the notebook's prose and updates its code cells to match the NO Emission reaction / EMIS.NO rate-parameter approach used in miem_nox_box_model.py, so the tutorial reflects the same emissions handling as the example script. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds an "Intro to Aerosol Modeling" tutorial to walk through the conceptual approach to multi-phase modeling used in MUSICA. It also includes some light clean up of the other aerosol and cloud related notebooks and the minor updates below.
Minor Updates
*rate_constantsas*rate_constantwhen only a single rate constant is expected.PhaseSpeciesto be created based on an existingSpeciesinstead of only by a name string