CMORgasbord: yaml parsing #368

ilaflott · 2025-02-25T19:46:10Z

Background

PR #363 was/is the pre-amble to this PR. This PR seeks to close issue #320, and make general improvements related to that as-needed.

Describe your changes

expanding this as i go. Currently we have:

expands fre cmor yaml call input arguments
takes the widely used join_constructor and puts it in fre/yamltools/constructors.py, to assist with avoiding circular import issues i encountered in PR 320.cmor yaml parsing #363, remove duplication of that code elsewhere
takes other widely used routines (output_yaml, yaml_load) and puts them in fre/yamltools/helpers.py, removed duplicates where applicable
centralize the way the yaml module gets imported and gets the !join constructor' inside fre/yamltools/__init__.py.
make everything that needs the yaml module with join_constructor import it via yamltools/__init__.py
leave commented-out logging efforts in fre/list_, since, in that case, we kind of WANT the stdout... so should ponder an intelligent work around and update the unittests for fre list in near future
DEBUG_RUN_ONE_MODE now a proper click flag option for fre cmor run/yaml calls, which should close CMORgasbord: make current DEBUG_MODE_RUN_ONE flag an optional arg to fre cmor run #324 ! called run_one_mode in CLI calls, and run_one within fre/cmor internals
function calls now "do the right thing" with that flag
relevant unit-test calls adjusted to use the new run_one_mode flag
new unit test, fre/tests/test_fre_yamltools_cli.py::test_cli_fre_yamltools_combine_cmoryaml
new unit test, fre/yamltools/tests/test_combine_yamls_script.py::test_combine_cmor_yaml
new test-data file, fre/yamltools/tests/AM5_example/cmor_yamls/cmor.am5.yaml
new test-data file for comparison to aforementioned unittests, fre/yamltools/tests/AM5_example/COMPARE_TEST_OUTPUT_cmor.yaml
new "info parser" holding a CMORYaml class definition, fre/yamltools/cmor_info_parser.py, which get leveraged for e.g. combining an un-parsed model yaml (AM5_example/am5.yaml) with a model-specific cmor-tool-yaml (AM5_example/cmor_yamls/cmor.am5.yaml)

Issue ticket number and link (if applicable)

#320 and via sidequest, #324

Checklist before requesting a review

I ran my code
I tried to make my code readable
I tried to comment my code
I wrote a new test, if applicable
I wrote new instructions/documentation, if applicable
I ran pytest and inspected it's output
I ran pylint and attempted to implement some of it's feedback
No print statements; all user-facing info uses logging module
- caveat here is fre/list_ things i touched: need to consider how we handle fre list stdout

return statement included before calling the run subtool- pprint out to screen instead

forgot the yamltools bit

turn on run-one-debug mode at top of testing script

fix minor syntax error causing all the failures

fix the import statements.

fix import statement

…ust import statements when join_constructor pops up

…verything in yamltools can just from . import * it. remove the (yaml)constructor arguments to yaml class constructors, remove the add_constructor bit from initializing since we get it by importing the yamltools directory now. remove dependence on combine_yamls when things like load_yaml or yaml_output are called- put these in helper and adjust statements everywhere where required. make attempt at sticking fre_logger into fre list calls- but actually this may need some thought about what we want since we are parsing std out for the unit tests...

ilaflott · 2025-02-26T18:53:18Z

OK! at this point, i did some cleaning up and refactoring to find my way better around yamltools as necessary for fre cmor yaml efforts. and I did so preserving the test outcomes- yay!

now, lets get back to the real work where PR #363 (R.I.P.) was getting interesting, which was the following call

ilaflott · 2025-02-26T18:56:35Z

...and that call was...

> fre -v yamltools combine-yamls -y fre/yamltools/tests/AM5_example/am5.yaml -e c96L65_am5f7b12r1_amip -p ncrc5.intel -t prod-openmp --use cmor --output FOO_cmor.yaml

... which produces

INFO:freyamltools.py:combine_yamls calling fre/yamltools/combine_yamls_script.py::consolidate_yamls
INFO:combine_yamls_script.py:consolidate_yamls combining+parsing yaml files, for cmor usage
INFO:combine_yamls_script.py:consolidate_yamls about to initialize the CmorYaml
INFO:cmor_info_parser.py:__init__ initializing a CMORYaml object
INFO:cmor_info_parser.py:__init__ CMORYaml initialized!
INFO:combine_yamls_script.py:consolidate_yamls 
    DONE initializing CmorYaml = 
    InitCMORYaml( 
                               yml = fre/yamltools/tests/AM5_example/am5.yaml 
                               name = c96L65_am5f7b12r1_amip 
                               platform = ncrc5.intel 
                               target = prod-openmp 
                               mainyaml_dir = fre/yamltools/tests/AM5_example

INFO:combine_yamls_script.py:consolidate_yamls will be writing out the combined yaml dictionary!
INFO:combine_yamls_script.py:get_combined_cmoryaml calling CmorYaml.combine_model() for yaml_content and loaded_yaml
INFO:cmor_info_parser.py:combine_model    model yaml: fre/yamltools/tests/AM5_example/am5.yaml
INFO:cmor_info_parser.py:combine_experiment ey_path = fre/yamltools/tests/AM5_example/cmor_yamls/cmor.am5.yaml
INFO:cmor_info_parser.py:merge_cmor_yaml    experiment yaml: cmor.am5.yaml
{ 'build': { 'compileYaml': 'compile_yamls/compile.yaml',
             'platformYaml': 'compile_yamls/platforms.yaml'},
  'cmor': { 'directories': { 'analysis_dir': '/nbhome/$USER/am5/am5f7b12r1/c96L65_am5f7b12r1_amip',
                             'exp_config_dir': 'fre/tests/test_files',
                             'fre_analysis_home': '/home/fms/local/opt/fre-analysis/test',
                             'history_dir': '/archive/$USER/am5/am5f7b12r1/c96L65_am5f7b12r1_amip/ncrc5.intel-prod-openmp/history',
                             'pp_dir': '/archive/$USER/am5/am5f7b12r1/c96L65_am5f7b12r1_amip/ncrc5.intel-prod-openmp/pp',
                             'ptmp_dir': '/xtmp/$USER/ptmp',
                             'table_dir': 'fre/tests/test_files/cmip6-cmor-tables/Tables'},
            'targets': { 'Omon-ts-gr1': { 'indir': 'fre/tests/test_files/ocean_sos_var_file',
                                          'json_exp_config': 'fre/tests/test_files/CMOR_input_example.json',
                                          'json_table_config': 'fre/tests/test_files/cmip6-cmor-tables/Tables/CMIP6_Omon.json',
                                          'json_var_list': 'fre/tests/test_files/varlist',
                                          'opt_var_name': None,
                                          'outdir': 'fre/tests/test_files/outdir'}}},
  'name': 'c96L65_am5f7b12r1_amip',
  'platform': 'ncrc5.intel',
  'target': 'prod-openmp'}

but more importantly, the contents of that yaml were, via cat FOO_cmor.yaml:

name: c96L65_am5f7b12r1_amip
platform: ncrc5.intel
target: prod-openmp
build:
  compileYaml: compile_yamls/compile.yaml
  platformYaml: compile_yamls/platforms.yaml
cmor:
  directories:
    history_dir: /archive/$USER/am5/am5f7b12r1/c96L65_am5f7b12r1_amip/ncrc5.intel-prod-openmp/history
    pp_dir: /archive/$USER/am5/am5f7b12r1/c96L65_am5f7b12r1_amip/ncrc5.intel-prod-openmp/pp
    analysis_dir: /nbhome/$USER/am5/am5f7b12r1/c96L65_am5f7b12r1_amip
    ptmp_dir: /xtmp/$USER/ptmp
    fre_analysis_home: /home/fms/local/opt/fre-analysis/test
    table_dir: fre/tests/test_files/cmip6-cmor-tables/Tables
    exp_config_dir: fre/tests/test_files
  targets:
    Omon-ts-gr1:
      json_table_config: fre/tests/test_files/cmip6-cmor-tables/Tables/CMIP6_Omon.json
      json_exp_config: fre/tests/test_files/CMOR_input_example.json
      indir: fre/tests/test_files/ocean_sos_var_file
      json_var_list: fre/tests/test_files/varlist
      outdir: fre/tests/test_files/outdir
      opt_var_name: null

so if we can get back here, preserving current testing results, we can definitely move forward with the approach

…flag, default false. add the debug run one mode to the yaml processing calls as well.

…his red x into a green check. will adjust testing/config file needs as we go along to ensure we are writing sensible things

…regious assumption tha the gfdl name is going to be the mip name. i know this is not the case- just a prototype for possibly interfacing with slurm.

…what worked last time!

…combined yaml file

ilaflott · 2025-02-27T20:05:57Z

yay!

(fre-cli) an210: fre-cli $] fre -v yamltools combine-yamls -y fre/yamltools/tests/AM5_example/am5.yaml -e c96L65_am5f7b12r1_amip -p ncrc5.intel -t prod-openmp --use cmor --output FOO_cmor.yaml
INFO:cmor_info_parser.py:__init__ initializing a CMORYaml object
INFO:cmor_info_parser.py:__init__ CMORYaml initialized!
INFO:combine_yamls_script.py:get_combined_cmoryaml calling CMORYaml.combine_model() for yaml_content and loaded_yaml
INFO:cmor_info_parser.py:combine_model    model yaml: fre/yamltools/tests/AM5_example/am5.yaml
INFO:cmor_info_parser.py:combine_experiment ey_path = fre/yamltools/tests/AM5_example/cmor_yamls/cmor.am5.yaml
INFO:cmor_info_parser.py:merge_cmor_yaml    experiment yaml: cmor.am5.yaml
INFO:combine_yamls_script.py:get_combined_cmoryaml Combined cmor-yaml information cleaned+saved as dictionary
INFO:combine_yamls_script.py:get_combined_cmoryaml Combined cmor-yaml information saved to FOO_cmor.yaml
(fre-cli) an210: fre-cli $] cat FOO_cmor.yaml 
name: c96L65_am5f7b12r1_amip
platform: ncrc5.intel
target: prod-openmp
build:
  compileYaml: compile_yamls/compile.yaml
  platformYaml: compile_yamls/platforms.yaml
cmor:
  directories:
    history_dir: /archive/$USER/am5/am5f7b12r1/c96L65_am5f7b12r1_amip/ncrc5.intel-prod-openmp/history
    pp_dir: /archive/$USER/am5/am5f7b12r1/c96L65_am5f7b12r1_amip/ncrc5.intel-prod-openmp/pp
    analysis_dir: /nbhome/$USER/am5/am5f7b12r1/c96L65_am5f7b12r1_amip
    ptmp_dir: /xtmp/$USER/ptmp
    fre_analysis_home: /home/fms/local/opt/fre-analysis/test
    table_dir: fre/tests/test_files/cmip6-cmor-tables/Tables
    exp_config_dir: fre/tests/test_files
  targets:
    Omon-ts-gr1:
      json_table_config: fre/tests/test_files/cmip6-cmor-tables/Tables/CMIP6_Omon.json
      json_exp_config: fre/tests/test_files/CMOR_input_example.json
      indir: fre/tests/test_files/ocean_sos_var_file
      json_var_list: fre/tests/test_files/varlist
      outdir: fre/tests/test_files/outdir
      opt_var_name: null

…e unit tests into existence and add in test case ive been running based on the am5 yaml etc. the contents of the yaml too must be checked

…ine-yaml test call add comparison conditions for lloading the fresh created output file and comparing to static test file data

…ot to test for the combination doing its job. test the combine_yamls as a python module call, and if the output combined yaml exists already, remove it before recreating

…statements in regrid app to logging.info statements

…r regridding

chanwilson

Nothing showstopper just avoiding technical debt

fre/app/regrid_xy/regrid_xy.py

fre/cmor/quickly_make_var_list.sh

fre/tests/test_fre_yamltools_cli.py

fre/yamltools/__init__.py

fre/yamltools/compile_info_parser.py

fre/yamltools/tests/test_combine_yamls_script.py

…eady for pushing yet. address chan comments RE debug info and logging calls

…test for yamls

ceblanton

Looks great! @singhd789 there are a few yamls-related touches in here that you're probably already aware of (helpers.py)

singhd789 · 2025-03-03T15:41:20Z

fre/yamltools/cmor_info_parser.py

+from . import *
+
+
+def experiment_check(mainyaml_dir, experiment, loaded_yaml):


There seem to be some shared functions in here and in pp_info_parser.py. These could probably eventually be cleaned and put in one script to use (like experiment_check, combine_model, combine_experiment) - maybe in an abstract class you mentioned to me one time. This could probably be a clean-up in the aftermath of CMIP tasks though

Ian told me to dismiss this, keep momentum going!!

ilaflott · 2025-03-03T16:26:53Z

I told @singhd789 to dismiss @chanwilson to keep the stuff moving!

init commit, smth small, to start draft PR

ea62768

ilaflott linked an issue Feb 25, 2025 that may be closed by this pull request

CMORgasbord: combine model.yaml fields with cmor.yaml configuration file for fre cmor yaml #320

Closed

ilaflott changed the title ~~init commit, smth small, to start draft PR~~ cmor yaml parsing Feb 25, 2025

ilaflott and others added 11 commits February 25, 2025 14:55

Create constructors.py

d742d5f

Update frecmor.py

f58f0fd

Update cmor_yamler.py

bb5b284

return statement included before calling the run subtool- pprint out to screen instead

Update cmor_yamler.py

c8a4c9c

forgot the yamltools bit

Update test_cmor_run_subtool.py

2d0cb3e

turn on run-one-debug mode at top of testing script

Update frecmor.py

d0a1233

fix minor syntax error causing all the failures

Update test_cmor_run_subtool.py

8fed0fa

fix the import statements.

Update test_cmor_run_subtool_ppan_only.py

dad33f8

fix import statement

new file, fre/yamltools/helpers.py to hold widely used functions. adj…

bac4bfe

…ust import statements when join_constructor pops up

quick tweak

ded54b5

ilaflott changed the title ~~cmor yaml parsing~~ CMORgasbord: yaml parsing Feb 26, 2025

ilaflott added 8 commits February 26, 2025 14:37

update cmor test calls. make debug run one mode an input argument as …

68c1195

…flag, default false. add the debug run one mode to the yaml processing calls as well.

fix for syntax in function call, missing comma

7ae3360

add initial test, add initial success conditions. Goal is now, turn t…

9e1d85b

…his red x into a green check. will adjust testing/config file needs as we go along to ensure we are writing sensible things

add new script for quickly making a vanilla variable list with the eg…

c5e28ce

…regious assumption tha the gfdl name is going to be the mip name. i know this is not the case- just a prototype for possibly interfacing with slurm.

small logger info edits

ef70afa

adjust calls to set run_one_mode for testings sake

4f417a2

add back in old cmor_info_parser, it wont work right away- but it is …

5fd5143

…what worked last time!

annnnnnd we are back! we now have a cmor info parser that produces a …

f130d6e

…combined yaml file

ilaflott added 3 commits February 27, 2025 15:30

new combine-yamls cli call test, also bring base fre yamltools combin…

da995eb

…e unit tests into existence and add in test case ive been running based on the am5 yaml etc. the contents of the yaml too must be checked

add yaml file containing expected output given the fre yamltools comb…

84eca3c

…ine-yaml test call add comparison conditions for lloading the fresh created output file and comparing to static test file data

put back the xfail under test cli fre cmor yaml case1, its not the sp…

b2120ce

…ot to test for the combination doing its job. test the combine_yamls as a python module call, and if the output combined yaml exists already, remove it before recreating

ilaflott marked this pull request as ready for review February 27, 2025 21:24

ilaflott requested review from ceblanton and singhd789 February 27, 2025 21:25

ilaflott self-assigned this Feb 27, 2025

ilaflott added new functioning New feature or request priority: HIGH labels Feb 27, 2025

ilaflott added 5 commits February 27, 2025 16:33

emacs whitespace-cleanup for every file touched

da12556

pylint edits, some print to logging changes in generate-time-averages

cb25640

remove unused output/error variables in regrid xy test. switch print …

cbdf1d4

…statements in regrid app to logging.info statements

remove more of those unused out and err variables in tests scripts fo…

733dd21

…r regridding

trivial pylint edits

553d97d

chanwilson previously requested changes Feb 28, 2025

View reviewed changes

ilaflott added 6 commits February 28, 2025 15:54

pylint the last few minutes of the week away... why not

cbfea55

remove prototype bash script for local variable list resolving- not r…

3554258

…eady for pushing yet. address chan comments RE debug info and logging calls

better comments describing import magic

3ecdd58

silly syntax error

e496cd2

remove some old commented out cruft- add in a doc string for my unit …

ddff672

…test for yamls

remove an old DEBUG flag thats been worked into a command line argument

c89bfe7

ilaflott requested a review from chanwilson February 28, 2025 21:37

ceblanton approved these changes Mar 3, 2025

View reviewed changes

singhd789 approved these changes Mar 3, 2025

View reviewed changes

ilaflott merged commit bb6f681 into main Mar 3, 2025
2 checks passed

ilaflott deleted the 320.cmor_yaml_redux branch March 3, 2025 16:27

ilaflott mentioned this pull request Mar 4, 2025

CMORgasbord: yaml-munging changes, file/directory/component targeting, amongst other changes #375

Merged

8 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CMORgasbord: yaml parsing #368

CMORgasbord: yaml parsing #368

ilaflott commented Feb 25, 2025 •

edited

Loading

ilaflott commented Feb 26, 2025 •

edited

Loading

ilaflott commented Feb 26, 2025

ilaflott commented Feb 27, 2025

chanwilson left a comment

ceblanton left a comment

singhd789 Mar 3, 2025

ilaflott Mar 3, 2025

ilaflott commented Mar 3, 2025

		from . import *


		def experiment_check(mainyaml_dir, experiment, loaded_yaml):

CMORgasbord: yaml parsing #368

CMORgasbord: yaml parsing #368

Conversation

ilaflott commented Feb 25, 2025 • edited Loading

Background

Describe your changes

Issue ticket number and link (if applicable)

Checklist before requesting a review

ilaflott commented Feb 26, 2025 • edited Loading

ilaflott commented Feb 26, 2025

ilaflott commented Feb 27, 2025

chanwilson left a comment

Choose a reason for hiding this comment

ceblanton left a comment

Choose a reason for hiding this comment

singhd789 Mar 3, 2025

Choose a reason for hiding this comment

ilaflott Mar 3, 2025

Choose a reason for hiding this comment

ilaflott commented Mar 3, 2025

ilaflott commented Feb 25, 2025 •

edited

Loading

ilaflott commented Feb 26, 2025 •

edited

Loading