[Bug]: 1. Failure to produce output for v3 data

[TonyB9000]

What happened?

Failure to produce output for v3 data, possibly due to (metadata)X(cmor), not e2c specifically.

To accommodate v3 (CMIP6) data beyond the CMIP6 closure, we have established a new project (and "mip_era"), "CMIP6-E3SM-Ext" (case sensitive).

This value replaces "CMIP6" in future CMIP6 dataset_id, and user_metadata files.

What did you expect to happen? Are there are possible answers you came across?

Ordinarily, the indicated output directory (here, /lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/product/) would receive the facet-path "CMIP6/CMIP/.../gr/vDate/" and contain the cmorized output files.

(for v3, this would be "CMIP6-E3SM-Ext/CMIP/.../gr/vDate/)

Also, the e3sm_to_cmip logfile (written to the the output directory) claims that a "cmor_logs" directory will also appear, but it does not.

Minimal Complete Verifiable Example (MVCE)

e3sm_to_cmip -v mlotst -u /lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/metadata/historical_r1i1p1f1.json -t /lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables -o YOUR_OUTPUT_DIR -i /lcrc/group/e3sm2/DSM/Ops/test/e2c_input -s --realm mpaso --map /lcrc/group/e3sm2/DSM/Staging/Resource/maps/map_IcoswISC30E3r5_to_cmip6_180x360_traave.20240221.nc

Note: The "-i input" directory has been prepared with the actual input files used.

Relevant log output

My post-processing creates a "run_report" directory:

/lcrc/group/e3sm2/DSM/Ops/DSM_Manager/TEST_REPORTS/test_record-CMIP6-E3SM-Ext.CMIP.E3SM-Project.E3SM-3-0.historical.r1i1p1f1.Omon.mlotst.gr-20250521_183645

containing various log data, and a summary report:

Report-20250521_183645

The log-data is mostly useful here for echoing the input parameters supplied.  But not "error" output seems to havbe been written.

The contents of that report is copied below:

RUN REPORT for dsm_manage_CMIP_production test ====================================

PRE-RUN Artifacts -----------------------------------

    Target CMIP dataset_id: CMIP6-E3SM-Ext.CMIP.E3SM-Project.E3SM-3-0.historical.r1i1p1f1.Omon.mlotst.gr
    Native E3SM dataset_id: E3SM.3_0.historical.LR.ocean.native.model-output.mon.ens1
    Native Case ID:         v3.LR.historical_0051
    Source Metadata file:   /lcrc/group/e3sm2/DSM/Staging/Resource/CMIP6-Metadata/E3SM-3-0/historical_r1i1p1f1.json

    cat /lcrc/group/e3sm2/DSM/Staging/Resource/CMIP6-Metadata/E3SM-3-0/historical_r1i1p1f1.json | grep -v license | grep -v #note | grep CMIP
          "activity_id": "CMIP",
          "parent_activity_id": "CMIP",
          "grid": "data regridded to a CMIP6 standard 1x1 degree lonxlat grid from the native grid using an area-average preserving method.",
          "outpath": "CMIP6-E3SM-Ext",
          "_controlled_vocabulary_file": "CMIP6_CV.json",
          "_AXIS_ENTRY_FILE": "CMIP6_coordinate.json",
          "_FORMULA_VAR_FILE": "CMIP6_formula_terms.json",
          "_cmip6_option": "CMIP6",
          "mip_era": "CMIP6-E3SM-Ext",
          "parent_mip_era": "CMIP6-E3SM-Ext",

    cat /lcrc/group/e3sm2/DSM/Staging/Resource/CMIP6-Metadata/E3SM-3-0/historical_r1i1p1f1.json | grep -v license | grep -v #note | grep source_id
          "parent_source_id": "E3SM-3-0",
          "source_id": "E3SM-3-0",
          "output_path_template": "<mip_era><activity_id><institution_id><source_id><experiment_id><_member_id><table><variable_id><grid_label><version>",
          "output_file_template": "<variable_id><table><source_id><experiment_id><_member_id><grid_label>"

POST-RUN Artifacts -----------------------------------

    ls -l tmp/v3.LR.historical_0051/product
        total 1041
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:15 20250521_181505_431691.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:15 20250521_181534_713909.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:15 20250521_181534_871635.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:15 20250521_181535_036291.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:16 20250521_181604_164331.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:16 20250521_181604_678067.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:16 20250521_181604_944584.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:16 20250521_181635_444463.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:16 20250521_181635_939495.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:17 20250521_181705_434127.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:17 20250521_181737_042226.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2  1740 May 21 13:18 20250521_181811_472290.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:18 20250521_181840_813651.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:18 20250521_181840_822019.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:18 20250521_181840_908557.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:19 20250521_181908_147324.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2 50580 May 21 13:19 20250521_181908_147338.log
        -rw-rw-r--+ 1 ac.bartoletti1 E3SM2  4059 May 21 13:19 user_metadata.json

    Content of e2c log tmp/v3.LR.historical_0051/product/20250521_181505_431691.log

        2025-05-21 13:15:05.449062 [INFO]: __main__.py(__init__:163) >> --------------------------------------
        2025-05-21 13:15:05.449220 [INFO]: __main__.py(__init__:164) >> | E3SM to CMIP Configuration
        2025-05-21 13:15:05.449289 [INFO]: __main__.py(__init__:165) >> --------------------------------------
        2025-05-21 13:15:05.452245 [INFO]: __main__.py(__init__:189) >>     * Timestamp: 20250521_181505_431691
        2025-05-21 13:15:05.452305 [INFO]: __main__.py(__init__:189) >>     * Version Info: version 1.11.3
        2025-05-21 13:15:05.452343 [INFO]: __main__.py(__init__:189) >>     * Mode: Serial
        2025-05-21 13:15:05.452382 [INFO]: __main__.py(__init__:189) >>     * Variable List: ['mlotst']
        2025-05-21 13:15:05.452416 [INFO]: __main__.py(__init__:189) >>     * Input Path: /lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/native_data/seg-1850
        2025-05-21 13:15:05.452449 [INFO]: __main__.py(__init__:189) >>     * Output Path: /lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/product
        2025-05-21 13:15:05.452481 [INFO]: __main__.py(__init__:189) >>     * Precheck Path: None
        2025-05-21 13:15:05.452512 [INFO]: __main__.py(__init__:189) >>     * Log Path: /lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/product/20250521_181505_431691.log
        2025-05-21 13:15:05.452542 [INFO]: __main__.py(__init__:189) >>     * CMOR Log Path: /lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/product/cmor_logs
        2025-05-21 13:15:05.452572 [INFO]: __main__.py(__init__:189) >>     * Temp Path for Processing MPAS Files: /lcrc/group/e3sm2/DSM/tmp
        2025-05-21 13:15:05.452601 [INFO]: __main__.py(__init__:189) >>     * Frequency: mon
        2025-05-21 13:15:05.452632 [INFO]: __main__.py(__init__:189) >>     * Realm: mpaso
        2025-05-21 13:15:06.368011 [WARNING]: warnings.py(_showwarnmsg:112) >> /home/ac.bartoletti1/anaconda3/envs/dsm_test_e2c/lib/python3.12/site-packages/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedelt
aCoder' instance.
          ds = xr.open_dataset(path)
        2025-05-21 13:15:06.588973 [WARNING]: warnings.py(_showwarnmsg:112) >> /home/ac.bartoletti1/anaconda3/envs/dsm_test_e2c/lib/python3.12/site-packages/e3sm_to_cmip/__main__.py:340: FutureWarning: In a future version of xarray decode_timedelta will default to False rather than None. To silence this warning, set decode_timedelta to True, False, or a 'CFTimedelt
aCoder' instance.
          ds = xr.open_dataset(path)

    dsm subordinate run script: tmp/v3.LR.historical_0051/scripts/CMIP6-E3SM-Ext.CMIP.E3SM-Project.E3SM-3-0.historical.r1i1p1f1.Omon.mlotst.gr-gen_CMIP6.py

    EXAMINE dsm subordinate run script log: tmp/v3.LR.historical_0051/caselogs/CMIP6-E3SM-Ext.CMIP.E3SM-Project.E3SM-3-0.historical.r1i1p1f1.Omon.mlotst.gr.sublog

    EXAMINE dsm_generate_CMIP6 log: tmp/mainlogs/dsm_gen-CMIP6-E3SM-Ext.CMIP.E3SM-Project.E3SM-3-0.historical.r1i1p1f1.Omon.mlotst.gr.log-20250521_181413_082176

Anything else we need to know?

No response

Environment

 active environment : dsm_test_e2c
active env location : /home/ac.bartoletti1/anaconda3/envs/dsm_test_e2c
        shell level : 2
   user config file : /home/ac.bartoletti1/.condarc

populated config files :
conda version : 23.7.4
conda-build version : 3.26.1
python version : 3.11.5.final.0
virtual packages : __archspec=1=x86_64
__glibc=2.28=0
__linux=4.18.0=0
__unix=0=0
base environment : /home/ac.bartoletti1/anaconda3 (writable)
conda av data dir : /home/ac.bartoletti1/anaconda3/etc/conda
conda av metadata url : None
channel URLs : https://repo.anaconda.com/pkgs/main/linux-64
https://repo.anaconda.com/pkgs/main/noarch
https://repo.anaconda.com/pkgs/r/linux-64
https://repo.anaconda.com/pkgs/r/noarch
package cache : /home/ac.bartoletti1/anaconda3/pkgs
/home/ac.bartoletti1/.conda/pkgs
envs directories : /home/ac.bartoletti1/anaconda3/envs
/home/ac.bartoletti1/.conda/envs
platform : linux-64
user-agent : conda/23.7.4 requests/2.31.0 CPython/3.11.5 Linux/4.18.0-553.16.1.el8_10.x86_64 rhel/8.10 glibc/2.28 aau/0.4.2 c/HGg8gxpvRuovNC12YQRpXg s/ClAe4uAavrqIHCJNKnding e/q9fFG7jpRw6ae6nxC1mGfQ
UID:GID : 22431:20001
netrc file : None
offline mode : False

[TonyB9000]

@tomvothecoder I have done a side-by-side run of e3sm_to_cmip, one for
v2.NARRM.historical.r2 Omon.mlotst, and one for
v3.LR.historical.r1 Omon.mlotst
(as close as I could get to identical processing, given the data on-hand).

The former creates expected output (except no "cmor_logs" appear in the output, only the e2c logs).
The latter only produces the e2c logs, no datafiles and (again) no cmor_logs.

The MVCE should reproduce the v3 issue.

[chengzhuzhang]

@TonyB9000 thank you for filing the issue!
The fact that the former runs as expected and v3.LR has some issue. I think one thing at least we should try is to add an entry of E3SM-3-0 following existing E3SM-2-0 entry under cmip6-cmor-tables/Tables/CMIP6_CV.json
Though I don't know why no "cmor_logs" for both cases.

[tomvothecoder]

Thanks @TonyB9000, I will take a look at why the cmor_logs aren't being produced tomorrow.

Once they are produced, it should hopefully give you a better idea for why cmor is failing to produce outputs.

[TonyB9000]

@tomvothecoder @chengzhuzhang I just discovered the missing cmor logs.

In the e3sm_to_cmip util.py, we have the function:

def setup_cmor(var_name, table_path, table_name, user_input_path):
    """
    Sets up cmor and logging for a single handler
    """
    var_name = str(var_name)
    table_path = str(table_path)
    table_name = str(table_name)
    user_input_path = str(user_input_path)

    logfile = os.path.join(os.getcwd(), "logs")
    if not os.path.exists(logfile):
        os.makedirs(logfile)

    logfile = os.path.join(logfile, var_name + ".log")
    cmor.setup(inpath=table_path, netcdf_file_action=cmor.CMOR_REPLACE, logfile=logfile)

    cmor.dataset_json(user_input_path)
    cmor.load_table(table_name)

So, a "logs" directory is created in your execution PWD (mine is rather crowded).

It appears that the failure to create output comes from violations of the "Controlled Vocabulary" file.
Both SourceID "E3SM-3-0" and our mip_era "CMIP6-E3SM-Ext" are not being recognized.

Here is the output from the latest CMOR log:

C Traceback:
In function:

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Logfile /lcrc/group/e3sm2/DSM/Ops/DSM_Manager/logs/mlotst.log already exist.
! Renamed to: /lcrc/group/e3sm2/DSM/Ops/DSM_Manager/logs/mlotst.log_2025-05-21T13:20:15
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkSourceID
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: The source_id, "E3SM-3-0", found in your
! input file (/lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/product/user_metadata.json) could not be found in
! your Controlled Vocabulary file. (/lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables/CMIP6_CV.json)
!
! Please correct your input file by using a valid source_id listed in your MIP tables' CV file.
! To add a new source_id to the /lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables/CMIP6_CV.json file, open a new issue in the
! table's Github repository. Managed project CMOR and MIP tables are listed at
! https://wcrp-cmip.github.io/WGCM_Infrastructure_Panel/cmor_and_mip_tables.html.
! Contact "pcmdi-cmip@llnl.gov" for additional guidance.
!
! See "http://cmor.llnl.gov/mydoc_cmor3_CV/" for further information about
! the "source_id" and "source" global attributes.
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkParentExpID
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Your parent_source_id "E3SM-3-0" defined in your input file
! could not be found in your Control Vocabulary file.(/lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables/CMIP6_CV.json)
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
In function: _CV_checkParentExpID
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Your input attribute "parent_mip_era" defined as "CMIP6-E3SM-Ext" will be replaced with
! "CMIP6" as defined in your Control Vocabulary file.
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
In function: _CV_checkSubExperiment
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Warning: Your input attribute "sub_experiment" was not defined and
! will be set to "none"
! as defined in your Control Vocabulary file "/lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables/CMIP6_CV.json".
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_ValidateAttribute
! called from: _CV_checkGblAttributes
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: The registered CV attribute "source_id" as defined as "E3SM-3-0" does not match any value in the CV.
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkGblAttributes
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Please fix required attributes mentioned in
! the warnings/error above and rerun. (aborting!)
!
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function: _CV_checkSourceID
! called from: cmor_write
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: The source_id, "E3SM-3-0", found in your
! input file (/lcrc/group/e3sm2/DSM/Ops/DSM_Manager/tmp/v3.LR.historical_0051/product/user_metadata.json) could not be found in
! your Controlled Vocabulary file. (/lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables/CMIP6_CV.json)
!
! Please correct your input file by using a valid source_id listed in your MIP tables' CV file.
! To add a new source_id to the /lcrc/group/e3sm2/DSM/Staging/Resource/cmor/cmip6-cmor-tables/Tables/CMIP6_CV.json file, open a new issue in the
! table's Github repository. Managed project CMOR and MIP tables are listed at
! https://wcrp-cmip.github.io/WGCM_Infrastructure_Panel/cmor_and_mip_tables.html.
! Contact "pcmdi-cmip@llnl.gov" for additional guidance.
!
! See "http://cmor.llnl.gov/mydoc_cmor3_CV/" for further information about
! the "source_id" and "source" global attributes.
!
!!!!!!!!!!!!!!!!!!!!!!!!!


C Traceback:
! In function:

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: Cannot continue until you fix the errors listed above: -4
!
!!!!!!!!!!!!!!!!!!!!!!!!!

[chengzhuzhang]

Let's try add the E3SM-3-0 entry, also try CMIP6 as mip_era.

We can deal with publisher complaint later.

[TonyB9000]

@tomvothecoder @chengzhuzhang The last Source_ID listed (for E3SM) is "E3SM-2-1", and reads:

            "E3SM-2-1":{
                "activity_participation":[
                    "CMIP"
                ],
                "cohort":[
                    "Registered"
                ],
                "institution_id":[
                    "E3SM-Project"
                ],
                "license_info":{
                    "exceptions_contact":"@llnl.gov <- e3sm-data-support",
                    "history":"2024-01-30: initially published under CC BY 4.0",
                    "id":"CC BY 4.0",
                    "license":"Creative Commons Attribution 4.0 International (CC BY 4.0; https://creativecommons.org/licenses/by/4.0/)",
                    "source_specific_info":"",
                    "url":"https://creativecommons.org/licenses/by/4.0/"
                },
                "source_id":"E3SM-2-1",
                "source":"E3SM 2.1 (2024): \naerosol: MAM4 w/ new resuspension, marine organics, secondary organics, and dust (atmos physics grid)\natmos: EAM (E3SMv2.1, cubed sphere spectral-element; 5400 els., 30x30 per cube face. Dynamics: degree 3 (p=3) polynomials within each spectral els., 112 km ave. resolution. Physics: 2x2 finite volume cells within each spectral els., 1.5 degree (168 km) average grid spacing; 72 vertical layers w/ top at 60 km)\natmosChem: Troposphere specified oxidants (except passive ozone with the lower boundary sink) for aerosols. Stratosphere linearized interactive ozone (LINOZ v2) (atmos physics grid)\nland: ELM (E3SMv2.1, atmos physics grid, satellite phenology mode), MOSART (E3SMv2.1, 0.5 deg lat/lon grid)\nlandIce: none\nocean: MPAS-Ocean (E3SMv2.1, EC30to60E2r2 unstructured SVTs mesh with 236853 cells and 719506 edges, variable resolution 60 km to 30 km; 60 levels; top grid cell 0-10 m)\nocnBgchem: none\nseaIce: MPAS-Seaice (E3SMv2.1, MPAS-Ocean grid; 5 ice categories, 7 ice layers, 5 snow layers)"
            },

and the ONBLY "mip_era" listed is

        "mip_era":[
            "CMIP6"
        ],

[TonyB9000]

@chengzhuzhang If I change the "mip_era" back to "CMIP6", this will cause CMOR to make the output directory match. Then when my process attempts to move the data to the warehouse (using the new dataset_id) it won;t find the data unless I make a special case substitution. The publisher will want the metadata to say "mip_era: CMIP6-E3SM-Ext", Juggling these will not be good.

I am (relatively) certain that cmorizing will "work" either way, just for test purposes. But when it comes to production, we need to get this ironed out.

I assume that The CMOR modules takes its cues from the CV file. We should probably try to djinn up some magic there.

I will experiment with editing the local CV file to see what damage I can do ...

[chengzhuzhang]

I understand that "mip_era" needs to be CMIP6-E3SM-Ext for publisher and your dataset_id based workflow. I just prefer to have CMIP6 in the file name and directory structure, but I guess using CMIP6-E3SM-Ext is just fine.

You could use the E3SM-2-1 as a template, but I do remember that we registered to more activities for E3SM-2-0, so for other .json files under cmore tables, we can use E3SM-2-0 as a template.

[TonyB9000]

@tomvothecoder Can we snake the "declared" CMOR logs location ([output_dir]/cmor_logs) down to the "setup_cmor()" function in util.py to replace the current "PWD behavior"?

@chengzhuzhang In the future, I will model the 3.0 Source_ID after E3SM-2-0. For now, we merely want to ensure that output cmorizing will work - and no hidden "3.0 secrets" in the data, w.r.t. mapping files, masks, etc, will be an issue.

[TonyB9000]

@tomvothecoder Perhaps a separate issue - but when "cmor.write" or "cmor.close" exits with a failure (as indicated in the cmor logs - "aborterd"), there MUST be a way to have e3sm_to_cmip return a non-zero exit status, alerting the caller that something went wrong.

@chengzhuzhang I am not entirely wedded to using "CMIP6-E3SM-Ext" everywhere. We can (with some effort) revert back to just "CMIP6". But these values are used in some 25 different places, and we need to be clear where they absolutely need to become the extended mip-era for publication (like the esg.yaml, and the file-embedded user-metadata). The issue with the user-metadata is that it is used BOTH during CMORizing, and during publication. If cmor demands it one way, we will need to edit 100,000+ datafiles pre-publication to make the publisher happy. I will need to re-edit the Archive_Map entries, the dataset_spec.yaml and other locations to be sure. At least, it should be easier to find "CMIP6-E3SM-Ext" than search for "CMIP6". ...

[tomvothecoder]

@tomvothecoder Can we snake the "declared" CMOR logs location ([output_dir]/cmor_logs) down to the "setup_cmor()" function in util.py to replace the current "PWD behavior"?

To summarize the issue, MPAS variables defined as mpas_vars modules have their own cmor.write function, which saves the cmor_logs under the PWD. However after merging #289, variables defined via handlers.yaml (using the VarHandler class) now save cmor_logs output under the -o/--output directory. As a result, there are two separate cmor_logs directories.

To address this issue, we can either 1) #200 for a long-term, sustainable fix, or 2) address #298 for a short-term fix. I think I will go with 2) #298 since my time is limited here.

@tomvothecoder Perhaps a separate issue - but when "cmor.write" or "cmor.close" exits with a failure (as indicated in the cmor logs - "aborterd"), there MUST be a way to have e3sm_to_cmip return a non-zero exit status, alerting the caller that something went wrong.

The question is whether we want to:

1. Keep processing variables even if there are any failures, but clearly show failures in the e3sm_to_cmip console output and log file
2. Stop processing upon the first failure -- this probably makes more sense on a debug mode?
3. Implement 1 and 2

Additionally, we want to capture cmor module logs in the e3sm_to_cmip log files (#215, #291)

[TonyB9000]

@tomvothecoder For the latter (exit status) issuem and not interferring with legacy behavior, I think a "--exit-on-cmor-error" flag (default: False) would be best. Users who provide a list of vars can ignore the flag. My scheme employs a separate e2c call per variable (actually, per variable_decade!)

Regarding the cmor_logs issue I read (but did not understand) the issues #200 and #298.

Also, I absolutely ... "dislike" ... the CMOR log format. Can we have this revamped to something more humane?