ensemble_run

Purpose

ensemble_run.py executes a single ensemble member for UQ (uncertainty quantification) workflows. It creates a perturbed copy of a pre-built parent case, applies parameter modifications, runs the model, and calculates normalized sum of squared errors (SSE) against observational constraints.

This script is designed to be invoked by manage_ensemble.py for each ensemble member in a parallel sweep, though it can also be run standalone for testing.

Critical constraint: Only works for single-point CLM/ELM cases compiled with MPI_SERIAL. Multi-point or MPI-parallel cases are not supported.

Command-line arguments

Required arguments

--parm_list <path>: File containing parameter names and PFT indices to modify. Format: one line per parameter, whitespace-separated <name> <pft_index>. PFT index is 1-based (0 = apply to all PFTs, -1 = scalar parameter). Examples in examples/parm_list*.

--parm_data <path>: File containing parameter values for this ensemble member. One value per line, order matching --parm_list. Generated by manage_ensemble.py from the sampling distribution.

--constraints <directory>: Directory containing observational constraint files used for SSE calculation. Each file is named after a model output variable (e.g., GPP.txt, STEMC.txt). See post-processing section below.

Optional arguments

--runroot <path>: Root directory where CIME cases live. Defaults to /lustre/pfs1/cades-ccsi/scratch/<username> on CADES, ../../run elsewhere. The script creates <runroot>/UQ/<casename>/g<ensnum>/ for each ensemble member.

--ens_num <integer>: Ensemble member number. Default: 1. Used to generate the member subdirectory name (g00001, g00002, etc.) and unique parameter file names.

--casename <string>: Name of the parent case. If the casename contains 20TR or 1850, treated as a single-phase transient run. Otherwise, assumes a full three-phase BGC spinup sequence and looks for <casename>_I1850CLM45CBCN_ad_spinup, <casename>_I1850CLM45CBCN, and <casename>_I20TRCLM45CBCN.

--machine <string>: Machine identifier. Default: cades. Used to set default --runroot path.

--norun: Copy files and apply parameter perturbations, but do not execute the model. Useful for debugging the ensemble setup.

Parameter perturbation application

For each ensemble member, the script:

Clones the parent case run directory (namelists, parameter files, domain/surface data, streams) into <runroot>/UQ/<casename>/g<ensnum>/. Old netCDF output is removed.
Patches namelists and streams to point to the new run directory and a unique parameter file (clm_params_<ensnum>.nc).
Applies parameter perturbations to clm_params_<ensnum>.nc using the NetCDF manipulation functions getvar / putvar:
- PFT-specific (pft_index > 0): modifies param[pft_index-1]
- Scalar (pft_index == 0): replaces the entire variable with the scalar value
- All PFTs (pft_index == -1): broadcasts the value across all PFT indices
Chains restart files for full spinup runs:
- ad_spinup → final spinup: reads restart from year 251 of ad_spinup
- final spinup → transient: reads restart from year 251 of final spinup
- Hard-coded spinup lengths: 250 years each
Executes the model by invoking acme.exe, e3sm.exe, or cesm.exe from the parent case’s bld/ directory. Logs to acme_log.txt / e3sm_log.txt / cesm_log.txt.

Post-processing and SSE calculation

After the model run completes, the script calculates the normalized sum of squared errors (SSE) between model output and observational constraints:

\[\text{SSE} = \sum_{i} \left( \frac{\text{model}_i - \text{obs}_i}{\text{unc}_i} \right)^2\]

Constraint files in the --constraints directory specify:

year or year-year: time window to average over
doy (optional): day of year for sub-annual constraints
month (optional): monthly constraint
pft (optional): PFT-specific constraint (uses h1 output)
depth (optional): soil layer constraint (maps to CLM layer depth)
value: observed value
unc (optional): observational uncertainty (defaults to 25% of value)

The script aggregates model output over the specified time window, converts flux units (gC/m²/s → gC/m²/yr for NPP/GPP/NEP/NEE), and accumulates SSE across all constraints. Final SSE is written to <workdir>/qpso_ssedata/mysse_<ensnum>.txt for collection by the ensemble manager.

Relationship to manage_ensemble.py

ensemble_run.py is the single-member worker invoked by manage_ensemble.py:

manage_ensemble.py reads a sampling file (--ens_file), distributes ensemble member parameter sets across MPI ranks, and spawns ensemble_run.py for each member via subprocess.
manage_ensemble.py passes --parm_list (fixed across all members) and generates a unique --parm_data file per member from the sampled parameter values.
After all members complete, manage_ensemble.py aggregates SSE results from qpso_ssedata/mysse_*.txt for sensitivity analysis or calibration.

Single-point MPI-serial constraint

Critical limitation: This script only works with single-point CLM/ELM cases compiled with MPI_SERIAL (--mpilib mpi-serial in create_newcase).

Multi-point regional/global cases are not supported.
MPI-parallel (--mpilib openmpi/--mpilib mpich) cases are not supported.
The script directly invokes acme.exe/e3sm.exe/cesm.exe without MPI launcher wrappers.

This constraint exists because ensemble UQ workflows assume lightweight, independent member execution suitable for embarrassingly parallel task farming via manage_ensemble.py’s MPI distribution layer.

For multi-site ensembles, build a separate single-point MPI-serial case for each site and run the ensemble workflow independently per site.