============ ensemble_run ============ Purpose ======= ``ensemble_run.py`` executes a single ensemble member for UQ (uncertainty quantification) workflows. It creates a perturbed copy of a pre-built parent case, applies parameter modifications, runs the model, and calculates normalized sum of squared errors (SSE) against observational constraints. This script is designed to be invoked by ``manage_ensemble.py`` for each ensemble member in a parallel sweep, though it can also be run standalone for testing. **Critical constraint**: Only works for single-point CLM/ELM cases compiled with ``MPI_SERIAL``. Multi-point or MPI-parallel cases are not supported. Command-line arguments ====================== Required arguments ------------------ .. option:: --parm_list File containing parameter names and PFT indices to modify. Format: one line per parameter, whitespace-separated `` ``. PFT index is 1-based (0 = apply to all PFTs, -1 = scalar parameter). Examples in ``examples/parm_list*``. .. option:: --parm_data File containing parameter values for this ensemble member. One value per line, order matching ``--parm_list``. Generated by ``manage_ensemble.py`` from the sampling distribution. .. option:: --constraints Directory containing observational constraint files used for SSE calculation. Each file is named after a model output variable (e.g., ``GPP.txt``, ``STEMC.txt``). See post-processing section below. Optional arguments ------------------ .. option:: --runroot Root directory where CIME cases live. Defaults to ``/lustre/pfs1/cades-ccsi/scratch/`` on CADES, ``../../run`` elsewhere. The script creates ``/UQ//g/`` for each ensemble member. .. option:: --ens_num Ensemble member number. Default: ``1``. Used to generate the member subdirectory name (``g00001``, ``g00002``, etc.) and unique parameter file names. .. option:: --casename Name of the parent case. If the casename contains ``20TR`` or ``1850``, treated as a single-phase transient run. Otherwise, assumes a full three-phase BGC spinup sequence and looks for ``_I1850CLM45CBCN_ad_spinup``, ``_I1850CLM45CBCN``, and ``_I20TRCLM45CBCN``. .. option:: --machine Machine identifier. Default: ``cades``. Used to set default ``--runroot`` path. .. option:: --norun Copy files and apply parameter perturbations, but do not execute the model. Useful for debugging the ensemble setup. Parameter perturbation application =================================== For each ensemble member, the script: 1. **Clones the parent case run directory** (namelists, parameter files, domain/surface data, streams) into ``/UQ//g/``. Old netCDF output is removed. 2. **Patches namelists and streams** to point to the new run directory and a unique parameter file (``clm_params_.nc``). 3. **Applies parameter perturbations** to ``clm_params_.nc`` using the NetCDF manipulation functions ``getvar`` / ``putvar``: - **PFT-specific** (``pft_index > 0``): modifies ``param[pft_index-1]`` - **Scalar** (``pft_index == 0``): replaces the entire variable with the scalar value - **All PFTs** (``pft_index == -1``): broadcasts the value across all PFT indices 4. **Chains restart files** for full spinup runs: - ad_spinup → final spinup: reads restart from year 251 of ad_spinup - final spinup → transient: reads restart from year 251 of final spinup - Hard-coded spinup lengths: 250 years each 5. **Executes the model** by invoking ``acme.exe``, ``e3sm.exe``, or ``cesm.exe`` from the parent case's ``bld/`` directory. Logs to ``acme_log.txt`` / ``e3sm_log.txt`` / ``cesm_log.txt``. Post-processing and SSE calculation ------------------------------------ After the model run completes, the script calculates the normalized sum of squared errors (SSE) between model output and observational constraints: .. math:: \text{SSE} = \sum_{i} \left( \frac{\text{model}_i - \text{obs}_i}{\text{unc}_i} \right)^2 Constraint files in the ``--constraints`` directory specify: - **year** or **year-year**: time window to average over - **doy** (optional): day of year for sub-annual constraints - **month** (optional): monthly constraint - **pft** (optional): PFT-specific constraint (uses ``h1`` output) - **depth** (optional): soil layer constraint (maps to CLM layer depth) - **value**: observed value - **unc** (optional): observational uncertainty (defaults to 25% of value) The script aggregates model output over the specified time window, converts flux units (gC/m²/s → gC/m²/yr for NPP/GPP/NEP/NEE), and accumulates SSE across all constraints. Final SSE is written to ``/qpso_ssedata/mysse_.txt`` for collection by the ensemble manager. Relationship to manage_ensemble.py =================================== ``ensemble_run.py`` is the single-member worker invoked by ``manage_ensemble.py``: - ``manage_ensemble.py`` reads a sampling file (``--ens_file``), distributes ensemble member parameter sets across MPI ranks, and spawns ``ensemble_run.py`` for each member via subprocess. - ``manage_ensemble.py`` passes ``--parm_list`` (fixed across all members) and generates a unique ``--parm_data`` file per member from the sampled parameter values. - After all members complete, ``manage_ensemble.py`` aggregates SSE results from ``qpso_ssedata/mysse_*.txt`` for sensitivity analysis or calibration. Single-point MPI-serial constraint =================================== **Critical limitation**: This script only works with single-point CLM/ELM cases compiled with ``MPI_SERIAL`` (``--mpilib mpi-serial`` in ``create_newcase``). - Multi-point regional/global cases are not supported. - MPI-parallel (``--mpilib openmpi``/``--mpilib mpich``) cases are not supported. - The script directly invokes ``acme.exe``/``e3sm.exe``/``cesm.exe`` without MPI launcher wrappers. This constraint exists because ensemble UQ workflows assume lightweight, independent member execution suitable for embarrassingly parallel task farming via ``manage_ensemble.py``'s MPI distribution layer. For multi-site ensembles, build a separate single-point MPI-serial case for each site and run the ensemble workflow independently per site. See also ======== - ``manage_ensemble.py``: MPI-parallel ensemble manager - ``ensemble_copy.py``: alternative member cloning strategy - ``examples/parm_list*``: parameter list format examples - ``examples/postproc_vars_example``: constraint file format documentation