GCM ModelE – Elephants Playing Mozart

I’ve done a first pass through the GCM (Global Circulation Model) Model E code.

If “With four parameters I can fit an elephant, and with five I can make him wiggle his trunk” then Model E is structured so that the Elephant can play Mozart while on parade…

(History of the saying in a short posting here by Steven Goddard.)

Never The Same Code Twice

First off, it follows in the tradition of GISStemp where every time you run the program it is recompiled. This means it is never the same code twice.

Compilers are very complicated beasts. They make decisions. LOTS of decisions. Sometimes they have bugs in them. Sometimes they do things in different order. They pull in bits of code from various libraries and link it all together, so if the libraries change, the product changes.

Historically, libraries were ‘static linked’ so that your program, when run, was a constant and unchanging product. You can still set static linking if desired. In the last few decades, dynamic linking has become more popular. The program only loads the bits of the libraries when it runs. This makes them smaller in memory footprint, and it means an update to a library doesn’t require recompiling all the programs that use it (unless the change is a big one and changes calling sequence or parameters). The downside is that changes to the library can cause changes, or failures, of the programs. In the Microsoft world that’s the DLL Dynamically Linked Library failures some may have run into.

While dynamic libraries are a benefit, they do add a little bit of risk of change. Generally manageable since libraries are strongly QA tested and any bugs usually surface fast as they are also widely used. So generally folks are OK with that. For those who are not, you can static link to the old libraries if desired.

Model E goes way past that.

Not only does it give you lots of potential changes from dynamic libraries and compiler runs, it lets you tune and prune compiler flags, linker flags, and more. Turning on or off various sections of the code, adding, changing, or removing bits of it as you like it. Then your “run” is compiled and executed.

I’d originally expected there to be a “run deck” of parameters for any given run, and some relatively static input files of historical data. I’d not expected a lot of “fiddling with the model code” on every run. Essentially the whole model is parameters. When someone says “I ran Model E” and got FOO results, what they really ought to be saying is “I tuned Model E to get FOO results”.

This may well be “normal” in what passes for “Climate Science”, but it is not the way to get stable and reliable results via modeling actual natural processes as defined by observations of nature.

Model, which Model?

You can get a copy of the Model E code from NASA/GISS via this link (and the link in it) that describes the model.

https://www.giss.nasa.gov/tools/modelE/

Your first clue that this is a “Moving River” (as in “You can never cross the same river twice.”) come in that text. Bolding mine.

The current incarnation of the GISS series of coupled atmosphere-ocean models is available here. Called ModelE,
[…]
Model versions used for the CMIP5 and CMIP6 simulations are available via the nightly snapshots of the current code repository, including the frozen ‘AR5_branch’. However, users should be aware that these snapshots are presented ‘as is’ and are not necessarily suitable for publication-quality experiments.

I note in passing the use of “experiments” to describe computer model fantasies… I have nothing against models as tools. They can be very useful. But they are NOT experiments. They are a computerized fantasy that is best used to “inform our ignorance”. Compare where the model diverges from reality to see where you have it wrong. That’s where you learn something.

Sidebar: I ran a supercomputer center for 7+ years mostly doing plastic modeling for making dies. The model was right about 90% of the time saving lots of money on dies. ONE fluid at ONE temperature in ONE fixed space, 10% wrong. That from a well tested widely used program designed for exactly that use, not something hacked up on each run with code changes. At $1/4 Million per set of dies, we were VERY motivated to get 100% right.

The link to the “AR5 Branch” is here:

https://simplex.giss.nasa.gov/snapshots/modelE2_AR5_branch.2017.01.18_08.50.02.tgz

I note in passing that the one I downloaded to do this examination was named:
modelE2_AR5_branch.2016.12.28_08.50.01.tgz

so it looks like that frozen AR5 branch changes daily or weekly…

Or maybe it’s one of these…

GISS Model E Source Code Snapshots

The following versions of Model E source code are available for public download:

modelE2_AR5_branch.2017.01.18_08.50.02.tgz – the version of the code used for AR5 simulations.
modelE2_AR5_v2_branch.2017.01.18_08.50.02.tgz – updated and improved version of AR5 code (still under development).
modelE1_patches.2017.01.18_08.50.02.tgz – the version of the code used for AR4 simulations.
modelE2.2017.01.18_08.50.02.tgz – the latest code from the develpment branch. This code is always under development by definition. It may be unstable or not even compile at all. Use at your own risk.
modelE2_planet_1.0.2017.01.18_08.50.02.tgz – version 1.0 of ROCKE3D model as it is currently used in our planetary simulations. (Still under development, functionality may change at later time.)

So their “AR5 Branch” of archived code is actually 2 versions, one under development, and a bunch of patches related to AR4, maybe, and they have a “development branch” too… Decisions decisions… (And I left out their ‘spectral binning’ code from further down the page…)

GISS Model E Tools

The following tools are available for public download:

Spectral_Binning_GISSGCM.2017.01.18_08.50.02.tgz – Scripts to generate ModelE radiation input files from a raw stellar spectrum.

So you can automate the creation of the stellar parameter fudge… Wonder if it handles the recent cool down of the sun with shift of 10% or so of the UV vs IR levels??…

That’s just for starters…

Down In The Weeds

When you unpack the code, you get this directory structure:

chiefio@Headend:/Climate/modelE2_AR5_branch$ ls
Makefile  aux   config  doc   init_cond  templates
README    cmor  decks   exec  model      tests

At some future time I’m going to describe the technical side of it more, but now I’m just looking at the control knobs.

Typically you start with reading the README file, and a look at the Makefile (that builds any given bit of software in the *Nix world). It looks fairly clean at this point with only one Makefile… but when you dig into it, that changes. First, though, some sizes.

chiefio@Headend:/Climate/modelE2_AR5_branch$ du -ks *
4	Makefile
4	README
1076	aux
92	cmor
132	config
36	decks
440	doc
224	exec
272	init_cond
15996	model
1360	templates
136	tests

Most of the code is in the “model” directory. Yet “aux” and “templates” are fairly large too. 1 MB (1076 kB) for AUX and 1.3 MB for templates. Those will be doing something big. Then init_cond will be setting some initial state too.

What does the README say? It says essentially don’t trust the README… and here’s some untrustworthy clues:

NOTE: The full documentation on how to run the model is in doc/HOWTO.html.
This summary is not complete, and not necessarily up-to-date either.

PLEASE READ THE FULL DOCUMENTATION – IT REALLY WILL MAKE YOUR LIFE EASIER!

The directory tree of the modelE has the following structure:

  modelE
        |
        |-/model   (the source code for GCM model)
        |
        |-/aux     (auxiliary programs such as pre- and post-processing)
        |
        |-/exec    (various scripts needed to compile and setup the model)
        |
        |-/doc     (directory for documentation)
        |
        |-/decks   (directory for rundecks)
                |
                |-.R     (rundeck for the run )
                |
                |-/_bin  (directory for binaries for )
                |
                |-/      (link to directory where you setup
                |                     and run )
                |-.R
                ................

Configuring the model on your local system

Intended working directory is directory modelE/decks. In this
directory type “gmake config”. This will create a default ~/.modelErc
file in your home directory. This should be edited so that run output,
rundeck libraries etc. can be properly directed, and so that the
compile options (multiple processing, compiler name , NetCDF libraries
etc.) can be set appropriately.

Compiling and running the model.

All rundecks should be created inside this directory and all “make”
commands should be run from there. The following is a typical example
of how to compile and setup a run with the name “my_run”:

cd decks # go to directory decks
gmake rundeck RUN=my_run # create rundeck for “my_run”

You will need to edit the rundeck in order to choose a configuration
that is appropriate. Once that is done…

gmake gcm RUN=my_run # compile the model for “my_run”
gmake setup RUN=my_run # run setup script for “my_run”

Make sure that you create the rundeck with “gmake rundeck …” before
running any other commands for this run, otherwise the Makefile will
not understand you. You can skip “gmake gcm …” and just do “gmake setup…”
in which case gcm will be compiled automatically.
Another command you want to run (after creating the rundeck) is:

gmake aux RUN=my_run

This will compile auxiliary programs in /aux. All the binaries (both for
model and for auxiliary programs) are put into /decks/my_run.bin .

The following is a list of targets currently supported by Makefile:

config – copy the default .modelErc setup to your home directory.
rundeck – create new rundeck
depend – create dependencies for specified rundeck
gcm – compile object files and build executable for specified rundeck
aux – compile standard auxiliary programs
auxqflux- compile auxiliary programs for computing qflux
auxdeep – compile auxiliary programs for setting deep ocean
setup – do setup for specified rundeck
clean – remove listings and error files
vclean – remove object files, .mod files and dependencies
newstart- remove all files in the run directory
exe – compile gcm and put executable into RUN directory
htmldoc – create web-based documentation for this RUN

If you run “gmake” without arguments it will print a short help.

The use of ‘gmake’ to run your model is a big whack with the Clue Stick that this thing does a rebuild on each run. Toward the bottom I’ve bolded a few bits where that is confirmed. Every rundeck a new model is built. We also see that ‘exec’ is where the real making of the program is done. So what’s in exec?

chiefio@Headend:/Climate/modelE2_AR5_branch$  ls exec
CommandEntry.pm         modelErc                runpmE
CommandPool.pm          pdE                     setup_e.pl
RegressionTools.pm      pproc_dep.pl            setup_e_new.pl
clone                   protect_ccp_options.pl  sfmakedepend
comp_mkdep.pl           r_to_mk.pl              split72.pl
compare_module_file.pl  regression.py           sswE
convertToFreeFormat.py  regressionTests.pl      sumacc_nc
editRundeck.sh          regression_tests.sh     unlock
gcmdoc.pl               runE
get_input_data          runE_new

A mix of Perl scripts (.pl) and shell scripts (.sh) and Python (.py) and more. These folks can’t even keep a standard for what language to use. I know, I know, right tool for the job and all that. BUT it make it a maintenance nightmare as you must be fluent in 4 languages right out the gate. (The good news is that they are not that different. The bad news is that they have pernicious differences in some details…)

I don’t know what a .pm file type is, so I looked inside the RegressionTools one. It looks like a bunch of parameter setting. (Is pm short for parameters?)

Here’s a sample from the top of that file:

#package RegressionTools;
use CommandEntry;

my $standardFlags = "SETUP_FLAGS=-wait";
my $HYCOM = "E1fzhyc";

my $extraFlags;
$extraFlags{SERIAL}   = "";
$extraFlags{MPI}      = "ESMF=YES NPES=\$npes";
$extraFlags{OPENMP}   = "EXTRA_FFLAGS=-mp MP=YES NPROCS=\$npes";
$extraFlags{SERIALMP} = "EXTRA_FFLAGS=-mp"; # Intel kludge for matching serial and OpenMP.

$extraFlags{E1M20} ="";
$extraFlags{E1F20} ="";
$extraFlags{E1oM20} ="";
$extraFlags{E001tr} ="";
$extraFlags{$HYCOM} ="EXTRA_FFLAGS+=-DCHECK_OCEAN_HYCOM";

my $numLinesCMP;
$numLinesCMP{E1M20}  = 120;
$numLinesCMP{E1F20}  = 120;
$numLinesCMP{E1oM20} = 110;
$numLinesCMP{E001tr} = 123;
$numLinesCMP{$HYCOM} = 138;

sub setModuleEnvironment {
#    require perlModule;
    require "$ENV{MODULESHOME}/init/perl";
    module (purge);
    module (load, "comp/intel-10.1.023", "lib/mkl-10.1.2.024", "mpi/impi-3.2.1.009");

Now you can see why I had that preamble about libraries. This thing is setting the library environment as a parameter…

How much of this stuff is there?

chiefio@Headend:/Climate/modelE2_AR5_branch/exec$ wc -l *.pm
  157 CommandEntry.pm
  148 CommandPool.pm
  221 RegressionTools.pm
  526 total

526 lines. IF only 1/2 of them actually set parameters, that’s over 250. If 5 wiggles the trunk, what does 5 x 5 x 5 x 2 do?

But that’s a crude guess at best. For one thing, the CommandEntry.pm file isn’t all parameter setting. It runs some scripts inside it that does a variety of things, some obscure at this stage. Some snips from it:

First a bit of Perl that looks to be reading in command line arguments (parameters):

#!/usr/bin/perl
package CommandEntry;

sub new {
  my $proto = shift;
  my ($args) = @_;

  my $class = ref($proto) || $proto;
  my $self  = {};

  $self -> {COMMAND} = " ";
  $self -> {NUM_PROCS} = 1;
  $self -> {QUEUE} = LOCAL; # or BATCH
  $self -> {DEPENDENCY} = ""; # place holder for list of references to other commands
  $self -> {STATUS} = PENDING;
  $self -> {STDOUT_LOG_FILE} = "LOG";
  $self -> {STDERR_LOG_FILE} = "ERR";

  # override defaults with arguments
  while ( my ($key, $value) = each(%$args) ) {
    $self -> {$key} = $value;
  }

  bless ($self, $class);
  return $self;
}

But further down in has an embedded ‘here script’ in standard Bourne Shell:

  my $script = <<EOF;
#!/bin/bash
#PBS -l select=$nodes:ncpus=$ncpus:proc=neha
#PBS -l walltime=$walltime
#PBS -W group_list=a940a
#PBS -N regression
#PBS -j oe
##PBS -o foo
#PBS $queueString

cd \$PBS_O_WORKDIR
. /usr/share/modules/init/bash
module purge
module load comp/intel-10.1.023 lib/mkl-10.1.2.024 mpi/impi-3.2.1.009

$commandString
EOF

and not only is it working off of a command string, but it fiddles with the libraries too.

Yet even that isn’t the end of it. Looking in regreessionTests.pl we find a bunch more environment setting. First off, it calls those other bits above so we know where they get executed:

use Env;
use CommandPool;
use CommandEntry;
use RegressionTools;

Then for some reason it makes an http reference when it is opening a log file…

open(LOG,">nightlyTests.log");
# Make the LOG hot: <http://www.thinkingsecure.com/docs/tpj/issues/vol3_3/tpj0303-0002.html>

{
    my $ofh = select LOG;
    $| = 1;
    select $ofh;
}

Then it gets down to setting a bunch of environment variables (think of them as parameters that change where the program run gets parts of its context…)

$ENV{CVS_RSH}="ssh";
$env{CVSROOT}="simplex.giss.nasa.gov:/giss/cvsroot";
my $scratchDir = $ENV{NOBACKUP}."/regression_scratch";
$env{SCRATCH_DIRECTORY}=$scratchDir;
$env{REFERENCE_DIRECTORY}="$env{SCRATCH_DIRECTORY}/modelE";
$env{BASELINE_DIRECTORY}="$ENV{NOBACKUP}/modelE_baseline";
$env{RESULTS_DIRECTORY} = $ENV{NOBACKUP}."/regression_results";

@modelErcVariables = (DECKS_REPOSITORY, MP, CMRUNDIR, EXECDIR, OVERWRITE,
                      OUTPUT_TO_FILES, VERBOSE_OUTPUT, SAVEDISK, GCMSEARCHPATH,
                      COMPILER, COMPILER_VERSION, BASELIBDIR, ESMF_BOPT, MPIDISTR, NETCDFHOME);

Now from my POV, the PITA (Pain In The Ass…) is that they have things like “simplex.giss.nasa.gov” coded into this thing. To run it elsewhere I will likely need to find and change those things. OK, nice that it is set in a parameter setting bit, but PITA to have it set at all. Typically you give a program a working directory path and leave it alone.

Now the good thing is it looks like they are actually using a CVS repository to capture changes of state. (If only that would do that for actual temperature data…) but the bad thing is they have it coded into the setup programs.

$env{DECKS_REPOSITORY}="$scratchDir/decks_repository";
$env{MP}="no";
$env{CMRUNDIR}="$scratchDir/cmrun";
$env{EXECDIR}="$scratchDir/exec";
$env{OVERWRITE}="YES";
$env{OUTPUT_TO_FILES}="YES";
$env{VERBOSE_OUTPUT}="YES";
$env{SAVEDISK}="$scratchDir/savedisk";
$env{GCMSEARCHPATH}="/discover/nobackup/projects/giss/prod_input_files";
$env{BASELIBDIR}="/usr/local/other/baselibs/ESMF222rp3_NetCDF362b6_10.1.017_intelmpi/Linux";

And we have yet more library fiddle… Just what are those libraries and where do I get a copy? Who knows… Yet Another Dig Here in getting this set up and run. Hopefully the docs cover it.

There’s a whole lot more of that kind of thing in that particular directory, but I’m going to move on to another bit as these weeds are already too deep.

AUX making Inputs

In the “aux” directory there are a lot of files, but the ‘make’ ones caught my eye first.

chiefio@Headend:/Climate/modelE2_AR5_branch$ ls aux/make*
aux/make_surface_transient_Alkenes_C90.sh
aux/make_surface_transient_BCB.sh
aux/make_surface_transient_BCII.sh
aux/make_surface_transient_CH4_C90.sh
aux/make_surface_transient_CO_C90.sh
aux/make_surface_transient_NH3.sh
aux/make_surface_transient_NOx_C90.sh
aux/make_surface_transient_OCB.sh
aux/make_surface_transient_OCII.sh
aux/make_surface_transient_Paraffin_C90.sh
aux/make_surface_transient_SO2.sh

So we’ve got 10 different programs “make”ing a set of transient parameters for specific gasses. Lots of knobs here.

Looking in the last one (SO2) a sample:

#!/bin/bash

module avail tool/idl
module load tool/idl-6.4
ulimit -s 6000000
ulimit -v unlimited

res='C90'
spec='SO2'

for year in 1850
do
  cd /discover/nobackup/dgueyffi/modelE/aux

   dmget /archive/g08/dmkoch/ftp/AR5/IPCC_emissions_${spec}_anthropogenic_${year}_0.5x0.5*nc
   cp /archive/g08/dmkoch/ftp/AR5/IPCC_emissions_${spec}_anthropogenic_${year}_0.5x0.5*nc .

   dmget /archive/g08/dmkoch/ftp/AR5/IPCC_emissions_${spec}_ships_${year}_0.5x0.5*nc
   cp /archive/g08/dmkoch/ftp/AR5/IPCC_emissions_${spec}_ships_${year}_0.5x0.5*nc .

  ./remap.pl -par ncregrid-ijl.par -in IPCC_emissions_${spec}_anthropogenic_${year}_0.5x0.5*nc -out IPCC_emissions_${spec}_anthropogenic_${year}_C90_Dec_2009.nc

  ./remap.pl -par ncregrid-ijl.par -in IPCC_emissions_${spec}_ships_${year}_0.5x0.5*nc -out IPCC_emissions_${spec}_ships_${year}_C90_Dec_2009.nc

  cd /gpfsm/dnb53/gfaluveg/AR5_emissions/v5_anthro
  rm -f AR5.bat
  ./make_AR5_program_${res}.ksh ${spec} ${year} anthropogenic C90_Dec_2009 dom ind wst awb tra
  echo ".com convert_${spec}.pro" >> ./AR5.bat
  echo ".run convert_${spec}.pro" >> ./AR5.bat
#now run the idl batch file:
[...]

Hoo boy… so it grabs a bunch of stuff from “/archives”. But we don’t have a “/archives”. Looks like their “usual dodge” of release the model but not the parameter files driving it; at least at first glance. “We’ll see” as I dig into it in the coming weeks (months? years?).

But wait, there’s More!

Looking at aux/regrid.par (the .par got me wondering if it was more parameters…) we find:

filedir=/discover/nobackup/projects/giss/prod_input_files/
regridfile=remap288-180C90-90.nc
imsource=288
jmsource=180
ntilessource=1
imtarget=90
jmtarget=90
ntilestarget=6
format=ij
nfields=1
title=yes

Yup. MORE parameters. For one it points to “Yet Another Missing Directory” in that /discover top level qualifier. Then sets about setting more parameters. How many .par files are there?

chiefio@Headend:/Climate/modelE2_AR5_branch$ ls aux/*.par
aux/ICOxref.par       aux/ncregrid.par          aux/regridNOxnatural.par
aux/ICgastracers.par  aux/regrid.par            aux/regridc1x1.par
aux/ICsulfate.par     aux/regrid1x1Q.par        aux/regride2x2.5.par
aux/ncregrid-ijl.par  aux/regridCH4natural.par

Hopefully they are only used in the post-processing… yet even that determines the results as presented to the world.

About That Makefile

Seems that when you look into it, there’s a lot more to do than that top level Makefile. In the config directory, we find:

chiefio@Headend:/Climate/modelE2_AR5_branch$ ls config
ESMF.default.mk     compiler.gfortran.mk  machine.IRIX.mk    mpi.mpich2.mk
base.mk             compiler.intel.mk     machine.IRIX64.mk  mpi.mpt.mk
compiler.IRIX.mk    compiler.intel_7.mk   machine.Linux.mk   mpi.mvapich.mk
compiler.IRIX64.mk  compiler.nag.mk       machine.OSF1.mk    mpi.mvapich2.mk
compiler.Lahey.mk   compiler.pgi.mk       mpi.IRIX64.mk      mpi.openmpi.mk
compiler.OSF1.mk    compiler.xlf.mk       mpi.LAM.mk         rules.mk
compiler.absoft.mk  machine.AIX.mk        mpi.SCALI.mk
compiler.g95.mk     machine.Darwin.mk     mpi.intel.mk

Now the really good thing is that this ought to have a custom .mk make file for most any kind of machine you want to run this on. But it does show that the “make” is going to be modestly complicated.

Looking in the compiler.g95.mk file as one example:

F90 = g95
FMAKEDEP = $(SCRIPTS_DIR)/sfmakedepend
CPPFLAGS += -DCOMPILER_G95
FFLAGS = -cpp -fno-second-underscore -O2 -fendian=big
F90FLAGS = -cpp -fno-second-underscore -O2 -ffree-form -fendian=big
LFLAGS =
# uncomment next two lines for extensive debugging
# the following switch adds extra debugging
ifeq ($(COMPILE_WITH_TRAPS),YES)
FFLAGS += -fbounds-check -ftrace=full -freal=nan -fpointer=invalid
LFLAGS += #-lefence
endif

We’ve got compiler flags set to adjust the compiler behaviour. Normally this would only interest the developer of a code, but since every run is a compile, these can adjust how the code behaves when run.

Similarly, the machine.Linux file sets things for that environment:

# Linux - specific options

CPP = /lib/cpp -P -traditional
CPPFLAGS = -DMACHINE_Linux
ECHO_FLAGS = -e

cpp is the C Pre-Processor and it has flags to change how it works.

Then you get to the MPI Message Passing Interface setting blocks where you can choose and tune which kind of MPI you are using.

Here’s the one for OpenMPI:

ifneq (${MPIDIR},)
FFLAGS += -I${MPIDIR}/include
F90FLAGS += -I${MPIDIR}/include
LIBS += -L${MPIDIR}/lib
endif

# try to work around memory leak
CPPFLAGS += -DMPITYPE_LOOKUP_HACK

#LIBS += -lcprts -limf -lm -lcxa -lunwind -lrt -ldl \
#-lfmpi -lmpi -lstdc++ -threads

##LIBS += -lcprts -limf -lm -lcxa -lunwind -lrt -ldl \
##-lmpi_f90 -lmpi_f77 -lmpi -lstdc++ -threads

LIBS += -lmpi_f77 -lmpi -lmpi_cxx -lstdc++
ifneq ($(shell uname),Darwin)
  LIBS += -lrt
endif

#LIBS +=  -lmpi -lmpigc3 -lmpigc4 -lmpigf -lmpigi -lmpiic4 -lmpiic -lmpiif -threads

That “try to work around memory leak” isn’t encouraging…

In any case, this is relatively normal for setting up an environment to port and compile a program. What is unusal about it is that it is needed for every run.

But that isn’t enough… they have a certain degree of ‘cross linking’ between the various make files. You find that in rules.mk and base.mk:

base.mk:

# standard makefile to be use in each "component" subdir

.PHONY: FORCE

#get defaults from ~/.modelErc
HOMEDIR = $(wildcard ~)
MODELERC ?= $(HOMEDIR)/.modelErc
sinclude $(MODELERC)

#some of modelE definitions
#SCRIPTS_DIR = $(GISSCLIM_DIR)/scripts
#MOD_DIR = $(GISSCLIM_DIR)/mod
#LIB_DIR = $(GISSCLIM_DIR)/lib

#hack : make sure that ESMF_Interface is set
#ESMF_Interface = .
[...]

and rules.mk:

#
# this file contains rules shared by Makefiles in "model" and "aux"
#

.PHONY:
ifdef MOD_DIR
VPATH = $(MOD_DIR)
endif

#VPATH = $(MODULES_CMPLR_OPT)
#VPATH = /usr/local/unsupported/esmf/1.0.3-551/include
######  Some user customizable settings:   ########

# EXTRA_FFLAGS specifies some extra flags you want to pass
# to Fortarn compiler, like
# -g        - include debugging information
# -listing  - create listings (.L)
EXTRA_FFLAGS =

# EXTRA_LFLAGS specifies some extra flags you want to pass
# to linker. Currently needed as a hack to compile hybrid MPI/OpenMP
# code to pass "-openmp" to linker
EXTRA_LFLAGS =

# hack to force compilation errors to be written to ERR files rather than scren
ifeq ($(OUTPUT_TO_FILES),YES)
  COMP_OUTPUT = > $*.ERR 2>&1 || { r=$$? ; cat $*.ERR ; exit $$r ; }
  LINK_OUTPUT = > $(RUN).ERR 2>&1 || { r=$$? ; cat $(RUN).ERR ; exit $$r ; }
else
  COMP_OUTPUT =
  LINK_OUTPUT =
endif
[...]

Oh, it says there’s another batch of make files in ‘model’… Oh Joy…

chiefio@Headend:/Climate/modelE2_AR5_branch/model$ ls tracers/
Makefile  Tracers_mod.F90
chiefio@Headend:/Climate/modelE2_AR5_branch/model$ ls solvers
Makefile  TRIDIAG.f
chiefio@Headend:/Climate/modelE2_AR5_branch/model$ ls shared
CONST.f                 Makefile                   StringUtilities_mod.F90
Dictionary_mod.F90      PARAM.f                    SystemTimers_mod.F90
FileManager_mod.F90     PARSER.f                   UTILDBL.f
GenericType_mod.F90     Parser_mod.F90             array_bundle.f
Geometry_mod.F90        RunTimeControls_mod.F90    cubic_eq.f
JulianCalendar_mod.F90  RunTimeControls_mod.m4F90  stop_model.F90
KeyValuePair_mod.F90    SYSTEM.f
chiefio@Headend:/Climate/modelE2_AR5_branch/model$ ls -l Makefile
-rw-r--r-- 1 chiefio chiefio 3400 Dec 30  2015 Makefile
chiefio@Headend:/Climate/modelE2_AR5_branch/model$ ls profiler
Makefile               TimeFormatUtilities_mod.F90  Timer_mod.F90
ProfileReport_mod.F90  TimerList_mod.F90
ReportColumn_mod.F90   TimerPackage_mod.F90
[...]

and several other directories with their own Makefile in them. Yet even that isn’t enough complexity. Seems you can choose what make files get used sometimes.

Looking in ‘doc’, we have this in Ent.txt

Working with Ent DGTEM (Dynamic Global Terrestrial Ecosystem Model)
===================================================================

This document describes how to checkout, compile and run modelE
coupled to Ent DGTEM.
[...]
Compiling the code
==================

Since in this new setup some of the code is located in subdirectories
the Makefile had to be modified to be able to work with
subdirectories. Basically a completely different makefile was written
for this purpose. To switch to this new makefile you have to specify

  NEW_MAKEFILE_FORMAT=YES

in your ~/.modelErc file. This new makefile expects two additional
sections to be present inside the rundeck. Section "Components:" lists
the names of the subdirectories (from now called "Components")
containig the code which has to be compiled with the model. The
section "Component Options:" contains the flags which have to be passed
to each component (i.e. the flags which have to be set when local
makefile is executed inside corresponding subdirectory). Each
component (subdirectory) contains a makefile which in most cases is
trivial: it contains the list of source files and includes standard
template makefiles from config/ directory (see model/shared/Makefile
for an example). If rundeck should be able to modify the list of
source files in the component that can be done by passing
corresponding "Component Options:" (see how it is done inside
model/Ent/Makefile). Current template rundeck for modelE coupled to
Ent is E1M20dv1.R (see it as an example of how to use "Components:"
and "Component Options:"). The commands supported by the new makefile
are basically the same with some small differences. To compile and set
up a typical modelE+Ent run you would do:

# remove old object files if necessary (old "vclean")
  make clean_all

# create a rundeck
  make rundeck RUN=my_run RUNSRC=E1M20dv1

# build dependencies
  make depend RUN=my_run ESMF=YES NPES=1

# compile and do setup
  make setup RUN=my_run ESMF=YES NPES=1

So all this environment fiddling ends up happening on each ‘compile and run’ of a job.

Interestingly, they left in some ‘local notes’ in the ‘local’ file: local_info.txt

I’ve bolded a bit where it shows why I talk about libraries and compile flags and such.

Running on Discover (Reto Ruedy)
+++++++++++++++++++
News:
====
[...]
Unresolved bugs:
===============
- occasionally, jobs tend to "hang" for extended period of times,
  usually during i/o processes (creation of PRT files)
     hint: for long runs use "kdiag=13*9," not "kdiag=13*0,"

- serial and parallel versions do not always give the same result
     hint: see next section (Consistency ...)

- intelmpi: may be used on all current and future nodes

- scali mpi: can only run on the relatively few old 4-cpu nodes

- openmpi: may be used on all nodes but is not recommended (use intelmpi)
   - compile with: EXTRA_FFLAGS="-DMPITYPE_LOOKUP_HACK" to avoid memory leak
   - Bundling openmpi jobs is currently impossible

Compiler issues:
===============
For off-line fortran compilations use:
   ifort -convert big_endian x.f -o x.exe

Before doing so, load the appropriate modules, currently:
   /etc/profile.d/modules.sh             ; # bash or ksh or:
   source /etc/profile.d/modules.csh     ; # tcsh or csh
and either
   module load comp/intel-9.1.042
   module load mpi/scali-5               ; # for scali mpi (on borga-e...)
or
   module load comp/intel-9.1.052        ; # for openmpi
   module load mpi/openmpi-1.2.5/intel-9
or
   module load comp/intel-10.1.017       ; # for intel mpi
   module load mpi/impi-3.2.011

This is to be put into your startup script (e.g. .profile if your login shell
is ksh), so these modules also get loaded when running in batch mode.
"runpbs (=run)" was modified to load the appropriate modules for mpi batch jobs
even if those commands are missing in the login script. But you still need to set
them appropriately during the "setup" step.

Sheesh… different contexts / compiles give different results and we swap around MPI libraries at compile time based on what we like or what what works today or…

All that BEFORE you get to the massive parameter setting inside the model!

Consistency between serial and parallelized executables:
-------------------------------------------------------
openMP and serial runs give the same results ONLY if using the
   option -mp (machine precision) in both cases, i.e. use
   gmake setup|exe RUN=.... EXTRA_FFLAGS=-mp EXTRA_LFLAGS=-mp
Warning: The current version of modelE no longer supports openMP

mpi and serial runs (without the -mp options) give identical results
   (but different from serial runs and openMP runs with -mp)

Um, maybe you ought to figure out why the results vary with different MPI context and just fix the code?… Just sayin’…

Having code that is significantly ‘context of run’ or ‘context of build’ sensitive is just not making my “Oooh that’s reliable” warm fuzzy feeling happen…

There’s lots more interesting in that file (and in other files) but this posting is too long already, so most of it will need to wait for “another day”. I’ll just put this interesting snip here as a parting tease:

For serial runs, no options are needed; however if you want results identical
to that produced by an openMP executable, add EXTRA_FFLAGS=-mp EXTRA_LFLAGS=-mp

NOTE: When changing options, make sure to use     "gmake clean vclean"
      to get rid of old object modules and avoid incompatibilities.

Yeah, if you skip a ‘make clean vclean’ you can have incompatibilities…

There are lots of other parameter settings described in other .txt files. Here’s just a sample from the oceans biology one. Note: This is NOT the “oceans”, just the oceans biology obio.txt:

Prescribed and Interactive Ocean Biology in modelE
(by Natassa Romanou, Watson Gregg and Gavin Schmidt)
=========================================

This document describes the steps and methodology for
running (non)interactive chlorophyll and interactive carbon
cycle etc.
This document describes the steps and methodology for
running (non)interactive chlorophyll and interactive carbon
cycle etc.

step 1: Check out the model from CVS as usual
cvs checkout modelE

step 2: Choose which ocean model to use if any at all

step 3: Choose whether you want to do
        i) chlorophyll-radiation feedbacks
       ii) real or prescribed chlorophyll and ocean biology
      iii) CO2 gas exchange with the atmosphere
       iv) prognostic alkalinity

Then in the decks directory use the appropriate rundeck.
=========================================

Description of CPPs to be used in the rundecks

#define CHECK_OCEAN_HYCOM           ! HYCOM ocean: needed for CMPE002 only
#define HYCOM_RESOLUTION_2deg       ! HYCOM ocean: use 2deg resolution for HYCOM ocean

#define CHECK_OCEAN                 ! RUSSELL ocean: needed to compile aux/file CMPE002

#define TRACERS_OCEAN               ! RUSSELL ocean: ocean tracers activated
#define TRACERS_OCEAN_INDEP         ! RUSSELL ocean: independently defined ocn tracers

#define OBIO_ON_GARYocean           ! RUSSELL ocean: prognostic ocean biology
#define CHL_from_SeaWIFs            ! RUSSELL ocean: read in SeaWIFs

#define TRACERS_OceanBiology        ! ANY ocean: Watson Gregg's ocean bio-geo-chem model

#define OBIO_RAD_coupling           ! ANY ocean: radiation -- ocean biology coupling

#define pCO2_ONLINE                 ! ANY ocean: pCO2_seawater computed online

#define TRACERS_ON                  ! ANY ocean: ATMOS tracers: include tracers code

#define TRACERS_GASEXCH_ocean       ! ANY ocean: special tracers to be passed to ocean
#define TRACERS_GASEXCH_ocean_CO2   ! ANY ocean: special tracers to be passed to ocean

#define CHL_from_OBIO               ! ANY ocean: interactive CHL

#define TRACERS_GASEXCH_CFC_Natassa ! ANY ocean: CFC-11 or CFC-12

#ifdef TRACERS_Alkalinity           ! ANY ocean: prognostic alkalinity

In Conclusion

What are in those rundecks? Rundeck.txt tells us:

Rundeck
===================================================================

A rundeck (a file with an extension .R) is a file which contains
a complete description of a particular model run, including
the description of model code used and run-time parameters.
Directory modelE/templates contains typical rundecks which
can be used as examples.


Rundeck structure
================

Rundeck consists of a number of sections describing different
aspects of the run. Each section starts with certain keywords
(like "Object modules:") and terminates either with "End ..."
statement or with the start of a new section. The sections
are supposed to follow in a pre-defined order and can't be
interchanged, though some unneeded sections can be skipped.
The character ! starts a comment. Everything to the right of !
until the end of the line is ignored. Here is the list
of rundeck sections in proper order:

 * Run name and comment
 * Preprocessor Options
 * Run Options
 * Object modules
 * Components
 * Component Options
 * Data input files
 * Label and Namelist
 * Run-time parameters
 * Restart and timing parameters

Yup. From preprocessor to compile and choosing object modules to lad and more… Decisions decisions. Just what tune to play and how do I want this elephant dancing while playing the trombone?…

* Components
============

Starts with a line

Components:

This section lists all "Components" (the subdirectories of
modelE/model) which have to be compiled with the executable. Keep in
mind that each Component will be compiled into a library which will be
then linked with the main executable. As a result in a case of a name
conflict (more than one subroutine with the same name is present) the
preference will be given to the one located in modelE/model
directory.


* Component Options
===================

Starts with a line

Component Options:

This section lists specific options which have to be passed to certain
components. For each Component the options are listed on a single line
as follows

OPTS_ = = =

where Opt1, Opt2 are variabes to be set and X, Y are corresponding
values. The instruction above is equivalent to setting Opt1=X, Opt2=Y
at the start of  Makefile.

There is a special debugging option OVERWRITE_FSRCS which can be
passed to any component (which uses base.mk). If this option is set to
a list of source files, these files will be used when compiling the
component (instead of the ones specified inside the component
Makefile). The use of this option is discouraged for the production
runs though (which should use the source files specified by the
component itself). Similar option OVERWRITE_F90SRCS can be used to
specify F90 files.

Every run is a unique mix of libraries, compiler options, parameters set, input data choices, model parts chosen to run, and even, if you like, special versions of any given bit of Fortran you might want to swap in…

The choices are endless. The whole thing from soup to nuts is a parameter driven process.

And that’s the whole problem.

Subscribe to feed