I’ve only been into it about 10 minutes, but already I’m liking the MPAS code a LOT more than the stuff from Hansen and that “Hansen Labs” (aka NASA / NOAA / NCDC / etc.)
I was first pointed at this by rogercaiazza in a comment here:
https://chiefio.wordpress.com/2017/11/28/circles-and-hexagons-and-models-oh-my/#comment-88879
So h/t to Roger!
Some specifics:
Building Environment
The “Makefile” lists several compiler and systems choices, largely large… so “some assembly required” to make it go on tiny iron. It does have a gfortran option called out, and an mpif90 option; so mostly finding out what FORTRAN compilers are available and who lines up with whom.
Here’s a couple of sample blocks from the topmost Makefile (that sets the environment for all the rest):
titan-cray: ( $(MAKE) all \ "FC_PARALLEL = ftn" \ "CC_PARALLEL = cc" \ "FC_SERIAL = ftn" \ "CC_SERIAL = gcc" \ "FFLAGS_PROMOTION = -default64" \ "FFLAGS_OPT = -s integer32 -O3 -f free -N 255 -em -ef" \ "CFLAGS_OPT = -O3" \ "LDFLAGS_OPT = -O3" \ "FFLAGS_OMP = " \ "CFLAGS_OMP = " \ "CORE = $(CORE)" \ "DEBUG = $(DEBUG)" \ "USE_PAPI = $(USE_PAPI)" \ "OPENMP = $(OPENMP)" \ "CPPFLAGS = $(MODEL_FORMULATION) -D_MPI -DUNDERSCORE" ) [... skipping some -EMS ] gfortran: ( $(MAKE) all \ "FC_PARALLEL = mpif90" \ "CC_PARALLEL = mpicc" \ "CXX_PARALLEL = mpicxx" \ "FC_SERIAL = gfortran" \ "CC_SERIAL = gcc" \ "CXX_SERIAL = g++" \ "FFLAGS_PROMOTION = -fdefault-real-8 -fdefault-double-8" \ "FFLAGS_OPT = -O3 -m64 -ffree-line-length-none -fconvert=big-endian -ffree-form" \ "CFLAGS_OPT = -O3 -m64" \ "CXXFLAGS_OPT = -O3 -m64" \ "LDFLAGS_OPT = -O3 -m64" \ "FFLAGS_DEBUG = -g -m64 -ffree-line-length-none -fconvert=big-endian -ffree-form -fbounds-check -fbacktrace -ffpe-trap=invalid,zero,overflow" \ "CFLAGS_DEBUG = -g -m64" \ "CXXFLAGS_DEBUG = -O3 -m64" \ "LDFLAGS_DEBUG = -g -m64" \ "FFLAGS_OMP = -fopenmp" \ "CFLAGS_OMP = -fopenmp" \ "CORE = $(CORE)" \ "DEBUG = $(DEBUG)" \ "USE_PAPI = $(USE_PAPI)" \ "OPENMP = $(OPENMP)" \ "CPPFLAGS = $(MODEL_FORMULATION) -D_MPI -DUNDERSCORE" ) gfortran-clang: ( $(MAKE) all \ "FC_PARALLEL = mpif90" \ "CC_PARALLEL = mpicc -cc=clang" \ "CXX_PARALLEL = mpicxx -cxx=clang++" \ "FC_SERIAL = gfortran" \ "CC_SERIAL = clang" \ "CXX_SERIAL = clang++" \ "FFLAGS_PROMOTION = -fdefault-real-8 -fdefault-double-8" \ "FFLAGS_OPT = -O3 -m64 -ffree-line-length-none -fconvert=big-endian -ffree-form" \ "CFLAGS_OPT = -O3 -m64" \ "CXXFLAGS_OPT = -O3 -m64" \ "LDFLAGS_OPT = -O3 -m64" \ "FFLAGS_DEBUG = -g -m64 -ffree-line-length-none -fconvert=big-endian -ffree-form -fbounds-check -fbacktrace -ffpe-trap=invalid,zero,overflow" \ "CFLAGS_DEBUG = -g -m64" \ "CXXFLAGS_DEBUG = -O3 -m64" \ "LDFLAGS_DEBUG = -g -m64" \ "FFLAGS_OMP = -fopenmp" \ "CFLAGS_OMP = -fopenmp" \ "CORE = $(CORE)" \ "DEBUG = $(DEBUG)" \ "USE_PAPI = $(USE_PAPI)" \ "OPENMP = $(OPENMP)" \ "CPPFLAGS = $(MODEL_FORMULATION) -D_MPI -DUNDERSCORE" )
The g95 spec also includes mpif90 as the parallel fortran compiler, and I didn’t find one with a simple “apt-get” so more poking around needed (could be just some odd name with release level appended or ‘whatever’…)
The code itself ends with .F and Linux seems to think they are just ascii files, so one assumes the nature of modern fortran has moved far enough from the original for the “file” command to not recognize it (or it needs .f ending).
Also, all the endian settings list “big”, so what to do on a little-endian machine?… One hopes the compiler is flexible enough to deal with that.
The Code
The code, from a very small sample, looks very well written and well commented. As a “community project”, I’d expect that since “many hands” means many eyes needing to find clue…
From:
chiefio@odroidxu4:~/MPAS/MPAS-Release-5.2/src/core_atmosphere$ h vi physics/mpas_atmphys_functions.F
An example:
! Copyright (c) 2013, Los Alamos National Security, LLC (LANS) ! and the University Corporation for Atmospheric Research (UCAR). ! ! Unless noted otherwise source code is licensed under the BSD license. ! Additional copyright and license information can be found in the LICENSE file ! distributed with this code, or at http://mpas-dev.github.com/license.html ! !================================================================================================================= module mpas_atmphys_functions implicit none private public:: gammln,gammp,wgamma,rslf,rsif contains !================================================================================================================= !NOTE: functions rslf and rsif are taken from module_mp_thompson temporarily for computing ! the diagnostic relative humidity. These two functions will be removed from this module ! when the Thompson cloud microphysics scheme will be restored to MPAS-Dev. ! Laura D. Fowler (birch.mmm.ucar.edu) / 2013-07-11. !+---+-----------------------------------------------------------------+ SUBROUTINE GCF(GAMMCF,A,X,GLN) ! --- RETURNS THE INCOMPLETE GAMMA FUNCTION Q(A,X) EVALUATED BY ITS ! --- CONTINUED FRACTION REPRESENTATION AS GAMMCF. ALSO RETURNS ! --- LN(GAMMA(A)) AS GLN. THE CONTINUED FRACTION IS EVALUATED BY ! --- A MODIFIED LENTZ METHOD. ! --- USES GAMMLN IMPLICIT NONE INTEGER, PARAMETER:: ITMAX=100 REAL, PARAMETER:: gEPS=3.E-7 REAL, PARAMETER:: FPMIN=1.E-30 REAL, INTENT(IN):: A, X [...] ! (C) Copr. 1986-92 Numerical Recipes Software 2.02 !+---+-----------------------------------------------------------------+ SUBROUTINE GSER(GAMSER,A,X,GLN) ! --- RETURNS THE INCOMPLETE GAMMA FUNCTION P(A,X) EVALUATED BY ITS ! --- ITS SERIES REPRESENTATION AS GAMSER. ALSO RETURNS LN(GAMMA(A)) ! --- AS GLN. ! --- USES GAMMLN IMPLICIT NONE INTEGER, PARAMETER:: ITMAX=100 REAL, PARAMETER:: gEPS=3.E-7 REAL, INTENT(IN):: A, X REAL:: GAMSER,GLN INTEGER:: N REAL:: AP,DEL,SUM GLN=GAMMLN(A) IF(X.LE.0.)THEN IF(X.LT.0.) PRINT *, 'X < 0 IN GSER' GAMSER=0. RETURN ENDIF AP=A SUM=1./A DEL=SUM DO 11 N=1,ITMAX AP=AP+1. DEL=DEL*X/AP SUM=SUM+DEL IF(ABS(DEL).LT.ABS(SUM)*gEPS)GOTO 1 11 CONTINUE PRINT *,'A TOO LARGE, ITMAX TOO SMALL IN GSER' 1 GAMSER=SUM*EXP(-X+A*LOG(X)-GLN) END SUBROUTINE GSER ! (C) Copr. 1986-92 Numerical Recipes Software 2.02
I jumped down to where they included a proprietary subroutine, yet it is under BSD license. Go figure.
Yet the style is easy to follow and the general purpose set out in comments. Nice.
So I’m going to wander through more of this code before I say much more about it. This is just the first blush and who knows what’s under the covers. What I can say with certainty is that open projects with many hands and eyes benefit greatly from the corrective efforts of the group. Nobody wants to explain their stupid human trick more than once to a Noob who spotted it ;-)
How to get it
They have a place where you can download a zip file for those of you not on *nix systems. Then there’s a place to fill in a sign-up for what you are doing with it. Nicely, you can skip the sign up (since I’m just a looky lou it’s a waste of everyone’s time at this point) and do a direct tarball download (for us *nix folks).
The direct download (both zip and tarball)
https://github.com/MPAS-Dev/MPAS-Release/releases
Directions and helpful info they want you to read, fully, before the download. (I’ll get to it IF I ever start to build / compile the thing ;-) Mostly it looks like help to push folks into using git, which is fine for folks who know git, or do systems work regularly; but not so fine if you just want to unpack a tarball and look it over…
https://mpas-dev.github.io/model.html
The request for a “registration”:
https://mpas-dev.github.io/registration.html
The top level of the download tree, which includes links that have names like “sample imput files”, oh the joy!…
http://mpas-dev.github.io/atmosphere/atmosphere_download.html
In Conclusion
I don’t know that this is the place I’ll be happy, but it sure is less musty and opaque than the alternatives to date. There’s also just a feeling like folks who think of scaling hexagonal grids have clue and like a tidy mind…
Time will show if first impression stand up, but so far I’m impressed.
hmmm…. This says you get mpif90 when you install openmpi:
https://www.linuxquestions.org/questions/linux-newbie-8/mpif90-missed-838887/
and do a configure and make of it…
SO one mystery semi-solved…
Configure system. Add / build OpenMPI. Check mpif90 exists. Use it in Makefile.
OK, got it…. (until the first bug bites… ;-)
Here’s good background on SCVTs. (The tesselations)
http://diginole.lib.fsu.edu/islandora/object/fsu:169024/datastream/PDF/view
Glad you like it
Nice find. I’m not sure that I will do much with it other than, as ChiefIO says, just look at it.
It has MPI processing built in, but not the use of coarray FORTRAN near as I can tell:
Which makes a lot of sense. It is mostly written a few years back and Fortran 2008 spec (with coarrays) was only released in 2010, so not many compilers in 2012 or so…
It might (or might not!) offer an ability to add some coarray processing to some routines as a further speed enhancement on some kinds of clusters.
Near as I can tell, there is a coarray Fortran available on the R. Pi, though I have no idea if it implements actual distributed coarray processing (often, for new features like this, the syntax is allowed but it gets mapped onto older functional ways… )
MPI on the Pi was slower than direct execution in my tests. Perhaps it needs improving on the Pi, or perhaps my tests were not “suited” to MPI methods. Some more testing required… BUT, it may well be that even IF the Pi is ill suited to MPI, there are many other boards that might be better suited. The Odroids, for example. Things with GigE and more attention to I/O. (Network / communications overhead hurts message passing based distributed processing) or perhaps a more distributed friendly implementation.
In any case, having it already set up for parallel processing is very very nice. Complicates understanding it a little, but means that running it across a cluster (whatever the efficiency) is already baked in.
I think I found where it starts…
One of hte odd bits of many big programs is just trying to find where, in all that source code, is the start of it all. Well, in the source directory in the driver directory is this little program called mpas.F that looks like the top of it all.:
Can’t get more “meat and potatoes” than “initialize, run, finalize/stop” ;-)
It does call a mpas_subdriver basket of actions, so digging into that will expose the actual activity. It’s in the same directory and isn’t very long.
So “only” 448 lines (much of it, thankfully, comments! Yeah MPAS Programmers!!)
Mostly just breaking out the steps of air vs ocean vs…
Here’s a snip of it:
So some environment variables checked to decide what parts to run (so where are things like CORE_OCEAN set?…) then it includes that code if needed and it calls / runs the bits as needed. Also sets up what looksl like a compute cluster ( at least, that’s my guess at the moment as to what a “domain” is in this context… I hate the way computing reuses some words for dozens of different things, and “domain” is one of them… so “some digging required” to figure out exactly what mpas folks mean by “domain”…)
In general, it looks like starting from here you can decompose the “tree” of processing actions and function names / meanings. From the top level “just do it” to the “do ocean and air and…” down into what the particular parts of each actually do.
I again note in passing that even this model seems very interested in land surface (dirt), the oceans (big water), and the atmosphere… but not very interested in water vapor as a driver. It is computed along the way in what (so far at least, at a VERY early stage of reading the code) looks like an effect more than a driver. However, as these models are many long cycles, what’s an effect in one loop becomes an initial state in the next, so can become a driving element. One must reach the level of understanding the carry forward / loop processing before dismissing an ‘effect’ as not being a ‘driver’…
Yet even the high level view from the README file has that focus:
Atmosphere, Land Ice, Ocean. Each a distinct “core”…
Where is the integrated water cycle as driver?
One hopes it is “hidden in the details”… and not just MIA.
Now, from my POV, their ought to be a Solar Module that accounts for solar variation over various time scales, and a global season / tilt / precession module to vary the Earth mode of interaction with that solar power source, then a “Water Cycle” major wrapper that generates the water state in the oceans, atmosphere and at the surface of the dirt. At that point you can run it forward for a cycle and get the various mass flows, cloud formation, precipitation events, ice melts or builds, and finally implicit temperatures, finally do an atmosphere radiative net effects; and then repeat the “water cycle” again. After a full day, increment the calendar. After a week or a month, update the solar state and seasonal tilt effects.
It would also be nice to see a lunar tidal module driving ocean effects, but I guess it’s a bit much to ask…
Oh Well. It’s MUCH better than the Model II and Model E stuff, IMHO. It just has that well structured well thought out organized feel to it. Sure, it’s still in the mode of “kit of parts” where you can choose to have an ocean, or not. Yeah, it’s still air focused water as afterthought. But it’s just nicely built. It will be interesting to try to compile it and make it go in a mini-cluster ;-)
Hey, if you are not willing to abuse hardware by loading it to 100% with code designed for a 100 x larger machine, you ought not to be playing with HPC and clusters anyway! After all, most of the problems left for “supercomputers” to solve are by definition too big for the available hardware…
Hmmm…. From the INSTALL top level file that tells how to compile it… it looks like both MPI and OpenMP (shared memory) are used. OpenMP inside one system of cores, MPI between systems…
From the wiki:
https://en.wikipedia.org/wiki/OpenMP
So not only parallel, but well thought out multiple layer parallel… though that makes compiler choice and install process more tricky…
I think I need to take a side road wander into CLANG, and Devuan / Debian support for OpenMP and MPI…
Hey Larry! It has some sublimation in it!
EMs-MacBook-Air:MPAS-Release-5.2 chiefio$ grep ublim src/*/*.F src/*/*/*.F
src/core_init_atmosphere/mpas_atmphys_constants.F: real(kind=RKIND),parameter:: xls = xlv + xlf !latent heat of sublimation [J/kg]
src/core_atmosphere/physics/mpas_atmphys_constants.F: real(kind=RKIND),parameter:: xls = xlv + xlf !latent heat of sublimation
Cool good to see they did not miss that.
Pretty nice looking code for a change :)
I’ve looked through the convection code. It’s pretty detailed. Looks generally well thought out. My only concern about it would be the degree of “choice” in it. There are lots of knobs to set and a couple of major modes of convective activity. Who knows how to chose the one that’s like reality.
It’s down in:
MPAS-Release-5.2/src/core_atmosphere/physics
in a program named:
mpas_atmphys_driver_convection.F
Has a Tiedtke and a New Tiedtke choice, each with lots of parameters. One “Laura D. Fowler” does good work and writes nice comments. Wish there were more like her writing code.
So when it calls one of these, how many parameters get passed?
Um “lots”… It will take a while to figure out what all these are and what they do and what the control knobs are set to and what that is like compared to reality and…
I don’t know why, but there’s just something about the way this code is all structured and written that leads me to think they have been careful to “get it right”. My only uneasy spot is over the question of parameter setting. I could easily see setting convection too low, water vapor too low, and then juicing the CO2 sensitivity to make up for it on the hindcasting.
I may just proceed to seeing if this thing can be compiled and / or run on “small iron” and then go looking for parameter fudge. I’m certainly not going to find much bogus in the way the code is written and structured. It’s just very well done.
I do wonder a bit about the number of nested calls and what that might do to efficiency, OTOH, that’s an “old school” worry and it looks like toward the bottom layers they bust out in multiple threads, so likely the call overhead isn’t adding all that much, really. It does help clarity of understanding and I’d hate to try to unwind and inline any significant percentage of the calls and still have it at all something you could understand…
Pingback: Weather Without Mountains Averaged Gives Climate Without Reality | Musings from the Chiefio