[Ncep.list.nems.announce] Upcoming commits: new NEMS build system and ESMF 8 support

Thu Oct 18 16:08:16 UTC 2018

Hi all,

I have gotten some excellent feedback on the new NEMS build system and made
the necessary changes.  I'm waiting for one FV3-MOM6-CICE5 commit that will
go in soon.  Then I'll start run the big, multi-machine, multi-app test,
one last time before committing.  I'm hoping to commit Friday afternoon,
but it depends in part on queue time and on the timing of the
FV3-MOM6-CICE5 commit.

You can expect another build system commit in a week or so, to add some
portability changes.

Sincerely,
Sam Trahan

On Tue, 9 Oct 2018 at 13:00, Samuel Trahan - NOAA Affiliate <
samuel.trahan at noaa.gov> wrote:

> NEMS developers,
>
> There are two commits coming into the NEMS git master soon.  We're
> expecting to commit them early next week, hopefully Monday.
>
> 1. Support for new ESMF 8 beta snapshots which break backward
> compatibility to ESMF 7.x and earlier.  This requires adding an #ifdef in
> NEMS code that triggers on the ESMF version.  The NEMS build system cannot
> support this, so we need a new NEMS build system.
>
> 2. A new NEMS build system.  It re-expresses the NEMSAppBuilder in GNU
> Make.
>
> We're working on merging change #1 into change #2, and we plan on
> committing them at the same time.
>
> The remainder of this email details the new NEMS build system.
>
> Sincerely,
> Sam Trahan
>
>
> *--------------------------------------------------------------------------------------------------------------------------------------------------------------*
>
> *The New NEMS Build System*
>
> *Overview*
>
> This is not the next-generation build system that will save the world; it
> is just a cleaned-up version of the old one, re-expressed in GNU Make.  The
> NEMSAppBuilder GUI is gone, but other than this, the front-end is
> backward-compatible.  You will not notice any changes to the build system
> if you're just building or running the NEMS or its Supported apps via the
> supported methods.  Specifically, the NEMSCompsetRun, NEMSAppBuilder,
> NEMSfv3gfs/tests, and NEMSGSM/oldtests will all work just like they did
> before.  This change is mostly to benefit those of us that do need to
> access or modify the internals of the NEMS build system.  It is intended to
> be easier to work with and easier to port than the old one.
>
> You will only be impacted by this commit if you have your own build
> system, if you use a component that is not in a Supported app, or if you
> intend to modify or port the build system.  For example, if you want to add
> components or change the component build rules, you will need to know the
> internals of the new system.  Also, the NEMS/src/configure and
> NEMS/src/makefile are gone.  The new build system moves the logic of
> NEMSAppBuilder into a top-level GNUmakefile and src/incmake/*.mk files for
> details of the build.  A description of the step-by-step logic, and purpose
> of the lower-level makefiles, can be found in the top-level GNUmakefile.
>
> The build system is only fully tested for Supported apps and their target
> platforms, listed below.  That means some actively-developed components and
> platforms are not tested.  All components are present in the build system,
> and were converted from shell to GNU make.  The platform logic should be
> able to handle any arbitrary platform, but only certain platforms are
> tested.  We will push to the git master before the non-Supported features
> are tested, so you can expect a few more commits over the coming weeks to
> fix any non-Supported features.
>
> I would like to re-iterate that the NEMSAppBuilder GUI mode is gone
> forever, by request of EMC management.  There are superior GUI build
> systems and Integrated Development Environments (IDE) available.  I prefer
> Emacs, but saner people seem to like Eclipse, which can be run on most
> (all?) NOAA clusters.  Now that the build system is written in Make, your
> favorite IDE will be better able to understand it.
>
> Read on for details...
>
>
> *Where to get the New NEMS Build System*
>
> You can find the new build system in the "rewrite-appbuilder" branches of
> these apps:
>
> git clone -b rewrite-appbuilder gerrit:NEMSfv3gfs
> git clone -b rewrite-appbuilder gerrit:EMC_GSM-MOM5-CICE5
> git clone -b rewrite-appbuilder gerrit:EMC_NEMSGSM
> git clone -b rewrite-appbuilder gerrit:EMC_FV3-WW3
> git clone -b rewrite-appbuilder gerrit:EMC_ATM-WW3
> git clone -b rewrite-appbuilder gerrit:EMC_HYCOM-GSM-CICE
> git clone -b rewrite-appbuilder gerrit:EMC_FV3-MOM6-CICE5
>
> The NEMS branch is rewrite-appbuilder for all apps except GSM-MOM5-CICE5.
> That app has fallen behind and cannot use the latest NEMS.  There is a
> rewrite-appbuilder-GSM-MOM5-CICE5 branch for that app's NEMS submodule.
>
>
> *What is Tested, Untested, or Removed?*
>
> This is what has been tested and confirmed to work:
>
>   - theia.intel, t/u jet, WCOSS Phase 1 & 2, WCOSS Cray, GAEA C3
>   - These apps: NEMSfv3gfs, GSM-MOM5-CICE5, NEMSGSM, FV3-WW3, ATM-WW3,
> HYCOM-GSM-CICE, and FV3-MOM6-CICE5
>   - These components: FV3, FMS, GSM, MOM5, MOM6, CICE5, WW3, HYCOM
>   - NEMSAppBuilder
>   - NEMSCompsetRun
>   - tests/compile.sh
>   - tests/rt.sh
>   - direct build via "cd NEMS ; make ... options ..."
>   - Building FV3 with CCPP, but not running it.
>
> This is being tested now:
>
>   - NEMSfv3gfs on WCOSS Dell -- we have no NEMSfv3gfs regression test
> system here yet, and no other apps target this platform
>   - NEMSGSM/oldtests/rt.sh -- works, except for GOCART
>   - GSM GOCART support
>
> This is untested:
>
>   - Running CCPP to test if its build is correct
>   - PGI & GNU compilers
>   - Mac OS/X
>   - Clusters outside of NOAA (Odin, Cheyenne, etc.)
>   - These components' build rules have been converted to the new build
> system but are untested: DATM, GSDCHEM, IMP, IPE, KISS, LIS, NOAH, NOAHMP,
> WRFHYDRO.
>   - Apps that are not Supported (other than GSM-MOM5-CICE5)
>
> This is intentionally removed:
>
>   - GUI NEMSAppBuilder.  This is permanently removed by request of EMC
> management.  May it rest in peace.
>
> This is moved or renamed:
>
>   - NEMS/src/configure is gone; configuration is done via "make configure"
>   - NEMS/src/makefile is now NEMS/src/GNUmakefile
>   - NEMS/src/ENS_Cpl/makefile is now NEMS/src/ENS_Cpl/GNUmakefile
>   - NEMS/src/conf/configure.nems.NUOPC is not in the repository.  It is
> now automatically generated from configure.nems.NUOPC.in, the
> component_*.mk files, and the component list.
>
>
>
> *Major Features and Notable Changes*
>
>
>
> *Adding Components and Modifying Component Build Rules*
>
> One of the goals of this system is to simplify adding and modifying
> components.  For this reason, the components each have their own file with
> build rules:
>
>   NEMS/src/incmake/component_FOO.mk
>
> for a given component FOO.  To add a new component, you just have to make
> a new component_FOO.mk file for it.  The file will be automatically used if
> the component is in the list of components to build (Makefile $(COMPONENTS)
> variable), and ignored otherwise.  Hence, two components can have
> conflicting logic as long as the components are not built together in the
> one executable.
>
> There is one more file for specifying dependencies between components,
> such as building FMS before FV3 and MOM6:
>
>   NEMS/src/incmake/dependencies.mk
>
>
> *Application-Level Overrides*
>
> Another big goal is to let apps easily override logic in the NEMS build
> system without having to maintain a separate NEMS branch.  For example,
> adding components, modifying component build rules, or changing platform
> configuration logic.
>
> This is accomplished by:
>
> 1. If a component_FOO.mk is in the application-level conf/ directory, it
> overrides any NEMS/src/incmake/component_FOO.mk.  New components or updated
> build logic can be added in that manner.
>
> 2. Three makefile fragments are loaded from the application level (if
> present) to override logic and add application-specific build rules:
>
>   - conf/before_components.mk - overrides platform decisions and provides
> defaults for component configuration
>   - conf/after_components.mk - overrides component configuration decisions
>   - conf/after_everything.mk - overrides anything else; sourced last,
> just before build rules are executed
>
> For example, the NEMSfv3gfs adds the tests/*.exe files as dependencies of
> the "build" rule.  That way, a "make build" will generate the tests/*.exe
> file without any extra "cp" commands previously present in compile.sh.
>
> 3. Makefile variables that can override platform and component logic.
> Documenting this properly is too much for an email.
>
> 4. The *.appBuilder files are still supported (make app=standaloneFV3)
>
>
> *Automated Tests*
>
> The NEMS/src/incmake/tests.mk is able to run automated tests (as in
> cmake, scons, or autotools) to determine basic aspects of the platform and
> environment.  Presently, it only detects case sensitivity of the
> filesystem, and the ESMF version.  A file NEMS/src/conf/test-results.mk
> is created with results of the tests.
>
>
> *Build Without a Wrapper*
>
> It is easy to build directly, without assistance of NEMSAppBuilder or
> compile.sh:
>
> cd NEMS
> make -j 8 COMPONENTS=FV3,MOM6,CICE build
>
> The -j 8 is optional, and will be passed down automatically to the
> underlying components.  However, only one component will be built at a time
> even if -j is specified.  Parallel build of components can work, but is
> explicitly disabled via ".NOTPARALLEL:" in NEMS/GNUmakefile because the
> build log is totally unreadable when building multiple components at once.
>
> Note that component dependencies are handled automatically.  If you
> specify MOM6 or FV3, then "FMS" will be added (if absent) and will be at
> the beginning of the list.  If you ask for CCPP and FV3, CCPP will be built
> first regardless of the order you specify.  The logic in NEMS/src/incmake/
> dependencies.mk adds components or changes the order in which they're
> built.  The individual component_*.mk contains logic to detect an
> impossible component list (such as CCPP without a supported atmospheric
> component).
>
>
> *Cleaning Components and NEMS*
>
> Unlike the NEMSAppBuilder, the NEMS/GNUmakefile will not clean before
> building.  You have to explicitly tell it to clean, just like in any
> makefile.  There are two levels of cleaning:
>
>   make ... options ... -k clean      # delete intermediate files
>   make ... options ... -k distclean     # also delete targets (libraries,
> modules, executables, etc.)
>
> The "-k" option is important when cleaning.  It ensures Make will run all
> of the "clean" or "distclean" rules, even if one fails.  Some of the
> components' underlying build systems will fail to clean themselves, and
> their "clean" rule will have non-zero status.  Without the "-k", Make will
> stop cleaning as soon as one component's "clean" or "distclean" rule
> aborts.  For the components in the Supported apps, I've put in workarounds
> for bugs I've found in the components' clean rules.  However, there is no
> 100% guarantee that each component will clean itself properly.
>
> If you want to be much more thorough, there is an aptly-named build
> target, "armageddon."  It will attempt to revert to the repository state,
> delete all untracked files and directories (even "ignored" ones), discard
> local changes, and force a new checkout of all submodules.  This is done
> recursively from the application level on down.  Some types of submodule
> conflicts can prevent armageddon because the "git submodule" implementation
> may refuse to switch branches if conflicting files are present.  Use this
> feature with great care, lest you discard what you meant to keep.
>
>
> *Fine-Grained Build*
>
> The build is split into multiple steps via Make rules.  These steps can be
> run directly to execute just one part of the build or configuration process.
>
> Components can be cleaned, distcleaned, and built individually:
>
>   make COMPONENTS=FV3,MOM6,CICE5 distclean_MOM6
>   make COMPONENTS=FV3,MOM6,CICE5 build_MOM6
>
> The NEMS source and executable can be cleaned independently of the
> components:
>
>   make COMPONENTS=FV3,MOM6,CICE5 clean_NEMS  # delete intermediate files
>   make COMPONENTS=FV3,MOM6,CICE5 distclean_NEMS  # also delete targets
>
> You can configure the NEMS without building anything:
>
>   make COMPONENTS=FV3,MOM6,CICE5 configure
>
> You can test the selected modulefile (or lack thereof) without building
> anything:
>
>   make COMPONENTS=FV3,MOM6,CICE5 module_test
>
> Or you could get information about what would be built, without actually
> building or configuring anything:
>
>   make COMPONENTS=FV3,MOM6,CICE5   # No build target.
>
> This runs the internal makefile logic, without executing any Make rules.
> The automated tests are not done via make rules, so they'll still be run.
> This is the default action if no target is specified, and so a help message
> is included in the output.
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nems.announce/attachments/20181018/5389d6b3/attachment-0001.html 