[Ncep.list.nems.announce] Upcoming commits: new NEMS build system and ESMF 8 support

Thu Oct 18 19:28:00 UTC 2018

Hi again,

The big, multi-machine, test of the build system is starting now.  I cannot
promise a Friday commit yet; I don't know how slow the queues will be.

Sincerely,
Sam Trahan

On Thu, 18 Oct 2018 at 12:08, Samuel Trahan - NOAA Affiliate <
samuel.trahan at noaa.gov> wrote:

> Hi all,
>
> I have gotten some excellent feedback on the new NEMS build system and
> made the necessary changes.  I'm waiting for one FV3-MOM6-CICE5 commit that
> will go in soon.  Then I'll start run the big, multi-machine, multi-app
> test, one last time before committing.  I'm hoping to commit Friday
> afternoon, but it depends in part on queue time and on the timing of the
> FV3-MOM6-CICE5 commit.
>
> You can expect another build system commit in a week or so, to add some
> portability changes.
>
> Sincerely,
> Sam Trahan
>
> On Tue, 9 Oct 2018 at 13:00, Samuel Trahan - NOAA Affiliate <
> samuel.trahan at noaa.gov> wrote:
>
>> NEMS developers,
>>
>> There are two commits coming into the NEMS git master soon.  We're
>> expecting to commit them early next week, hopefully Monday.
>>
>> 1. Support for new ESMF 8 beta snapshots which break backward
>> compatibility to ESMF 7.x and earlier.  This requires adding an #ifdef in
>> NEMS code that triggers on the ESMF version.  The NEMS build system cannot
>> support this, so we need a new NEMS build system.
>>
>> 2. A new NEMS build system.  It re-expresses the NEMSAppBuilder in GNU
>> Make.
>>
>> We're working on merging change #1 into change #2, and we plan on
>> committing them at the same time.
>>
>> The remainder of this email details the new NEMS build system.
>>
>> Sincerely,
>> Sam Trahan
>>
>>
>> *--------------------------------------------------------------------------------------------------------------------------------------------------------------*
>>
>> *The New NEMS Build System*
>>
>> *Overview*
>>
>> This is not the next-generation build system that will save the world; it
>> is just a cleaned-up version of the old one, re-expressed in GNU Make.  The
>> NEMSAppBuilder GUI is gone, but other than this, the front-end is
>> backward-compatible.  You will not notice any changes to the build system
>> if you're just building or running the NEMS or its Supported apps via the
>> supported methods.  Specifically, the NEMSCompsetRun, NEMSAppBuilder,
>> NEMSfv3gfs/tests, and NEMSGSM/oldtests will all work just like they did
>> before.  This change is mostly to benefit those of us that do need to
>> access or modify the internals of the NEMS build system.  It is intended to
>> be easier to work with and easier to port than the old one.
>>
>> You will only be impacted by this commit if you have your own build
>> system, if you use a component that is not in a Supported app, or if you
>> intend to modify or port the build system.  For example, if you want to add
>> components or change the component build rules, you will need to know the
>> internals of the new system.  Also, the NEMS/src/configure and
>> NEMS/src/makefile are gone.  The new build system moves the logic of
>> NEMSAppBuilder into a top-level GNUmakefile and src/incmake/*.mk files for
>> details of the build.  A description of the step-by-step logic, and purpose
>> of the lower-level makefiles, can be found in the top-level GNUmakefile.
>>
>> The build system is only fully tested for Supported apps and their target
>> platforms, listed below.  That means some actively-developed components and
>> platforms are not tested.  All components are present in the build system,
>> and were converted from shell to GNU make.  The platform logic should be
>> able to handle any arbitrary platform, but only certain platforms are
>> tested.  We will push to the git master before the non-Supported features
>> are tested, so you can expect a few more commits over the coming weeks to
>> fix any non-Supported features.
>>
>> I would like to re-iterate that the NEMSAppBuilder GUI mode is gone
>> forever, by request of EMC management.  There are superior GUI build
>> systems and Integrated Development Environments (IDE) available.  I prefer
>> Emacs, but saner people seem to like Eclipse, which can be run on most
>> (all?) NOAA clusters.  Now that the build system is written in Make, your
>> favorite IDE will be better able to understand it.
>>
>> Read on for details...
>>
>>
>> *Where to get the New NEMS Build System*
>>
>> You can find the new build system in the "rewrite-appbuilder" branches of
>> these apps:
>>
>> git clone -b rewrite-appbuilder gerrit:NEMSfv3gfs
>> git clone -b rewrite-appbuilder gerrit:EMC_GSM-MOM5-CICE5
>> git clone -b rewrite-appbuilder gerrit:EMC_NEMSGSM
>> git clone -b rewrite-appbuilder gerrit:EMC_FV3-WW3
>> git clone -b rewrite-appbuilder gerrit:EMC_ATM-WW3
>> git clone -b rewrite-appbuilder gerrit:EMC_HYCOM-GSM-CICE
>> git clone -b rewrite-appbuilder gerrit:EMC_FV3-MOM6-CICE5
>>
>> The NEMS branch is rewrite-appbuilder for all apps except
>> GSM-MOM5-CICE5.  That app has fallen behind and cannot use the latest
>> NEMS.  There is a rewrite-appbuilder-GSM-MOM5-CICE5 branch for that app's
>> NEMS submodule.
>>
>>
>> *What is Tested, Untested, or Removed?*
>>
>> This is what has been tested and confirmed to work:
>>
>>   - theia.intel, t/u jet, WCOSS Phase 1 & 2, WCOSS Cray, GAEA C3
>>   - These apps: NEMSfv3gfs, GSM-MOM5-CICE5, NEMSGSM, FV3-WW3, ATM-WW3,
>> HYCOM-GSM-CICE, and FV3-MOM6-CICE5
>>   - These components: FV3, FMS, GSM, MOM5, MOM6, CICE5, WW3, HYCOM
>>   - NEMSAppBuilder
>>   - NEMSCompsetRun
>>   - tests/compile.sh
>>   - tests/rt.sh
>>   - direct build via "cd NEMS ; make ... options ..."
>>   - Building FV3 with CCPP, but not running it.
>>
>> This is being tested now:
>>
>>   - NEMSfv3gfs on WCOSS Dell -- we have no NEMSfv3gfs regression test
>> system here yet, and no other apps target this platform
>>   - NEMSGSM/oldtests/rt.sh -- works, except for GOCART
>>   - GSM GOCART support
>>
>> This is untested:
>>
>>   - Running CCPP to test if its build is correct
>>   - PGI & GNU compilers
>>   - Mac OS/X
>>   - Clusters outside of NOAA (Odin, Cheyenne, etc.)
>>   - These components' build rules have been converted to the new build
>> system but are untested: DATM, GSDCHEM, IMP, IPE, KISS, LIS, NOAH, NOAHMP,
>> WRFHYDRO.
>>   - Apps that are not Supported (other than GSM-MOM5-CICE5)
>>
>> This is intentionally removed:
>>
>>   - GUI NEMSAppBuilder.  This is permanently removed by request of EMC
>> management.  May it rest in peace.
>>
>> This is moved or renamed:
>>
>>   - NEMS/src/configure is gone; configuration is done via "make configure"
>>   - NEMS/src/makefile is now NEMS/src/GNUmakefile
>>   - NEMS/src/ENS_Cpl/makefile is now NEMS/src/ENS_Cpl/GNUmakefile
>>   - NEMS/src/conf/configure.nems.NUOPC is not in the repository.  It is
>> now automatically generated from configure.nems.NUOPC.in, the
>> component_*.mk files, and the component list.
>>
>>
>>
>> *Major Features and Notable Changes*
>>
>>
>>
>> *Adding Components and Modifying Component Build Rules*
>>
>> One of the goals of this system is to simplify adding and modifying
>> components.  For this reason, the components each have their own file with
>> build rules:
>>
>>   NEMS/src/incmake/component_FOO.mk
>>
>> for a given component FOO.  To add a new component, you just have to make
>> a new component_FOO.mk file for it.  The file will be automatically used if
>> the component is in the list of components to build (Makefile $(COMPONENTS)
>> variable), and ignored otherwise.  Hence, two components can have
>> conflicting logic as long as the components are not built together in the
>> one executable.
>>
>> There is one more file for specifying dependencies between components,
>> such as building FMS before FV3 and MOM6:
>>
>>   NEMS/src/incmake/dependencies.mk
>>
>>
>> *Application-Level Overrides*
>>
>> Another big goal is to let apps easily override logic in the NEMS build
>> system without having to maintain a separate NEMS branch.  For example,
>> adding components, modifying component build rules, or changing platform
>> configuration logic.
>>
>> This is accomplished by:
>>
>> 1. If a component_FOO.mk is in the application-level conf/ directory, it
>> overrides any NEMS/src/incmake/component_FOO.mk.  New components or updated
>> build logic can be added in that manner.
>>
>> 2. Three makefile fragments are loaded from the application level (if
>> present) to override logic and add application-specific build rules:
>>
>>   - conf/before_components.mk - overrides platform decisions and
>> provides defaults for component configuration
>>   - conf/after_components.mk - overrides component configuration
>> decisions
>>   - conf/after_everything.mk - overrides anything else; sourced last,
>> just before build rules are executed
>>
>> For example, the NEMSfv3gfs adds the tests/*.exe files as dependencies of
>> the "build" rule.  That way, a "make build" will generate the tests/*.exe
>> file without any extra "cp" commands previously present in compile.sh.
>>
>> 3. Makefile variables that can override platform and component logic.
>> Documenting this properly is too much for an email.
>>
>> 4. The *.appBuilder files are still supported (make app=standaloneFV3)
>>
>>
>> *Automated Tests*
>>
>> The NEMS/src/incmake/tests.mk is able to run automated tests (as in
>> cmake, scons, or autotools) to determine basic aspects of the platform and
>> environment.  Presently, it only detects case sensitivity of the
>> filesystem, and the ESMF version.  A file NEMS/src/conf/test-results.mk
>> is created with results of the tests.
>>
>>
>> *Build Without a Wrapper*
>>
>> It is easy to build directly, without assistance of NEMSAppBuilder or
>> compile.sh:
>>
>> cd NEMS
>> make -j 8 COMPONENTS=FV3,MOM6,CICE build
>>
>> The -j 8 is optional, and will be passed down automatically to the
>> underlying components.  However, only one component will be built at a time
>> even if -j is specified.  Parallel build of components can work, but is
>> explicitly disabled via ".NOTPARALLEL:" in NEMS/GNUmakefile because the
>> build log is totally unreadable when building multiple components at once.
>>
>> Note that component dependencies are handled automatically.  If you
>> specify MOM6 or FV3, then "FMS" will be added (if absent) and will be at
>> the beginning of the list.  If you ask for CCPP and FV3, CCPP will be built
>> first regardless of the order you specify.  The logic in NEMS/src/incmake/
>> dependencies.mk adds components or changes the order in which they're
>> built.  The individual component_*.mk contains logic to detect an
>> impossible component list (such as CCPP without a supported atmospheric
>> component).
>>
>>
>> *Cleaning Components and NEMS*
>>
>> Unlike the NEMSAppBuilder, the NEMS/GNUmakefile will not clean before
>> building.  You have to explicitly tell it to clean, just like in any
>> makefile.  There are two levels of cleaning:
>>
>>   make ... options ... -k clean      # delete intermediate files
>>   make ... options ... -k distclean     # also delete targets (libraries,
>> modules, executables, etc.)
>>
>> The "-k" option is important when cleaning.  It ensures Make will run all
>> of the "clean" or "distclean" rules, even if one fails.  Some of the
>> components' underlying build systems will fail to clean themselves, and
>> their "clean" rule will have non-zero status.  Without the "-k", Make will
>> stop cleaning as soon as one component's "clean" or "distclean" rule
>> aborts.  For the components in the Supported apps, I've put in workarounds
>> for bugs I've found in the components' clean rules.  However, there is no
>> 100% guarantee that each component will clean itself properly.
>>
>> If you want to be much more thorough, there is an aptly-named build
>> target, "armageddon."  It will attempt to revert to the repository state,
>> delete all untracked files and directories (even "ignored" ones), discard
>> local changes, and force a new checkout of all submodules.  This is done
>> recursively from the application level on down.  Some types of submodule
>> conflicts can prevent armageddon because the "git submodule" implementation
>> may refuse to switch branches if conflicting files are present.  Use this
>> feature with great care, lest you discard what you meant to keep.
>>
>>
>> *Fine-Grained Build*
>>
>> The build is split into multiple steps via Make rules.  These steps can
>> be run directly to execute just one part of the build or configuration
>> process.
>>
>> Components can be cleaned, distcleaned, and built individually:
>>
>>   make COMPONENTS=FV3,MOM6,CICE5 distclean_MOM6
>>   make COMPONENTS=FV3,MOM6,CICE5 build_MOM6
>>
>> The NEMS source and executable can be cleaned independently of the
>> components:
>>
>>   make COMPONENTS=FV3,MOM6,CICE5 clean_NEMS  # delete intermediate files
>>   make COMPONENTS=FV3,MOM6,CICE5 distclean_NEMS  # also delete targets
>>
>> You can configure the NEMS without building anything:
>>
>>   make COMPONENTS=FV3,MOM6,CICE5 configure
>>
>> You can test the selected modulefile (or lack thereof) without building
>> anything:
>>
>>   make COMPONENTS=FV3,MOM6,CICE5 module_test
>>
>> Or you could get information about what would be built, without actually
>> building or configuring anything:
>>
>>   make COMPONENTS=FV3,MOM6,CICE5   # No build target.
>>
>> This runs the internal makefile logic, without executing any Make rules.
>> The automated tests are not done via make rules, so they'll still be run.
>> This is the default action if no target is specified, and so a help message
>> is included in the output.
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nems.announce/attachments/20181018/03152500/attachment-0001.html 