[Ncep.list.nems.announce] Comments Requested: Moving FMS out of MOM6 and FV3 repos

Tue Sep 18 20:21:52 UTC 2018

Hi all,

Addendum.  We're still waiting for the FV3 commit to go through, so this
commit may not be in until tomorrow evening.  That's just as well, because
the WCOSS Cray batch queues are especially full, so it will likely take
until next evening for the tests to complete anyway.

Sincerely,
Sam Trahan

On Tue, Sep 18, 2018 at 3:00 PM, Samuel Trahan - NOAA Affiliate <
samuel.trahan at noaa.gov> wrote:

> Hi all,
>
> We are going ahead with the "FMS Component" commit that moves FMS from the
> MOM6 and FV3 up to the application level.  We intend to commit this
> evening.  An interim code management process is in place, described below.
>
>
> This commit differs from the prior suggested commit in that:
>
> 1. The GFS FMS will come from git://github.com/NOAA-EMC/FMS, which is a
> fork of github.com/NOAA-GFDL/FMS updated to match the FMS currently
> residing in EMC's FV3 repository.  Note that "git://" is used instead of
> "https://" as a workaround for errors in the WCOSS "git" installation.
> This FMS fork will be referred to as the "GFS FMS" for the purposes of this
> email.
>
> 2. A Cheyenne workaround that was mistakenly removed from
> NEMSfv3gfs/tests/compile.sh is put back in.   (Default to three "make"
> threads instead of eight when building on Cheyenne.)
>
>
> We do not yet have a code manager for the GFS FMS fork, nor a concrete
> plan for how to get changes to and from GFDL; we are awaiting guidance from
> the governance working group and EMC management on that matter.  However,
> we do have an interim plan that ensures we have one fork of FMS in EMC,
> with proper version tracking, and the ability to send changes to GFDL and
> back:
>
>
> 1. All Supported NEMS apps will point to the head of the "GFS-FMS" branch
> in github.com/NOAA-EMC/FMS (either git:// or https:// URL is okay).  The
> master of the github.com/NOAA-EMC/FMS repository will match the
> github.com/NOAA-GFDL/FMS repository.
>
> 2. Updates to the GFS-FMS branch require running all compsets (AKA
> regression tests) for all NEMS apps that use FMS, on each platform the app
> supports, verifying reproducibility against a baseline.  This is the same
> commit process as for NEMS itself, except that only apps that use FMS will
> be tested.
>
> 3. The EMC FV3 code manager (presently Jun Wang) will be the ultimate
> arbiter of what does or does not go into the GFS FMS fork.
>
> 4. The NEMS code manager (presently Sam Trahan) will run the mechanics of
> the test and commit process.
>
>
> This interim plan will probably be sufficient to keep things working until
> we have a major update to or from the GFDL FMS master, or until we have
> differences that cannot easily be reconciled.  At that point, we will need
> a code manager for the GFS FMS fork.
>
> Sincerely,
> Sam Trahan
>
>
> On Mon, Aug 27, 2018 at 9:47 PM, Samuel Trahan - NOAA Affiliate <
> samuel.trahan at noaa.gov> wrote:
>
>> Hi all,
>>
>> We have a proposed commit that is urgently needed, but may have major
>> negative impacts to a lot of users.  Please read this email and comment.
>> Our comment period ends at noon Tuesday, September 4.  The commit, if
>> accepted, will happen Wednesday, September 5.
>>
>> *The Commit: Overview*
>>
>> In NEMS apps, the FMS exists within both the FV3 and MOM6 repositories;
>> hence some apps have two copies of FMS.  We have a proposed commit that
>> will move the FMS up one level so there is only one FMS per app.  The FMS
>> we've chosen is the one from the GFDL repository.  The technical details of
>> the changes, are largely inconsequential; the main issue is the choice of
>> repositories.
>>
>> *Consequences*
>>
>> These are the consequences of choosing GFDL's FMS repository.
>>
>> 1. Portability changes to the FMS will be lost permanently because GFDL
>> rejected them.  This means the FV3 GFS will not be usable except with
>> certain versions of the Intel compiler.  This has been confirmed.
>>
>> 2. Recent and upcoming FMS bug fixes will be lost because they are not in
>> the GFDL repository.
>>
>> 3. In general, any FV3 changes will have to wait several weeks if they
>> need changes to FMS.
>>
>> Items #2 and #3 are unavoidable.  GFDL FMS has a large userbase and
>> lengthy test process.  They will, out of necessity, have a long delay
>> before commits.
>>
>> *Alternatives*
>>
>> I know of three alternatives to our FMS repository problem.  Any of these
>> may require funding or tasking changes, in or from NOAA:
>>
>> 1. GFDL handles the FV3 GFS as a special case, maintaining a branch of
>> FMS for the FV3 GFS, and applying changes within 2-3 days.  GFDL ports FMS
>> to other compilers.
>>
>> 2. DTC maintains a fork of FMS which all NEMS apps use.  DTC maintains
>> any portability changes and regularly pulls changes from GFDL FMS.  They
>> receive any changes needed for NEMS coupling and resolve conflicts as
>> needed.
>>
>> 3. EMC maintains the aforementioned fork.
>>
>> Note that DTC is presently tasked with porting the FMS, and has
>> successfully ported it to other platforms and maintained an FMS repository.
>>
>> *The Branches*
>>
>> The changes can be found in the "fms-component" branches of the
>> NEMSfv3gfs and EMC_FV3-MOM6-CICE5 repositories.  These will be updated
>> later this week when incoming FV3 changes are available.  We will be sure
>> to test before committing to the relevant branches.
>>
>> When testing this yourself, note that the EMC_FV3-MOM6-CICE5 warm start
>> test will not reliably reproduce itself; this is a known issue that is
>> unrelated to this commit.  Jet baselines are out-of-date, so you cannot
>> test there.  All other tests should pass for both apps on all supported
>> platforms.
>>
>> *The Commit: Details*
>>
>> Overview of the changes:
>>
>> 1. FMS is removed from the build system in MOM6 and FV3
>>
>> 2. FMS is added at the app level from git://github.com/NOAA-GFDL/FMS.
>> The "git" url makes this submodule read-only.  That is a workaround for
>> problems in the installation of "git" on WCOSS Cray and Dell which make
>> github unreachable via https.  This "git" installation problem has been a
>> known issue for several months and remains unresolved, so such a workaround
>> is necessary.
>>
>> 3. The FMS/fv3gfs directory is updated to be able to build FMS outside of
>> FV3.  This change was accepted by GFDL in their repository.
>>
>> 4. The NEMSAppBuilder is updated to build FMS.  It is treated the same
>> way as a coupled component in the build system.
>>
>> 5. Any appBuilder files must include FMS before FV3 or MOM6 in their
>> component list.
>>
>> 6. The NEMSfv3gfs tests/compile.sh is updated to build FMS before FV3.
>>
>> Sincerely,
>> Sam Trahan
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nems.announce/attachments/20180918/a85b6e5d/attachment-0001.html 