[Ncep.list.nems.announce] Feedback requested on replacing the NEMS build system.

Gerhard Theurich theurich at sourcespring.net
Tue Feb 7 17:18:54 UTC 2017


Rocky, your last paragraph I think summarizes the fundamental issue 
nicely. Quoting: "If everything goes back to separate builds of separate 
components and apps, then eventually you will have a bunch of separate 
systems."

Now it is true that any unified system requires dependent systems to 
stay in sync to some degree. And in a sense that is the whole idea of 
wanting a unified system in the first place. We've built plenty of 
flexibility into the system with the app concept, allowing apps to pull 
in their own versions of constituent components. Sam has further cleaned 
this up by moving app stuff out of the common layer, and together with 
the ATM-refactor I think we are pretty much at the point where the NEMS 
part that gets pulled into each app is pretty much what it should be. 
Just thinking practically, we should soon be at a point there moving to 
a later rev on the NEMS trunk for your app should not be a big deal 
anymore - at least not from a build perspective. The bigger piece as far 
as NEMS is concerned are the Mediators it provides. But again, that is 
sort of the whole Idea of the  unified system. That is where branching 
and merging will be used, so if an app during its development modifies a 
Mediator, it will need to be on a branch of NEMS.

Again, none of the above says anything about how the AppBuilder should 
be implemented. Personally I am more concerned in this discussion about 
separating the pieces correctly in light of creating a unified system. 
So if the AppBuilder needs to be reimplemented, it should be done. But I 
don't think we want to loose NEMS as the common backbone of the unified 
system.

-Gerhard

On 02/07/2017 08:56 AM, Rocky Dunlap - NOAA Affiliate wrote:
> Since NEMS is a "unified" system, a big question here is what goes under
> NEMS and what goes under each component and what goes under each app.
> Per Gerhard's comment, a very nice feature of the NEMSAppBuilder is that
> if I want to pull in an existing component into my app, I do not really
> have to know how to build it--that was figured out once by a developer
> and that knowledge is then put into the NEMSAppBuilder under the
> build_xxx() routines.
>
> Now, per Sam's point, having to update the NEMS trunk when the build of
> a single component changes is not ideal. Instead, what about a hybrid
> approach where:
> - the build system framework is under NEMS  (something like NEMSAppBuilder)
> - component-specific builds are under each component  (something like
> build_xxx())
>
> The NEMSAppBuilder could then pull in the individual pieces from each
> component--they would need to be in standard locations with standard
> parameters, etc.  NEMS would only need to be updated if the build system
> framework itself changed, which should not be too often.
>
> Ideally, the build_xxx() piece could ALSO be used standalone, e.g., to
> build a library or executable from a single component.
> Keep in mind that as more things move out to the app or component level,
> the less "unified" NEMS becomes. If everything goes back to separate
> builds of separate components and apps, then eventually you will have a
> bunch of separate systems. My understanding is the point of NEMS is to
> move away from that.
>
> Rocky
>
>
>
> On Tue, Feb 7, 2017 at 9:27 AM, Gerhard Theurich
> <theurich at sourcespring.net <mailto:theurich at sourcespring.net>> wrote:
>
>     Hi,
>
>     Just for the immediate, I am wondering what the problem is with
>     NEMSAppBuilder on WCOSS? Has anybody looked into it? I thought Patrick
>     was building the coupled system on WCOSS, so it does surprise me that
>     Moorthi cannot get GSM standalone working.
>
>     Most important for the long term that I see is a _maintained_ system.
>     Any system that is not maintained will break, especially if it has to
>     cover a wide range of complex cases that keep changing (e.g. new
>     components coming in). A maintained system will be able to change
>     according to changing needs.
>
>     One fundamental concept of the NEMSAppBuilder was to provide a single
>     place where the build recipes of all the various components are kept.
>     That is why it is part of the NEMS infrastructure: any app that uses a
>     specific model component (even if it is a different revision of that
>     component) should be able to just tell the AppBuilder that it wants to
>     build that certain component, not have to know how to do this. (In the
>     current AppBuilder this is implemented via the build_xxx() routines in
>     the script. E.g. build_cice() will tell the system how to build CICE.
>     These routines are typically just a few lines long and use a few
>     variables to get the job done.)
>
>     Another basic concept was that the configuration for an app should be
>     provided by a very simple file, telling the app builder what components
>     are to be built and linked together. Those are the *.appBuilder files.
>     Obviously the implementation details, including how maybe a GUI is
>     implemented could change, but I do think the fundamental concepts and
>     how they are divided between app layer and NEMS layer are still valid.
>
>     -Gerhard
>
>     On 02/07/2017 06:04 AM, Shrinivas Moorthi wrote:
>      > I can't compile standalone GSM on wcoss.  How do I do it without
>      > blackbox (appbuilder)?
>      > Thanks
>      > Moorthi
>      >
>      > On 02/07/2017 08:46 AM, Samuel Trahan - NOAA Affiliate wrote:
>      >> Hi all,
>      >>
>      >> The NEMS build system, NEMSAppBuilder, is causing a lot of problems,
>      >> especially for the atmospheric model developers.  We plan on
>     replacing
>      >> the build system.  What do you want out of the new build
>     system?  What
>      >> do you dislike about the old one?
>      >>
>      >> Some questions to ponder:
>      >>
>      >> - How should the build system work internally?  Shell script?  Make?
>      >> Cmake?
>      >>
>      >> - How should you run the build system?  Shell script?  Run "make?"
>      >> Run a GUI?
>      >>
>      >> - When do we replace the build system?  Do we do it now, and risk
>      >> breaking coupled systems? Do we wait until we can test it with the
>      >> coupled applications?
>      >>
>      >> Feedback thus far:
>      >>
>      >> 1. It is difficult to navigate the 1900 line NEMSAppBuilder to
>     figure
>      >> out how to change the build commands.
>      >>
>      >> 2. Some users want to be able to manually build the NEMS without
>      >> running an overarching script.
>      >>
>      >> 3. Some users want a simple graphical interface to select components
>      >> and build the NEMS.  (Yes, there are users that want this.)
>      >>
>      >> 4. Sometimes, components compile with options that are incompatible
>      >> with the linking options.  This is because each component has
>     its own
>      >> configuration system.  This causes problems, as we saw with FV3.
>      >>
>      >> 5. The logic for building a component is in the NEMS framework
>     level.
>      >> Any time a component's build system changes, the NEMS framework
>     has to
>      >> be updated, breaking applications that use older versions of the
>      >> component. This forces applications to use non-trunk versions of
>     the NEMS.
>      >>
>      >> 6.  The NEMS/src/configure script contains application-specific
>      >> logic.  This also forces applications to use non-trunk versions
>     of the
>      >> NEMS.
>      >>
>      >> Sincerely,
>      >> Sam Trahan
>      >>
>      >>
>      >> _______________________________________________
>      >> Ncep.list.nems.announce mailing list
>      >> Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
>     <mailto:Ncep.list.nems.announce at lstsrv.ncep.noaa.gov>
>      >>
>     https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce
>     <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce>
>      >
>      >
>      > --
>      > Dr. Shrinivas Moorthi
>      > Research Meteorologist
>      > Global Climate and Weather Modeling Branch
>      > Environmental Modeling Center / National Centers for
>     Environmental Prediction
>      > 5830 University Research Court - (W/NP23), College Park MD 20740 USA
>      > Tel:(301)683-3718 <tel:%28301%29683-3718>
>      >
>      >
>      >
>      > _______________________________________________
>      > Ncep.list.nems.announce mailing list
>      > Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
>     <mailto:Ncep.list.nems.announce at lstsrv.ncep.noaa.gov>
>      >
>     https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce
>     <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce>
>      >
>     _______________________________________________
>     Ncep.list.nems.announce mailing list
>     Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
>     <mailto:Ncep.list.nems.announce at lstsrv.ncep.noaa.gov>
>     https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce
>     <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce>
>
>


More information about the Ncep.list.nems.announce mailing list