[Ncep.list.nems.announce] Feedback requested on replacing the NEMS build system.

Samuel Trahan - NOAA Affiliate samuel.trahan at noaa.gov
Tue Feb 7 17:12:16 UTC 2017


Rocky,

I know some people are able to understand and modify the NEMSAppBuilder.
Unfortunately, most of the NEMS community cannot.  The standalone
atmosphere developers refuse to use or maintain it.  That is causing
problems for the coupled model development.  Some of the concerns are
critical ones, so I agree that we need to act on them.

If you want to discuss the internals of NEMSAppBuilder though...

The process of adding a component is not as simple as you describe, partly
because the bulk of the logic is not in the build_xxx() function.  There
are environment variables set outside that function in NEMSAppBuilder.  The
appBuilder files themselves have additional settings.  There are more in
the configure.nems files, and sometimes conflicting information in the
per-component configuration.

Secondly, the NEMSAppBuilder internally re-implements two aspects of the
"make" command, which "make" does better and more reliably: dependency
tracking and process tracking.  Around a third of the NEMSAppBuilder
re-implements dependency tracking, which we can express in "make" simple
"target: dependency list" lines.  Another third tracks build processes.  If
the user stops the build, the NEMSAppBuilder will kill ALL gmake processes,
not just the ones related to the NEMSAppBuilder.  Make has a better
implementation of this, and correctly handles control-C.  If you control-C
NEMSAppBuilder, it ignores you and keeps building.

The implementation of the GUI is another problem.  It uses one of two
tools, whiptail and dialog, that are not always available, and frequently
have bugs.  (Look through the NEMSAppBuilder and you'll see workarounds.)
It would be better to use a more ubiquitous tool like "curses."

If the system is reduced to a makefile, with a GUI wrapped around it, it
will be dramatically simpler to understand and modify but still usable as a
black box GUI system.  The same simplicity of build_xxx() functions will be
in "make" blocks.  The environment variables and NEMS conf files can be
rolled into a single file.  We can still call it an "appBuilder" file, and
it can still have a GUI wrapped around it.  The result would be something
that users will more easily understand, but can still be used as a black
box to anyone who doesn't want to think about internals.

Sincerely,
Sam Trahan

On Tue, Feb 7, 2017 at 11:56 AM, Rocky Dunlap - NOAA Affiliate <
rocky.dunlap at noaa.gov> wrote:

> Since NEMS is a "unified" system, a big question here is what goes under
> NEMS and what goes under each component and what goes under each app. Per
> Gerhard's comment, a very nice feature of the NEMSAppBuilder is that if I
> want to pull in an existing component into my app, I do not really have to
> know how to build it--that was figured out once by a developer and that
> knowledge is then put into the NEMSAppBuilder under the build_xxx()
> routines.
>
> Now, per Sam's point, having to update the NEMS trunk when the build of a
> single component changes is not ideal. Instead, what about a hybrid
> approach where:
> - the build system framework is under NEMS  (something like NEMSAppBuilder)
> - component-specific builds are under each component  (something like
> build_xxx())
>
> The NEMSAppBuilder could then pull in the individual pieces from each
> component--they would need to be in standard locations with standard
> parameters, etc.  NEMS would only need to be updated if the build system
> framework itself changed, which should not be too often.
>
> Ideally, the build_xxx() piece could ALSO be used standalone, e.g., to
> build a library or executable from a single component.
>
> Keep in mind that as more things move out to the app or component level,
> the less "unified" NEMS becomes. If everything goes back to separate builds
> of separate components and apps, then eventually you will have a bunch of
> separate systems. My understanding is the point of NEMS is to move away
> from that.
>
> Rocky
>
>
>
> On Tue, Feb 7, 2017 at 9:27 AM, Gerhard Theurich <
> theurich at sourcespring.net> wrote:
>
>> Hi,
>>
>> Just for the immediate, I am wondering what the problem is with
>> NEMSAppBuilder on WCOSS? Has anybody looked into it? I thought Patrick
>> was building the coupled system on WCOSS, so it does surprise me that
>> Moorthi cannot get GSM standalone working.
>>
>> Most important for the long term that I see is a _maintained_ system.
>> Any system that is not maintained will break, especially if it has to
>> cover a wide range of complex cases that keep changing (e.g. new
>> components coming in). A maintained system will be able to change
>> according to changing needs.
>>
>> One fundamental concept of the NEMSAppBuilder was to provide a single
>> place where the build recipes of all the various components are kept.
>> That is why it is part of the NEMS infrastructure: any app that uses a
>> specific model component (even if it is a different revision of that
>> component) should be able to just tell the AppBuilder that it wants to
>> build that certain component, not have to know how to do this. (In the
>> current AppBuilder this is implemented via the build_xxx() routines in
>> the script. E.g. build_cice() will tell the system how to build CICE.
>> These routines are typically just a few lines long and use a few
>> variables to get the job done.)
>>
>> Another basic concept was that the configuration for an app should be
>> provided by a very simple file, telling the app builder what components
>> are to be built and linked together. Those are the *.appBuilder files.
>> Obviously the implementation details, including how maybe a GUI is
>> implemented could change, but I do think the fundamental concepts and
>> how they are divided between app layer and NEMS layer are still valid.
>>
>> -Gerhard
>>
>> On 02/07/2017 06:04 AM, Shrinivas Moorthi wrote:
>> > I can't compile standalone GSM on wcoss.  How do I do it without
>> > blackbox (appbuilder)?
>> > Thanks
>> > Moorthi
>> >
>> > On 02/07/2017 08:46 AM, Samuel Trahan - NOAA Affiliate wrote:
>> >> Hi all,
>> >>
>> >> The NEMS build system, NEMSAppBuilder, is causing a lot of problems,
>> >> especially for the atmospheric model developers.  We plan on replacing
>> >> the build system.  What do you want out of the new build system?  What
>> >> do you dislike about the old one?
>> >>
>> >> Some questions to ponder:
>> >>
>> >> - How should the build system work internally?  Shell script?  Make?
>> >> Cmake?
>> >>
>> >> - How should you run the build system?  Shell script?  Run "make?"
>> >> Run a GUI?
>> >>
>> >> - When do we replace the build system?  Do we do it now, and risk
>> >> breaking coupled systems? Do we wait until we can test it with the
>> >> coupled applications?
>> >>
>> >> Feedback thus far:
>> >>
>> >> 1. It is difficult to navigate the 1900 line NEMSAppBuilder to figure
>> >> out how to change the build commands.
>> >>
>> >> 2. Some users want to be able to manually build the NEMS without
>> >> running an overarching script.
>> >>
>> >> 3. Some users want a simple graphical interface to select components
>> >> and build the NEMS.  (Yes, there are users that want this.)
>> >>
>> >> 4. Sometimes, components compile with options that are incompatible
>> >> with the linking options.  This is because each component has its own
>> >> configuration system.  This causes problems, as we saw with FV3.
>> >>
>> >> 5. The logic for building a component is in the NEMS framework level.
>> >> Any time a component's build system changes, the NEMS framework has to
>> >> be updated, breaking applications that use older versions of the
>> >> component. This forces applications to use non-trunk versions of the
>> NEMS.
>> >>
>> >> 6.  The NEMS/src/configure script contains application-specific
>> >> logic.  This also forces applications to use non-trunk versions of the
>> >> NEMS.
>> >>
>> >> Sincerely,
>> >> Sam Trahan
>> >>
>> >>
>> >> _______________________________________________
>> >> Ncep.list.nems.announce mailing list
>> >> Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
>> >> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>> nems.announce
>> >
>> >
>> > --
>> > Dr. Shrinivas Moorthi
>> > Research Meteorologist
>> > Global Climate and Weather Modeling Branch
>> > Environmental Modeling Center / National Centers for Environmental
>> Prediction
>> > 5830 University Research Court - (W/NP23), College Park MD 20740 USA
>> > Tel:(301)683-3718
>> >
>> >
>> >
>> > _______________________________________________
>> > Ncep.list.nems.announce mailing list
>> > Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
>> > https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>> nems.announce
>> >
>> _______________________________________________
>> Ncep.list.nems.announce mailing list
>> Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce
>>
>
>
> _______________________________________________
> Ncep.list.nems.announce mailing list
> Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nems.announce/attachments/20170207/5c40740b/attachment-0001.html 


More information about the Ncep.list.nems.announce mailing list