[Ncep.list.nems.announce] Feedback requested on replacing the NEMS build system.
Anthony.P.Craig
anthony.p.craig at gmail.com
Tue Feb 7 17:43:33 UTC 2017
I think there are many ways this could be done and I'm not going to
advocate a particular solution,
but CESM has had to deal with the same issues. The approach taken by
CESM is similar to what Rocky
suggested. There is a central build script that calls the "build_xxx"
of each component and that
build_xxx lives within the component. There are standard names and
locations for those so the central
script can find them. In addition, in CESM all components use the same
Makefile to build. So the central build,
depending on the app, calls the appropriate component build_xxx routines
and those execute a gmake call that
all share a common Makefile with several variables defined that allow
each component to customize the
build as needed. A common Makefile is not actually required but I
believe you also cannot allow each model
to build with arbitrary options, the modules and some compile settings
need to be common.
CESM has taken the same approach to setting up the "namelist" and other
input data. A central script
calls into the "buildnml_xxx" for each component and that generates the
namelist (or equivalent text)
input files as well as creates a list of "binary" input files needed to
run the model. Again, there are several
common variables and conventions but the script that does most of this
work lives in each component.
It is important to separate the "central" scripts from the "component"
scripts cleanly and that is not always
easy to do, especially as they need to be flexible and extensible. But
as much as possible, that separation
should be clear and the requirements on either side of that separation
should be clearly defined. I also
agree that keeping it simple is better for most users and to keep
maintenance lower. But, there is also
likely to be a point where the scripts work well enough and meet user
requirements well enough that
most users won't need to know the details of the implementation.
Ultimately, I think that's where you want
to end up. A clearly implemented set of scripts that the users don't
ultimately care much about.
To provide just a few more details. As I said, the CESM build uses a
common Makefile. It also has some
other common tools that the Makefile uses like a tool to build the
dependencies on the fly, that all components
can use. Also, the component scripts that build the namelist for each
component also generate a list of
input files needed. We then have a tool that checks that all those file
exist and are ready to use. If not,
the job will report a problem before running. That's been very handy.
thanks,
tony.......
On 2/7/17 8:56 AM, Rocky Dunlap - NOAA Affiliate wrote:
> Since NEMS is a "unified" system, a big question here is what goes
> under NEMS and what goes under each component and what goes under each
> app. Per Gerhard's comment, a very nice feature of the NEMSAppBuilder
> is that if I want to pull in an existing component into my app, I do
> not really have to know how to build it--that was figured out once by
> a developer and that knowledge is then put into the NEMSAppBuilder
> under the build_xxx() routines.
>
> Now, per Sam's point, having to update the NEMS trunk when the build
> of a single component changes is not ideal. Instead, what about a
> hybrid approach where:
> - the build system framework is under NEMS (something like
> NEMSAppBuilder)
> - component-specific builds are under each component (something like
> build_xxx())
>
> The NEMSAppBuilder could then pull in the individual pieces from each
> component--they would need to be in standard locations with standard
> parameters, etc. NEMS would only need to be updated if the build
> system framework itself changed, which should not be too often.
>
> Ideally, the build_xxx() piece could ALSO be used standalone, e.g., to
> build a library or executable from a single component.
> Keep in mind that as more things move out to the app or component
> level, the less "unified" NEMS becomes. If everything goes back to
> separate builds of separate components and apps, then eventually you
> will have a bunch of separate systems. My understanding is the point
> of NEMS is to move away from that.
>
> Rocky
>
>
>
> On Tue, Feb 7, 2017 at 9:27 AM, Gerhard Theurich
> <theurich at sourcespring.net <mailto:theurich at sourcespring.net>> wrote:
>
> Hi,
>
> Just for the immediate, I am wondering what the problem is with
> NEMSAppBuilder on WCOSS? Has anybody looked into it? I thought Patrick
> was building the coupled system on WCOSS, so it does surprise me that
> Moorthi cannot get GSM standalone working.
>
> Most important for the long term that I see is a _maintained_ system.
> Any system that is not maintained will break, especially if it has to
> cover a wide range of complex cases that keep changing (e.g. new
> components coming in). A maintained system will be able to change
> according to changing needs.
>
> One fundamental concept of the NEMSAppBuilder was to provide a single
> place where the build recipes of all the various components are kept.
> That is why it is part of the NEMS infrastructure: any app that uses a
> specific model component (even if it is a different revision of that
> component) should be able to just tell the AppBuilder that it wants to
> build that certain component, not have to know how to do this. (In the
> current AppBuilder this is implemented via the build_xxx() routines in
> the script. E.g. build_cice() will tell the system how to build CICE.
> These routines are typically just a few lines long and use a few
> variables to get the job done.)
>
> Another basic concept was that the configuration for an app should be
> provided by a very simple file, telling the app builder what
> components
> are to be built and linked together. Those are the *.appBuilder files.
> Obviously the implementation details, including how maybe a GUI is
> implemented could change, but I do think the fundamental concepts and
> how they are divided between app layer and NEMS layer are still valid.
>
> -Gerhard
>
> On 02/07/2017 06:04 AM, Shrinivas Moorthi wrote:
> > I can't compile standalone GSM on wcoss. How do I do it without
> > blackbox (appbuilder)?
> > Thanks
> > Moorthi
> >
> > On 02/07/2017 08:46 AM, Samuel Trahan - NOAA Affiliate wrote:
> >> Hi all,
> >>
> >> The NEMS build system, NEMSAppBuilder, is causing a lot of
> problems,
> >> especially for the atmospheric model developers. We plan on
> replacing
> >> the build system. What do you want out of the new build
> system? What
> >> do you dislike about the old one?
> >>
> >> Some questions to ponder:
> >>
> >> - How should the build system work internally? Shell script? Make?
> >> Cmake?
> >>
> >> - How should you run the build system? Shell script? Run "make?"
> >> Run a GUI?
> >>
> >> - When do we replace the build system? Do we do it now, and risk
> >> breaking coupled systems? Do we wait until we can test it with the
> >> coupled applications?
> >>
> >> Feedback thus far:
> >>
> >> 1. It is difficult to navigate the 1900 line NEMSAppBuilder to
> figure
> >> out how to change the build commands.
> >>
> >> 2. Some users want to be able to manually build the NEMS without
> >> running an overarching script.
> >>
> >> 3. Some users want a simple graphical interface to select
> components
> >> and build the NEMS. (Yes, there are users that want this.)
> >>
> >> 4. Sometimes, components compile with options that are incompatible
> >> with the linking options. This is because each component has
> its own
> >> configuration system. This causes problems, as we saw with FV3.
> >>
> >> 5. The logic for building a component is in the NEMS framework
> level.
> >> Any time a component's build system changes, the NEMS framework
> has to
> >> be updated, breaking applications that use older versions of the
> >> component. This forces applications to use non-trunk versions
> of the NEMS.
> >>
> >> 6. The NEMS/src/configure script contains application-specific
> >> logic. This also forces applications to use non-trunk versions
> of the
> >> NEMS.
> >>
> >> Sincerely,
> >> Sam Trahan
> >>
> >>
> >> _______________________________________________
> >> Ncep.list.nems.announce mailing list
> >> Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
> <mailto:Ncep.list.nems.announce at lstsrv.ncep.noaa.gov>
> >>
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce
> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce>
> >
> >
> > --
> > Dr. Shrinivas Moorthi
> > Research Meteorologist
> > Global Climate and Weather Modeling Branch
> > Environmental Modeling Center / National Centers for
> Environmental Prediction
> > 5830 University Research Court - (W/NP23), College Park MD 20740 USA
> > Tel:(301)683-3718 <tel:%28301%29683-3718>
> >
> >
> >
> > _______________________________________________
> > Ncep.list.nems.announce mailing list
> > Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
> <mailto:Ncep.list.nems.announce at lstsrv.ncep.noaa.gov>
> >
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce
> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce>
> >
> _______________________________________________
> Ncep.list.nems.announce mailing list
> Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
> <mailto:Ncep.list.nems.announce at lstsrv.ncep.noaa.gov>
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce
> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce>
>
>
>
>
> _______________________________________________
> Ncep.list.nems.announce mailing list
> Ncep.list.nems.announce at lstsrv.ncep.noaa.gov
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.nems.announce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nems.announce/attachments/20170207/2f8a4c25/attachment.html
More information about the Ncep.list.nems.announce
mailing list