[Ncep.list.nems.announce] two upcoming NEMS commits

Fri Sep 29 20:19:44 UTC 2017

Hi all,

The commit is complete.  Revisions of trunks:

NEMS - r98008
apps/NEMSfv3gfs - r98009
apps/NEMSGSM - r98010
apps/WW3-GSM - r98011
apps/GSM-MOM5-CICE5 - r98013
apps/HYCOM-GSM-CICE - r98014

The new GSM-MOM5-CICE5 baseline directory is still transferring to Tide.  I
suggest you wait until 4:40 Eastern before using it.

Sincerely,
Sam Trahan

On Thu, Sep 28, 2017 at 7:19 PM, Samuel Trahan - NOAA Affiliate <
samuel.trahan at noaa.gov> wrote:

> Hi all,
>
> At long last, my testing is complete, and I've fixed all bugs I've found
> in NEMS and apps.  Details are below.  I plan on committing at 4 PM Eastern
> time Friday September 29.
>
> Tests run:
>
> NEMSGSM (Theia, WCOSS Phase 1)
>   - created a baseline, validated against it, validated against original
> baseline
> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSGSM/
> branches/nccmp-deliver
>
> NEMSfv3gfs (Theia; Jet; WCOSS Phase 1, Phase2, Cray)
>   - created a baseline, validated against it, validated against original
> baseline
> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/
> branches/nccmp-deliver
>
> WW3-GSM (Theia, WCOSS Phase 2)
>   - created a baseline, validated against it, validated against original
> baseline
> https://svnemc.ncep.noaa.gov/projects/nems/apps/WW3-GSM/
> branches/nccmp-deliver
>
> HYCOM-GSM-CICE (Theia only)
>   - created a baseline, validated against it, validated against original
> baseline
> https://svnemc.ncep.noaa.gov/projects/nems/apps/HYCOM-GSM-
> CICE/branches/nccmp-deliver
>
> GSM-MOM5-CICE5
>   - added netcdf variable-by-variable comparisons to both enabled tests
>   - created new baseline
>   - validated against now baseline
>
> I did not validate against the old baseline for GSM-MOM5-CICE5 because the
> test now has additional baseline files that are not in the old baseline
> area.
>
> https://svnemc.ncep.noaa.gov/projects/nems/apps/GSM-MOM5-
> CICE5/branches/nccmp-deliver
>
> NEMS branch:
>
> https://svnemc.ncep.noaa.gov/projects/nems/branches/nccmp-deliver
>
>
>
>
>
> In addition to the prior mentioned fixes, this commit also corrects the
> following:
>
>
>
> 1. produtil: When generating a baseline, the NEMSCompsetRun --resume
> feature forgot it was generating a baseline and reset to execution
> (verification) mode.  This is corrected.
>
> 2. NEMS + apps: All /gpfs/hps/emc paths I found are now /gpfs/hps3/emc.
> (produtil had none)
>
> 3. NEMS: Removed an erroneous fingerprint file check from baseline
> generation during a time at which the target directory does not yet exist.
>
> 4. NEMSGSM: The gfs_slg_rsthst compset needs a dependency on gfs_slg
> because it requires files created by that job when run in baseline
> generation mode.
>
> 5. NEMSfv3gfs: On Theia, the NEMSAppBuilder and compile.sh do not generate
> bitwise identical executables but do generate bitwise identical output.  In
> all of the NEMSfv3gfs tests, there is a checksum run on the executable to
> make sure it did not change during the test.  The fv3_appbuilder regression
> test was mistakenly comparing its NEMSAppBuilder-compiled executable's
> checksum against the compile.sh-compiled executable's checksum.  This
> failed on Theia due to the executables not matching bit-for-bit.  The
> regression test is updated to check the NEMSAppBuilder-compiled executable
> against itself.
>
> 6. NEMSfv3gfs: Three of the compsets had their output directories set to
> the same directory as the fv3_control.  (Not just the control directory,
> but the output directory too.)  Occasionally, more than one would delete
> and create the directory at the same time, causing occasional failures.
> They now each have their own output directory.
>
> 7. Most apps: This does not work during baseline creation:
>
>     BASELINE="/path/to/some/dir"
>     BASELINE_TEMPLATE="@[BASELINE]"
>
> In baseline creation mode, the BASELINE variable is replaced with a scrub
> or user-specified directory.  A job, prep_baseline, copies the contents of
> BASELINE_TEMPLATE to BASELINE.  If BASELINE_TEMPLATE="@[BASELINE]" then
> it will copy the directory to itself.  The correct settings are:
>
>     BASELINE="/path/to/some/dir"
>     BASELINE_TEMPLATE="/path/to/some/dir"
>
> This is fixed now.
>
> 8. HYCOM-GSM-CICE: had no directory with logs of past regression tests.  I
> added one with the results of this commit's test.
>
>
>
>
> On Wed, Sep 27, 2017 at 9:57 AM, Samuel Trahan - NOAA Affiliate <
> samuel.trahan at noaa.gov> wrote:
>
>> Hi all,
>>
>> Still testing.  If you're wondering why it is taking so long: most of the
>> apps cannot generate their baseline even when run from their unmodified
>> trunk.  The only one that works reliably is GSM-MOM5-CICE5.  Usually FV3
>> will generate its baseline correctly.  The others simply don't work.  I've
>> fixed about 2/3 of the app*platform combinations but I'm still working on
>> the rest.
>>
>> Sincerely,
>> Sam Trahan
>>
>> On Mon, Sep 25, 2017 at 1:27 PM, Samuel Trahan - NOAA Affiliate <
>> samuel.trahan at noaa.gov> wrote:
>>
>>> Hi all,
>>>
>>> I'm still testing this.  I'm hoping to finish soon.
>>>
>>> Sincerely,
>>> Sam Trahan
>>>
>>> On Wed, Sep 20, 2017 at 11:40 AM, Samuel Trahan - NOAA Affiliate <
>>> samuel.trahan at noaa.gov> wrote:
>>>
>>>> Hi all,
>>>>
>>>> There are two NEMS commits going in soon, both of which modify the
>>>> NEMSCompsetRun.  The commits are backwards-compatible and low risk.
>>>> However, they will need twice as much testing as usual because they affect
>>>> baseline generation.  I'll send out branches soon.  Testing will start
>>>> Friday, and I'll try to finish by the end of the weekend.
>>>>
>>>> The timing of the commit is partly contingent on whether there is
>>>> enough disk space somewhere on Theia to run the tests.  NEMS areas and all
>>>> four stmps hit quota over the past few days, and emc.nemspara cannot write
>>>> anywhere else.  I'll test on Gyre, Surge, and Jet first to detect any
>>>> likely issues before I test on Theia.
>>>>
>>>>
>>>> - The Changes -
>>>>
>>>>
>>>> 1. Bug fix from Jun Wang to .bitcmp. comparison.  If an output file is
>>>> compared to a baseline by specifying the baseline directory (instead of
>>>> filename within the directory) then the baseline creation fails.  It does
>>>> not realize that the target is a directory because the directory does not
>>>> exist before the baseline is created:
>>>>
>>>>     "RESTART/my_output.rst" .bitcmp. "@[BASELINE_DIR]/RESTART/"
>>>>
>>>> The fix is to assume the target is a directory if:
>>>>
>>>>  - it ends with a /
>>>>  - the last path component is .
>>>>  - the last path component is ..
>>>>
>>>> Hence, after the commit, "@[BASELINE_DIR]/RESTART/" will work correctly
>>>> if it ends in a /
>>>>
>>>>
>>>> 2. Add the capability to NEMSCompsetRun to compare NetCDF files
>>>> variable-by-variable instead of bit-for-bit.  This is needed for the CICE5
>>>> files, which contain a global metadata value containing the date at which
>>>> the file was created.  The actual contents of the variables should match.
>>>>
>>>>
>>>> The affected repos are:
>>>>
>>>> - produtil
>>>> - NEMS
>>>> - NEMSfv3gfs
>>>> - NEMSGSM
>>>> - HYCOM-GSM-CICE
>>>> - GSM-MOM5-CICE5
>>>> - WW3-GSM
>>>>
>>>> Sincerely,
>>>> Sam Trahan
>>>>
>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nems.announce/attachments/20170929/c3efa109/attachment-0001.html 