[Ncep.list.fv3-announce] Important: ongoing and upcoming global dump archive changes

Kate Friedman - NOAA Federal Kate.Friedman at noaa.gov
Tue Jun 11 14:29:58 UTC 2019


Dear users of the global dump archive,

Below are updates on some important ongoing and upcoming global dump
archive (GDA) changes. Most changes are for the WCOSS GDAs, although all
users should know of the source change for GFS production dump data (first
item below).

*TL;DR <https://en.wiktionary.org/wiki/too_long;_didn%27t_read#English>:
New GFS production dump file pickup source as of 2019061212 (tomorrow!), no
change on user end required. Also new phase 3 GDA coming soon to WCOSS
users!*

*GFS production dump pickup location change with FV3GFS implementation June
12th 2019 12z*

Tomorrow Wednesday June 12th the new FV3GFS will be implemented into
operations on the Dells. As of 12z that day (2019061212) the GDA will
follow GFS production and begin picking up production dump data from the
Dell GFS prod location and no longer pick up from the Cray GFS prod
location. *This should be a seamless change for users with no need to
change the path of your $DMPDIR variables on WCOSS (yet).*

For anyone running in real-time please note that the 12z data may arrive
late in the GDAs that day if there are any issues. Please let me know if
you are running a real-time FV3GFS experiment on the Dells so I may know
who to contact if there are issues during the switch to the new production
location.

The new production dump data will also populate the Theia GDA from
2019061212 forward. *No change needed for Theia users.*

*Migration of primary WCOSS GDA from phase 1 to phase 3 (also directory
structure change!)*

Additionally the migration of the WCOSS GDA from phase 1/2/Cray to phase
3/Cray is still ongoing. The new phase 3 GDA job is running in test mode
and I am continuing to populate the phase 3 GDA area with data from phase
1. It will take another few weeks to complete the migration of the 25TBs of
data in the primary part of the GDA.

The oldest GDA dump files are still in place on the Cray and phase 2. The
Cray portion of the GDA will remain past the retirement of phase 1/2. As
was done on phase 1 the Cray dump folders will be symlinked to from the
primary phase 3 GDA location. Users will continue to point to the primary
$DMPDIR to get any of the available dump data. More on the future of the
phase 2 portion of the GDA below.

The biggest part of this phase 1 to phase 3 change is the structure of the
GDA. The new phase 3 GDA will mimic the production directory structure
changes introduced with the FV3GFS:

current GDA:

$DMPDIR/YYYYMMDDCC/${CDUMP}${SUFFIX}/$FILE.$CDUMP.YYYYMMDDCC

new GDA:

$DMPDIR/${CDUMP}${SUFFIX}.YYYYMMDD/CC/$CDUMP.tCCz.$FILE

....where SUFFIX is nothing for production data, "nr" for non-restricted
data, "p" for pre-production data, and "x" for experimental data.

Any needed workflow changes to global-workflow will be announced and
committed to the master when the new phase 3 GDA is ready for use.

*Retiring phase 1/2 GDA and parallel naming conventions for dump files*

When the migration of data is complete and the phase 3 GDA job is stable I
will proceed with retiring the phase 1/2 GDA and making the phase 3 GDA the
primary one on WCOSS. Notices with more details will be sent when this is
going to occur. My aim is that this change will also be seamless to users.

There are two big parts of this retirement:

1) The age-off of the oldest dump data currently living on phase 2
(2015010100 - mid-2016). If you have plans to run FV3GFS experiments on
WCOSS for the oldest data please try to complete them before the retirement
of the phase 1/2 parts of WCOSS. I will set a date to remove them later
this year.

2) Retirement of parallel naming conventions for dump filenames

old (parallel naming convention): $FILE.$CDUMP.YYYYMMDDCC
new (production naming convention): $CDUMP.tCCz.$FILE

#2 is already in the FV3GFS (global-workflow master). Please check your
scripts to make sure the production naming convention is being used now.
The production naming convention is already supported in the current phase
1 GDA.

*That's it for now! Let me know if you have any questions about the above.
Additional notices will be sent in the coming weeks/months.*

Thanks everyone!

Kate Friedman
NOAA/NWS/NCEP/EMC Engineering and Implementation Branch
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.fv3-announce/attachments/20190611/a7f1dfd2/attachment.html 


More information about the Ncep.list.fv3-announce mailing list