[Ncep.list.fv3-announce] Important: ongoing and upcoming global dump archive changes

[Kate Friedman - NOAA Federal]

All,

Item #1 is now complete (GFS production dump pickup location change with
FV3GFS implementation June 12th 2019 12z). As of 12z today the GDA is
populating itself with FV3GFS production output from the Dells. The data is
being mirrored to the GDAs on both sides of WCOSS and Theia.

Users will find two new OMPS dump files being archived from production now:

$CDUMP.t$CCz.ompsn8.tm00.bufr_d
$CDUMP.t$CCz.ompst8.tm00.bufr_d


Hooray for the FV3GFS being implemented today! Congrats to all
involved! 🎉 🎉🎉🎉🎉

Kate Friedman
NOAA/NWS/NCEP/EMC Engineering and Implementation Branch


On Tue, Jun 11, 2019 at 10:29 AM Kate Friedman - NOAA Federal <
Kate.Friedman at noaa.gov> wrote:

> Dear users of the global dump archive,
>
> Below are updates on some important ongoing and upcoming global dump
> archive (GDA) changes. Most changes are for the WCOSS GDAs, although all
> users should know of the source change for GFS production dump data (first
> item below).
>
> *TL;DR <https://en.wiktionary.org/wiki/too_long;_didn%27t_read#English>:
> New GFS production dump file pickup source as of 2019061212 (tomorrow!), no
> change on user end required. Also new phase 3 GDA coming soon to WCOSS
> users!*
>
> *GFS production dump pickup location change with FV3GFS implementation
> June 12th 2019 12z*
>
> Tomorrow Wednesday June 12th the new FV3GFS will be implemented into
> operations on the Dells. As of 12z that day (2019061212) the GDA will
> follow GFS production and begin picking up production dump data from the
> Dell GFS prod location and no longer pick up from the Cray GFS prod
> location. *This should be a seamless change for users with no need to
> change the path of your $DMPDIR variables on WCOSS (yet).*
>
> For anyone running in real-time please note that the 12z data may arrive
> late in the GDAs that day if there are any issues. Please let me know if
> you are running a real-time FV3GFS experiment on the Dells so I may know
> who to contact if there are issues during the switch to the new production
> location.
>
> The new production dump data will also populate the Theia GDA from
> 2019061212 forward. *No change needed for Theia users.*
>
> *Migration of primary WCOSS GDA from phase 1 to phase 3 (also directory
> structure change!)*
>
> Additionally the migration of the WCOSS GDA from phase 1/2/Cray to phase
> 3/Cray is still ongoing. The new phase 3 GDA job is running in test mode
> and I am continuing to populate the phase 3 GDA area with data from phase
> 1. It will take another few weeks to complete the migration of the 25TBs of
> data in the primary part of the GDA.
>
> The oldest GDA dump files are still in place on the Cray and phase 2. The
> Cray portion of the GDA will remain past the retirement of phase 1/2. As
> was done on phase 1 the Cray dump folders will be symlinked to from the
> primary phase 3 GDA location. Users will continue to point to the primary
> $DMPDIR to get any of the available dump data. More on the future of the
> phase 2 portion of the GDA below.
>
> The biggest part of this phase 1 to phase 3 change is the structure of the
> GDA. The new phase 3 GDA will mimic the production directory structure
> changes introduced with the FV3GFS:
>
> current GDA:
>
> $DMPDIR/YYYYMMDDCC/${CDUMP}${SUFFIX}/$FILE.$CDUMP.YYYYMMDDCC
>
> new GDA:
>
> $DMPDIR/${CDUMP}${SUFFIX}.YYYYMMDD/CC/$CDUMP.tCCz.$FILE
>
> ....where SUFFIX is nothing for production data, "nr" for non-restricted
> data, "p" for pre-production data, and "x" for experimental data.
>
> Any needed workflow changes to global-workflow will be announced and
> committed to the master when the new phase 3 GDA is ready for use.
>
> *Retiring phase 1/2 GDA and parallel naming conventions for dump files*
>
> When the migration of data is complete and the phase 3 GDA job is stable I
> will proceed with retiring the phase 1/2 GDA and making the phase 3 GDA the
> primary one on WCOSS. Notices with more details will be sent when this is
> going to occur. My aim is that this change will also be seamless to users.
>
> There are two big parts of this retirement:
>
> 1) The age-off of the oldest dump data currently living on phase 2
> (2015010100 - mid-2016). If you have plans to run FV3GFS experiments on
> WCOSS for the oldest data please try to complete them before the retirement
> of the phase 1/2 parts of WCOSS. I will set a date to remove them later
> this year.
>
> 2) Retirement of parallel naming conventions for dump filenames
>
> old (parallel naming convention): $FILE.$CDUMP.YYYYMMDDCC
> new (production naming convention): $CDUMP.tCCz.$FILE
>
> #2 is already in the FV3GFS (global-workflow master). Please check your
> scripts to make sure the production naming convention is being used now.
> The production naming convention is already supported in the current phase
> 1 GDA.
>
> *That's it for now! Let me know if you have any questions about the above.
> Additional notices will be sent in the coming weeks/months.*
>
> Thanks everyone!
>
> Kate Friedman
> NOAA/NWS/NCEP/EMC Engineering and Implementation Branch
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.fv3-announce/attachments/20190612/4983653a/attachment.html