[Ncep.list.nws-ncep-management] DATABASE ISSUES EXPERIENCED AT THE GATEWAY - Sunday June 21, 2015

NOAA SDM sdm at noaa.gov
Sun Jun 21 20:03:09 UTC 2015

Here is a summary off issues at the NCO-SS / NWSTG.  TG nvestigations will
continue into the initial root cause continues.

At 1654Z, the TG began processing data slowly.  At 1714Z one of the two
openswitches at the TG crashed.  Data continued to be processed slowly.
Tier II support was contacted.

At around 1730Z, the second openswitch went down and the TG stopped
processing all data.  The SDM began receiving numerous phone call inquires
while the TG continued to work the issue.  A conference call for Tier II
support was established.

Just after 18z..  Openswitches were restarted and the TG processed about
50,000 messages very slowly.  At this point the DBA noticed that the CPU
utilization was very high. Added more resources to the primary DB, added
threads to process old data, restarted hubs at approximately 1836Z and all
data was caught up by 1900Z.

The NWS field offices and NCEP centers experienced data delays for a
variety of datasets.. but most importantly sfc metars and lightning
datasets into AWIPS and NAWIPS. Numerous NOUS42 Admin messages were sent to
the field about the current status of the Gateway problems.

The NCEP SDM noted hourly dataset delays for 17z and 18z but eventually
most data was recovered/received by 1900z for both hours.

Hourly dataset products were delayed about 23 minutes for 17z.. and delays
up to 33 minutes were experienced for 18z products.

Senior Duty Meteorologist
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nws-ncep-management/attachments/20150621/f356faa5/attachment.html 

More information about the Ncep.list.nws-ncep-management mailing list