[Ncep.list.nws-ncep-management] Server failure impacting NIDS/CMS applications

NOAA SDM sdm at noaa.gov
Fri Dec 16 02:23:56 UTC 2016


All,

Update:

CMS is fully functional.  All WFO sites are processing normally.

The final slave database sync is ongoing.  When complete, they will then
rebalance forecast.weather.gov and radar.weather.gov.


SDM Patrick O'Reilly

On 15 December 2016 at 20:59, NOAA SDM <sdm at noaa.gov> wrote:

> All,
>
> NCO IWSB reports the corrupt tables are clean.
>
> They are syncing the slaves now.  Once the initial slaves are back in the
> site should be functional again.
>
> Customer impact continues to be that NWS Users are unable to update web
> content.
>
> All WWA products on forecast.weather.gov, radar.weather.gov and
> alerts.weather.gov are fully functional.
>
>
> SDM Patrick O'Reilly
>
>
>
> On 15 December 2016 at 20:02, NOAA SDM <sdm at noaa.gov> wrote:
>
>> All,
>>
>> The NCO IWSB Web team continues to restore corrupt database tables.  The
>> process is taking quite some time with the number of records that are in
>> the database.
>>
>>
>> SDM Patrick O'Reilly
>>
>> On 15 December 2016 at 18:46, NOAA SDM <sdm at noaa.gov> wrote:
>>
>>> We have found several corrupt tables from the database server crash.  We
>>> are working to restore each database now.
>>>
>>> alerts.weather.gov, radar.weather.gov, forecast.weather.gov continue to
>>> be off the Kansas City site.
>>>
>>> CMS users continue to be impacted.  All watch warning advisory products
>>> are updating and disseminating properly, but CMS sites are unable to update
>>> web content including any office specific DSS items.
>>>
>>> SDM Patrick O'Reilly
>>>
>>> On 15 December 2016 at 17:55, NOAA SDM <sdm at noaa.gov> wrote:
>>>
>>>> All,
>>>>
>>>> NCO IWSB Kansas City suffered a catastrophic failure of a server which
>>>> hosts 4 virtual machines today.  The team was able to quickly migrate three
>>>> of the four machines to other host servers, however the database server was
>>>> very slow to migrate.
>>>>
>>>> alerts.weather.gov and forecast.weather.gov were moved away from the
>>>> Kansas City location, however there remains an impact to WFO ability to
>>>> update content on their web sites.
>>>>
>>>> The data base server is on line now and data is being replicated to the
>>>> slaves.  This process should take about another 45 minutes.
>>>>
>>>> Additionally, there was an outage to the NWWS Open Interface which has
>>>> no been resolved.
>>>>
>>>>
>>>> SDM Patrick O'Reilly
>>>>
>>>> --
>>>> Senior Duty Meteorologist
>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>
>>>
>>>
>>>
>>> --
>>> Senior Duty Meteorologist
>>> NOAA/NWS/NCEP/NCO/OMB
>>>
>>
>>
>>
>> --
>> Senior Duty Meteorologist
>> NOAA/NWS/NCEP/NCO/OMB
>>
>
>
>
> --
> Senior Duty Meteorologist
> NOAA/NWS/NCEP/NCO/OMB
>



-- 
Senior Duty Meteorologist
NOAA/NWS/NCEP/NCO/OMB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nws-ncep-management/attachments/20161215/e5ce815d/attachment.html 


More information about the Ncep.list.nws-ncep-management mailing list