[Ncep.list.nws-ncep-management] Server failure impacting NIDS/CMS applications

NOAA SDM sdm at noaa.gov
Fri Dec 16 02:59:36 UTC 2016


All,

Both Silver Spring and Kansas City web farms are operational.

CMS is operational.

NCO IWSB has asked users who reported issues to verify that their issues
are resolved.


SDM Patrick

On 15 December 2016 at 21:23, NOAA SDM <sdm at noaa.gov> wrote:

> All,
>
> Update:
>
> CMS is fully functional.  All WFO sites are processing normally.
>
> The final slave database sync is ongoing.  When complete, they will then
> rebalance forecast.weather.gov and radar.weather.gov.
>
>
> SDM Patrick O'Reilly
>
> On 15 December 2016 at 20:59, NOAA SDM <sdm at noaa.gov> wrote:
>
>> All,
>>
>> NCO IWSB reports the corrupt tables are clean.
>>
>> They are syncing the slaves now.  Once the initial slaves are back in the
>> site should be functional again.
>>
>> Customer impact continues to be that NWS Users are unable to update web
>> content.
>>
>> All WWA products on forecast.weather.gov, radar.weather.gov and
>> alerts.weather.gov are fully functional.
>>
>>
>> SDM Patrick O'Reilly
>>
>>
>>
>> On 15 December 2016 at 20:02, NOAA SDM <sdm at noaa.gov> wrote:
>>
>>> All,
>>>
>>> The NCO IWSB Web team continues to restore corrupt database tables.
>>> The process is taking quite some time with the number of records that are
>>> in the database.
>>>
>>>
>>> SDM Patrick O'Reilly
>>>
>>> On 15 December 2016 at 18:46, NOAA SDM <sdm at noaa.gov> wrote:
>>>
>>>> We have found several corrupt tables from the database server crash.
>>>> We are working to restore each database now.
>>>>
>>>> alerts.weather.gov, radar.weather.gov, forecast.weather.gov continue
>>>> to be off the Kansas City site.
>>>>
>>>> CMS users continue to be impacted.  All watch warning advisory products
>>>> are updating and disseminating properly, but CMS sites are unable to update
>>>> web content including any office specific DSS items.
>>>>
>>>> SDM Patrick O'Reilly
>>>>
>>>> On 15 December 2016 at 17:55, NOAA SDM <sdm at noaa.gov> wrote:
>>>>
>>>>> All,
>>>>>
>>>>> NCO IWSB Kansas City suffered a catastrophic failure of a server which
>>>>> hosts 4 virtual machines today.  The team was able to quickly migrate three
>>>>> of the four machines to other host servers, however the database server was
>>>>> very slow to migrate.
>>>>>
>>>>> alerts.weather.gov and forecast.weather.gov were moved away from the
>>>>> Kansas City location, however there remains an impact to WFO ability to
>>>>> update content on their web sites.
>>>>>
>>>>> The data base server is on line now and data is being replicated to
>>>>> the slaves.  This process should take about another 45 minutes.
>>>>>
>>>>> Additionally, there was an outage to the NWWS Open Interface which has
>>>>> no been resolved.
>>>>>
>>>>>
>>>>> SDM Patrick O'Reilly
>>>>>
>>>>> --
>>>>> Senior Duty Meteorologist
>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Senior Duty Meteorologist
>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>
>>>
>>>
>>>
>>> --
>>> Senior Duty Meteorologist
>>> NOAA/NWS/NCEP/NCO/OMB
>>>
>>
>>
>>
>> --
>> Senior Duty Meteorologist
>> NOAA/NWS/NCEP/NCO/OMB
>>
>
>
>
> --
> Senior Duty Meteorologist
> NOAA/NWS/NCEP/NCO/OMB
>



-- 
Senior Duty Meteorologist
NOAA/NWS/NCEP/NCO/OMB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nws-ncep-management/attachments/20161215/892bbb72/attachment-0001.html 


More information about the Ncep.list.nws-ncep-management mailing list