[Ncep.list.nws-ncep-management] UPDATE #9 (Re: Network maintenance work - network traffic move to upgraded router)

NOAA SDM sdm at noaa.gov
Tue Apr 18 17:19:52 UTC 2017


UPDATE #9: As of 1700Z, DNS capacity is at 83 percent, with technical
support troubleshooting an issue on one DNS server that remains offline.
In the meantime, technical support continues to work on restoring backup
Intranet service for the NCEP Centers, along with restoring backup IDP
Postgres database services.



On 18 April 2017 at 16:11, NOAA SDM <sdm at noaa.gov> wrote:

> UPDATE #8: Technical staff are continuing work on the College Park data
> center, restoring IT infrastructure to achieve full redundancy.  As of
> 1555Z, staff made some headway on the College Park IDP infrastructure by
> returning a secondary dataflow server (ncodf1) back to service; one that
> also run data ingest for all IDP applications.  In the meantime, staff is
> troubleshooting a technical issue that includes an MRMS server, and a
> bridge server between Boulder and College Park.
>
> On 18 April 2017 at 14:53, NOAA SDM <sdm at noaa.gov> wrote:
>
>> UPDATE #7: Technical staff have resumed work on the College Park data
>> center.  As of 1445Z, Canadian surface observations and radar Level 2 data
>> are up and running, in addition to a dataflow server (ncodf2) that runs
>> data ingest for all IDP applications.
>>
>> SDM
>>
>>
>> On 18 April 2017 at 00:50, NOAA SDM <sdm at noaa.gov> wrote:
>>
>>> UPDATE #6: FINAL UPDATE FOR THE EVENING
>>>
>>> NCO technical staff have made enough headway to stabilize the NCWCP data
>>> center, though IDP is operating in a degraded mode.  This will cause the
>>> following to operate in a degraded mode: now COAST, IDP-GIS and Chart Tile
>>> applications.  Canadian surface observations and radar data are also
>>> impacted, coming in later than usual.  The technical staff will resume work
>>> on the College Park data center on Tuesday morning.
>>>
>>> SDM
>>>
>>> On 17 April 2017 at 21:55, NOAA SDM <sdm at noaa.gov> wrote:
>>>
>>>> UPDATE #5: The recovery continues with the NCO technical staff making
>>>> headway on College Park IDP infrastructure.  Nowcoast is now significantly
>>>> recovered and serving current data.  Reminder that all other major IDP
>>>> applications are and have been running cleanly from the Boulder instance of
>>>> IDP.
>>>>
>>>> SDM
>>>>
>>>> On 17 April 2017 at 20:15, NOAA SDM <sdm at noaa.gov> wrote:
>>>>
>>>>> UPDATE #4: NCO technical staff are still working through fallout from
>>>>> the earlier network maintenance.  Operational Nowcoast/IDP-GIS are still
>>>>> impacted as they run solely from College Park.  There are also ongoing DNS
>>>>> issues in College Park affecting local centers product dissemination, these
>>>>> are being worked with a high priority. There are still numerous
>>>>> infrastructure and database systems down.  More updates will be forthcoming.
>>>>>
>>>>> On 17 April 2017 at 18:20, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>
>>>>>> UPDATE #3: NCO technical support have identified residual issues that
>>>>>> include NCEP listservers and NCEP Centers' virtual machines being in
>>>>>> read-only mode, with impacts including a delayed receipt in list server
>>>>>> email messages.  Support staff are actively working to restore these
>>>>>> systems to full service.
>>>>>>
>>>>>>
>>>>>> On 17 April 2017 at 17:55, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>
>>>>>>> UPDATE #2: Shortly before 16Z (1200L), NCO/NCEP network support
>>>>>>> encountered a network outage while attempting to move network traffic to an
>>>>>>> upgraded router.  NCO/NCEP network support now confirms that Internet
>>>>>>> services have been restored.
>>>>>>>
>>>>>>> In the meantime, IDP services and other NCEP services within the
>>>>>>> College Park data center are still encountering residual issues that are
>>>>>>> being addressed.  This include DNS, NLETS database, and Nowcoast/IDP-GIS
>>>>>>> applications.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 17 April 2017 at 17:25, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>>
>>>>>>>> *UPDATE: NCO/NCEP network support encountered a technical issue
>>>>>>>> while moving network traffic to an upgraded router.   Changes done to bring
>>>>>>>> down a second router when reversed, did not go in as expected.  Support
>>>>>>>> performed a manual review of configuration differences between the routers
>>>>>>>> and resolved the situation.   More updates will be provided as they arrive.*
>>>>>>>>
>>>>>>>> On 17 April 2017 at 15:49, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>>>
>>>>>>>>> All,
>>>>>>>>>
>>>>>>>>> As part of today's network maintenance, NCO/NCEP network support
>>>>>>>>> is scheduled to move network traffic to an upgraded router at 1545Z
>>>>>>>>> today.   Outages of up to two minutes may occur while the traffic switch
>>>>>>>>> takes place.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Senior Duty Meteorologist
>>>>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Senior Duty Meteorologist
>>>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Senior Duty Meteorologist
>>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Senior Duty Meteorologist
>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Senior Duty Meteorologist
>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Senior Duty Meteorologist
>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>
>>>
>>>
>>>
>>> --
>>> Senior Duty Meteorologist
>>> NOAA/NWS/NCEP/NCO/OMB
>>>
>>
>>
>>
>> --
>> Senior Duty Meteorologist
>> NOAA/NWS/NCEP/NCO/OMB
>>
>
>
>
> --
> Senior Duty Meteorologist
> NOAA/NWS/NCEP/NCO/OMB
>



-- 
Senior Duty Meteorologist
NOAA/NWS/NCEP/NCO/OMB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nws-ncep-management/attachments/20170418/0d803ebc/attachment.html 


More information about the Ncep.list.nws-ncep-management mailing list