[Ncep.list.nws-ncep-management] UPDATE #7 - CMS issues

NOAA SDM sdm at noaa.gov
Thu Aug 3 06:23:04 UTC 2017


UPDATE:  The SDM's were asking the NWS field offices and others for
feedback about on-going CMS issues with each of tonight's NOUS42 and
NWSchat messages.. currently 19 WFO and/or CWSU's are still reporting
issues. It is assumed that a few of these have not replied back when the
issues were resolved there. But more then half of these have responded
recently that problems still exist. No new updates on trouble
shooting/resolving these issues have been rec'd recently. Expect the next
update will be coming after 6am or when noteworthy feedback is received
from the field.

NCEP/SDM  Grant Newby



On 3 August 2017 at 00:19, NOAA SDM <sdm at noaa.gov> wrote:

> UPDATE: A handful of sites are still reporting some web content issues.
> NCO will now move to working these on a case-by-case basis until all are
> resolved.
>
> SDM
>
>
> On 2 August 2017 at 18:12, NOAA SDM <sdm at noaa.gov> wrote:
>
>> UPDATE: Though CMS performance has improved, WFOs continue to report
>> issues with web content updating.. NCO is currently working to get root
>> cause of the remaining issues, thought to be related to yesterday's
>> rolled-back upgrade of CMS.
>>
>> SDM
>>
>>
>> On 2 August 2017 at 16:59, NOAA SDM <sdm at noaa.gov> wrote:
>>
>>> UPDATE: NCO believes they have discovered the root cause of the CMS
>>> issues since yesterday afternoon.  The Central Region ArcSite appliance
>>> was replaced yesterday and was returned to service earlier this afternoon.
>>> While it was out of service, NIDS CMS connections would time out trying to
>>> connect to the ArcSite appliance to send log information, not allowing new
>>> connections for other sites. Once the CR ArcSite appliance was returned to
>>> service this afternoon, the NIDS CMS connections would be successful,
>>> making NIDS CMS healthy.
>>>
>>> Sites are reporting better connectivity and web content is updating
>>> where it previously hadn't.  We will continue to monitor until we are
>>> certain the issue has been rectified.
>>>
>>> NCO will also ensure that any maintenance on the CR ArcSite server does
>>> not impact NIDS CMS in the future.
>>>
>>> SDM
>>>
>>> On 2 August 2017 at 15:00, NOAA SDM <sdm at noaa.gov> wrote:
>>>
>>>> UPDATE: Red Hat is engaged with NCO tech support towards resolving the
>>>> CMS rsync issues.   We are awaiting their analysis of the situation.
>>>>
>>>> On 2 August 2017 at 17:50, NOAA SDM <sdm at noaa.gov> wrote:
>>>>
>>>>> UPDATE: NCO tech support rebooted the CMS machine, in an attempt to
>>>>> resolve the rsync issues.  The machine reboot was unsuccessful.  In
>>>>> response, tech support have opened a high priority case with Red Hat, to
>>>>> enlist additional troubleshooting support.
>>>>>
>>>>> On 2 August 2017 at 16:29, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>
>>>>>> UPDATE: NCO tech support has performed tests on a system that's used
>>>>>> to run CMS rsync processes.  So far, the CMS rsync process is not
>>>>>> responding to test requests.  In response, NCO tech support will reboot the
>>>>>> affected CMS machine that's currently running the rsync process.
>>>>>>
>>>>>> On 2 August 2017 at 15:33, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>
>>>>>>> UPDATE: NCO tech support are using a dedicated system to identify
>>>>>>> and formulate a solution to test, in the attempt to resolve ongoing rsync
>>>>>>> issues with the Content Management System (CMS).  In the meantime, support
>>>>>>> are investigating the root cause behind the CMS rsync issues.
>>>>>>>
>>>>>>> On 2 August 2017 at 09:51, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>>
>>>>>>>> Update:
>>>>>>>>
>>>>>>>> Most functionality to CMS was restored around 2am EDT.  There are
>>>>>>>> still a few isolated issues affecting some NWS offices ability to send web
>>>>>>>> content to CMS or updating weather headlines. NCO support continues to look
>>>>>>>> into these lingering issues.
>>>>>>>>
>>>>>>>> SDM
>>>>>>>>
>>>>>>>> On 2 August 2017 at 01:34, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>>>
>>>>>>>>> Update:
>>>>>>>>>
>>>>>>>>> The issues with IRIS/iNWS and HazCollect applications are
>>>>>>>>> resolved. All alerts are flowing to FEMA and iNWS.
>>>>>>>>>
>>>>>>>>> NCO continues to work through issues rolling back the CMS upgrade
>>>>>>>>> today. Tier 3 support has been activated to assist.
>>>>>>>>>
>>>>>>>>> SDM
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 2 August 2017 at 00:43, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Update:
>>>>>>>>>>
>>>>>>>>>> The IRIS application (alerts to FEMA) has almost recovered.
>>>>>>>>>> Admins are finishing up recovery.
>>>>>>>>>>
>>>>>>>>>> The CMS upgrade that was implemented this afternoon is being
>>>>>>>>>> rolled back to the previous version.  The changes should be reverted within
>>>>>>>>>> 30 minutes, restoring the WFOs ability to upload custom content to their
>>>>>>>>>> web sites.
>>>>>>>>>>
>>>>>>>>>> Will update when all systems are restored.
>>>>>>>>>>
>>>>>>>>>> SDM
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Senior Duty Meteorologist
>>>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Senior Duty Meteorologist
>>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Senior Duty Meteorologist
>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Senior Duty Meteorologist
>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Senior Duty Meteorologist
>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>
>>>
>>>
>>>
>>> --
>>> Senior Duty Meteorologist
>>> NOAA/NWS/NCEP/NCO/OMB
>>>
>>
>>
>>
>> --
>> Senior Duty Meteorologist
>> NOAA/NWS/NCEP/NCO/OMB
>>
>
>
>
> --
> Senior Duty Meteorologist
> NOAA/NWS/NCEP/NCO/OMB
>



-- 
Senior Duty Meteorologist
NOAA/NWS/NCEP/NCO/OMB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nws-ncep-management/attachments/20170803/b7093f11/attachment.html 


More information about the Ncep.list.nws-ncep-management mailing list