[Ncep.list.nws-ncep-management] UPDATE #8 - CMS issues

NOAA SDM sdm at noaa.gov
Thu Aug 3 20:28:21 UTC 2017


UPDATE: NCO IT support confirmed that the API has been returned to full
functionality.  They are now addressing any lingering issues, as being
reported by individual WFOs.

On 3 August 2017 at 13:47, NOAA SDM <sdm at noaa.gov> wrote:

> All,
>
> Numerous WFOs continue reporting issues with their CMS updates.
>
> In the meantime, our IT team has determined that there is an issue with
> the ingest of data into the College Park API, which is a service that many
> WFOs use to feed their CMS pages.  In addition, the Boulder API systems are
> throwing errors due to an upstream issue with NGITWS feeding data into the
> API.  NCO IT Support has teams working on both of these issues.
>
> Thanks,
> SDM Kelly Kempisty
>
> On 3 August 2017 at 06:23, NOAA SDM <sdm at noaa.gov> wrote:
>
>> UPDATE:  The SDM's were asking the NWS field offices and others for
>> feedback about on-going CMS issues with each of tonight's NOUS42 and
>> NWSchat messages.. currently 19 WFO and/or CWSU's are still reporting
>> issues. It is assumed that a few of these have not replied back when the
>> issues were resolved there. But more then half of these have responded
>> recently that problems still exist. No new updates on trouble
>> shooting/resolving these issues have been rec'd recently. Expect the next
>> update will be coming after 6am or when noteworthy feedback is received
>> from the field.
>>
>> NCEP/SDM  Grant Newby
>>
>>
>>
>> On 3 August 2017 at 00:19, NOAA SDM <sdm at noaa.gov> wrote:
>>
>>> UPDATE: A handful of sites are still reporting some web content issues.
>>> NCO will now move to working these on a case-by-case basis until all are
>>> resolved.
>>>
>>> SDM
>>>
>>>
>>> On 2 August 2017 at 18:12, NOAA SDM <sdm at noaa.gov> wrote:
>>>
>>>> UPDATE: Though CMS performance has improved, WFOs continue to report
>>>> issues with web content updating.. NCO is currently working to get root
>>>> cause of the remaining issues, thought to be related to yesterday's
>>>> rolled-back upgrade of CMS.
>>>>
>>>> SDM
>>>>
>>>>
>>>> On 2 August 2017 at 16:59, NOAA SDM <sdm at noaa.gov> wrote:
>>>>
>>>>> UPDATE: NCO believes they have discovered the root cause of the CMS
>>>>> issues since yesterday afternoon.  The Central Region ArcSite
>>>>> appliance was replaced yesterday and was returned to service earlier this
>>>>> afternoon. While it was out of service, NIDS CMS connections would time out
>>>>> trying to connect to the ArcSite appliance to send log information, not
>>>>> allowing new connections for other sites. Once the CR ArcSite appliance was
>>>>> returned to service this afternoon, the NIDS CMS connections would be
>>>>> successful, making NIDS CMS healthy.
>>>>>
>>>>> Sites are reporting better connectivity and web content is updating
>>>>> where it previously hadn't.  We will continue to monitor until we are
>>>>> certain the issue has been rectified.
>>>>>
>>>>> NCO will also ensure that any maintenance on the CR ArcSite server
>>>>> does not impact NIDS CMS in the future.
>>>>>
>>>>> SDM
>>>>>
>>>>> On 2 August 2017 at 15:00, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>
>>>>>> UPDATE: Red Hat is engaged with NCO tech support towards resolving
>>>>>> the CMS rsync issues.   We are awaiting their analysis of the situation.
>>>>>>
>>>>>> On 2 August 2017 at 17:50, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>
>>>>>>> UPDATE: NCO tech support rebooted the CMS machine, in an attempt to
>>>>>>> resolve the rsync issues.  The machine reboot was unsuccessful.  In
>>>>>>> response, tech support have opened a high priority case with Red Hat, to
>>>>>>> enlist additional troubleshooting support.
>>>>>>>
>>>>>>> On 2 August 2017 at 16:29, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>>
>>>>>>>> UPDATE: NCO tech support has performed tests on a system that's
>>>>>>>> used to run CMS rsync processes.  So far, the CMS rsync process is not
>>>>>>>> responding to test requests.  In response, NCO tech support will reboot the
>>>>>>>> affected CMS machine that's currently running the rsync process.
>>>>>>>>
>>>>>>>> On 2 August 2017 at 15:33, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>>>
>>>>>>>>> UPDATE: NCO tech support are using a dedicated system to identify
>>>>>>>>> and formulate a solution to test, in the attempt to resolve ongoing rsync
>>>>>>>>> issues with the Content Management System (CMS).  In the meantime, support
>>>>>>>>> are investigating the root cause behind the CMS rsync issues.
>>>>>>>>>
>>>>>>>>> On 2 August 2017 at 09:51, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Update:
>>>>>>>>>>
>>>>>>>>>> Most functionality to CMS was restored around 2am EDT.  There are
>>>>>>>>>> still a few isolated issues affecting some NWS offices ability to send web
>>>>>>>>>> content to CMS or updating weather headlines. NCO support continues to look
>>>>>>>>>> into these lingering issues.
>>>>>>>>>>
>>>>>>>>>> SDM
>>>>>>>>>>
>>>>>>>>>> On 2 August 2017 at 01:34, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> Update:
>>>>>>>>>>>
>>>>>>>>>>> The issues with IRIS/iNWS and HazCollect applications are
>>>>>>>>>>> resolved. All alerts are flowing to FEMA and iNWS.
>>>>>>>>>>>
>>>>>>>>>>> NCO continues to work through issues rolling back the CMS
>>>>>>>>>>> upgrade today. Tier 3 support has been activated to assist.
>>>>>>>>>>>
>>>>>>>>>>> SDM
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 2 August 2017 at 00:43, NOAA SDM <sdm at noaa.gov> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Update:
>>>>>>>>>>>>
>>>>>>>>>>>> The IRIS application (alerts to FEMA) has almost recovered.
>>>>>>>>>>>> Admins are finishing up recovery.
>>>>>>>>>>>>
>>>>>>>>>>>> The CMS upgrade that was implemented this afternoon is being
>>>>>>>>>>>> rolled back to the previous version.  The changes should be reverted within
>>>>>>>>>>>> 30 minutes, restoring the WFOs ability to upload custom content to their
>>>>>>>>>>>> web sites.
>>>>>>>>>>>>
>>>>>>>>>>>> Will update when all systems are restored.
>>>>>>>>>>>>
>>>>>>>>>>>> SDM
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Senior Duty Meteorologist
>>>>>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Senior Duty Meteorologist
>>>>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Senior Duty Meteorologist
>>>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Senior Duty Meteorologist
>>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Senior Duty Meteorologist
>>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Senior Duty Meteorologist
>>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Senior Duty Meteorologist
>>>> NOAA/NWS/NCEP/NCO/OMB
>>>>
>>>
>>>
>>>
>>> --
>>> Senior Duty Meteorologist
>>> NOAA/NWS/NCEP/NCO/OMB
>>>
>>
>>
>>
>> --
>> Senior Duty Meteorologist
>> NOAA/NWS/NCEP/NCO/OMB
>>
>
>
>
> --
> Senior Duty Meteorologist
> NOAA/NWS/NCEP/NCO/OMB
>



-- 
Senior Duty Meteorologist
NOAA/NWS/NCEP/NCO/OMB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.nws-ncep-management/attachments/20170803/e4e90f5f/attachment-0001.html 


More information about the Ncep.list.nws-ncep-management mailing list