[Ncep.list.idp.mrms.users] MRMS product degraded -- 20150802

Tue Aug 4 10:47:01 UTC 2015

Ben, Daryl,

We have alerts when any product hasn't been updated in 1.5x the frequency
we expect.  So for a 2 minute product we expect to see a delay of no more
than 3 minutes.  So yes we have flags for all of this, but depending on the
issue there might not be much we can do simply because of the high
expiration of the products we are producing.  If we get a CPU flag on one
of our radar machines and the data propagates slowly through the system or
it's late there is little to nothing we can do except try and find out what
caused the CPU issue and add resources accordingly if that is the problem.

It appears the problems were/are related to something in VMware.  I think
our hardware team is still working with them on a solution, but I have not
gotten all the details yet since I was out yesterday.   I'm afraid you will
probably continue to see this until it is resolved.  Also, when we have
problems like this the SDM calls us and we try to fix it, but this was not
application specific so any attempts that were made to correct the system
was simply not working due to the VMware problems.

Thanks,
Holly

On Mon, Aug 3, 2015 at 9:45 PM, Benjamin Schwedler <
benjamin.schwedler at noaa.gov> wrote:

> My basic monitoring based on file age also indicates that we have been
> missing multiple files today, particularly precipitation accumulations.
> Things were semi-stable by about 15Z, but it appears that there may still
> be lingering issues.
>
> Ben
>
>
> On 08/03/2015 08:18 PM, daryl herzmann wrote:
>
>> Hello,
>>
>> I have noticed a number of files missing in the LDM feed today (Monday)
>> and even now (this evening, 4 August UTC).  For example these two files
>> are missing
>>
>> PrecipRate_00.00_20150804-003800.grib2.gz
>> PrecipRate_00.00_20150804-004000.grib2.gz
>>
>> I checked this http://140.172.25.182/OPERATIONAL_NCEP_MRMS_GRIB2 and they
>> are missing there as well.  Do you folks not have alerting of when this
>> happens?
>>
>> daryl
>>
>> On Mon, 3 Aug 2015, Jaysankar Iyer - NOAA Affiliate wrote:
>>
>> Hi Daryl,
>>>
>>> We had numerous hardware and system related issues (hopefully all
>>> resolved)
>>> over the weekend that led to IDP MRMS systems degrade in performance.
>>>
>>> Let us know if you continue to see issues.
>>>
>>> Thanks much
>>>
>>> -Jay Iyer (MRMS Support Team)
>>>
>>> On Mon, Aug 3, 2015 at 4:51 PM, daryl herzmann <akrherz at iastate.edu>
>>> wrote:
>>>
>>> Hello,
>>>>
>>>> Any updates on this issue?  I have noticed that problems persisted into
>>>> this morning (Monday).
>>>>
>>>> daryl
>>>>
>>>> On Sun, 2 Aug 2015, Holly Uhlenhake - NOAA Affiliate wrote:
>>>>
>>>> It looks like it's hardware related right now.  Maybe Derek will get
>>>>> more
>>>>> info soon :-\
>>>>>
>>>>> On Sun, Aug 2, 2015 at 10:07 AM, Karen Cooper <karen.cooper at noaa.gov>
>>>>>
>>>> wrote:
>>>>
>>>>> Any idea what's causing the delays?
>>>>>>
>>>>>> Sent from my iPhone
>>>>>>
>>>>>> On Aug 2, 2015, at 8:46 AM, Derek VanPelt - NOAA Affiliate <
>>>>>>>
>>>>>> derek.vanpelt at noaa.gov> wrote:
>>>>>>
>>>>>>> The IDP MRMS systems have been suffering from slowdowns in processing
>>>>>>>
>>>>>> this morning since ~10Z.  Some MRMS products may be missing or delayed
>>>>>> until we can get a fix in place.
>>>>>>
>>>>>>> We are working the issue, and will send out an update when we have
>>>>>>> one.
>>>>>>>
>>>>>>> Thank you,
>>>>>>>
>>>>>>> Derek Van Pelt
>>>>>>> NCEP/NCO/IDP
>>>>>>> _______________________________________________
>>>>>>> Ncep.list.idp.mrms.users mailing list
>>>>>>> Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
>>>>>>>
>>>>>>>
>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
>>>>
>>>>> _______________________________________________
>>>>>> Ncep.list.idp.mrms.users mailing list
>>>>>> Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
>>>>>>
>>>>>>
>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
>>>> --
>>>> /**
>>>>    * Daryl Herzmann
>>>>    * Assistant Scientist -- Iowa Environmental Mesonet
>>>>    * http://mesonet.agron.iastate.edu
>>>>    */
>>>> _______________________________________________
>>>> Ncep.list.idp.mrms.users mailing list
>>>> Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
>>>>
>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
>>>>
>>>>
> --
> Benjamin R. J. Schwedler
> NWS Aviation Weather Testbed - NextGen/AWT Development Meteorologist
> CIRA/Colorado State University
> NOAA/Aviation Weather Center
> 7220 NW 101st Terrace
> Kansas City, MO 64153-2371
> 816.584.7268
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.idp.mrms.users/attachments/20150804/4862416e/attachment.html 