[Ncep.list.idp.mrms.users] MRMS product degraded -- 20150802

Wed Aug 5 12:47:29 UTC 2015

Hi Stefan,

We are hoping to get an update at our operations meeting this morning (13Z)
and then we'll update our users.  Expect to hear from one of us later this
morning.

Thanks,
Holly

On Wed, Aug 5, 2015 at 8:44 AM, Cecelski, Stefan <
Stefan.Cecelski at weatheranalytics.com> wrote:

> All,
>
> FYI - Still encountering some missing files, specifically the 1-HR
> Gauge-Corrected QPE @ 7,8,10 UTC today and all hail related products
> (MESH,MESH-30,MESH-60,POSH,…) between ~9:10UTC and ~10:16UTC today. Missing
> over LDM as well as the HTTP server.
>
> V/R,
>
> Stefan
> *Stefan Cecelski, Ph.D.*
> *Weather Analytics*
> Director of Scientific Modeling
> www.weatheranalytics.com
>
> On Aug 4, 2015, at 6:47 AM, Holly Uhlenhake - NOAA Affiliate <
> holly.uhlenhake at noaa.gov> wrote:
>
> Ben, Daryl,
>
> We have alerts when any product hasn't been updated in 1.5x the frequency
> we expect.  So for a 2 minute product we expect to see a delay of no more
> than 3 minutes.  So yes we have flags for all of this, but depending on the
> issue there might not be much we can do simply because of the high
> expiration of the products we are producing.  If we get a CPU flag on one
> of our radar machines and the data propagates slowly through the system or
> it's late there is little to nothing we can do except try and find out what
> caused the CPU issue and add resources accordingly if that is the problem.
>
> It appears the problems were/are related to something in VMware.  I think
> our hardware team is still working with them on a solution, but I have not
> gotten all the details yet since I was out yesterday.   I'm afraid you will
> probably continue to see this until it is resolved.  Also, when we have
> problems like this the SDM calls us and we try to fix it, but this was not
> application specific so any attempts that were made to correct the system
> was simply not working due to the VMware problems.
>
> Thanks,
> Holly
>
> On Mon, Aug 3, 2015 at 9:45 PM, Benjamin Schwedler <
> benjamin.schwedler at noaa.gov> wrote:
>
>> My basic monitoring based on file age also indicates that we have been
>> missing multiple files today, particularly precipitation accumulations.
>> Things were semi-stable by about 15Z, but it appears that there may still
>> be lingering issues.
>>
>> Ben
>>
>>
>> On 08/03/2015 08:18 PM, daryl herzmann wrote:
>>
>>> Hello,
>>>
>>> I have noticed a number of files missing in the LDM feed today (Monday)
>>> and even now (this evening, 4 August UTC).  For example these two files
>>> are missing
>>>
>>> PrecipRate_00.00_20150804-003800.grib2.gz
>>> PrecipRate_00.00_20150804-004000.grib2.gz
>>>
>>> I checked this http://140.172.25.182/OPERATIONAL_NCEP_MRMS_GRIB2 and
>>> they
>>> are missing there as well.  Do you folks not have alerting of when this
>>> happens?
>>>
>>> daryl
>>>
>>> On Mon, 3 Aug 2015, Jaysankar Iyer - NOAA Affiliate wrote:
>>>
>>> Hi Daryl,
>>>>
>>>> We had numerous hardware and system related issues (hopefully all
>>>> resolved)
>>>> over the weekend that led to IDP MRMS systems degrade in performance.
>>>>
>>>> Let us know if you continue to see issues.
>>>>
>>>> Thanks much
>>>>
>>>> -Jay Iyer (MRMS Support Team)
>>>>
>>>> On Mon, Aug 3, 2015 at 4:51 PM, daryl herzmann <akrherz at iastate.edu>
>>>> wrote:
>>>>
>>>> Hello,
>>>>>
>>>>> Any updates on this issue?  I have noticed that problems persisted into
>>>>> this morning (Monday).
>>>>>
>>>>> daryl
>>>>>
>>>>> On Sun, 2 Aug 2015, Holly Uhlenhake - NOAA Affiliate wrote:
>>>>>
>>>>> It looks like it's hardware related right now.  Maybe Derek will get
>>>>>> more
>>>>>> info soon :-\
>>>>>>
>>>>>> On Sun, Aug 2, 2015 at 10:07 AM, Karen Cooper <karen.cooper at noaa.gov>
>>>>>>
>>>>> wrote:
>>>>>
>>>>>> Any idea what's causing the delays?
>>>>>>>
>>>>>>> Sent from my iPhone
>>>>>>>
>>>>>>> On Aug 2, 2015, at 8:46 AM, Derek VanPelt - NOAA Affiliate <
>>>>>>>>
>>>>>>> derek.vanpelt at noaa.gov> wrote:
>>>>>>>
>>>>>>>> The IDP MRMS systems have been suffering from slowdowns in
>>>>>>>> processing
>>>>>>>>
>>>>>>> this morning since ~10Z.  Some MRMS products may be missing or
>>>>>>> delayed
>>>>>>> until we can get a fix in place.
>>>>>>>
>>>>>>>> We are working the issue, and will send out an update when we have
>>>>>>>> one.
>>>>>>>>
>>>>>>>> Thank you,
>>>>>>>>
>>>>>>>> Derek Van Pelt
>>>>>>>> NCEP/NCO/IDP
>>>>>>>> _______________________________________________
>>>>>>>> Ncep.list.idp.mrms.users mailing list
>>>>>>>> Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
>>>>>>>>
>>>>>>>>
>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
>>>>>
>>>>>> _______________________________________________
>>>>>>> Ncep.list.idp.mrms.users mailing list
>>>>>>> Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
>>>>>>>
>>>>>>>
>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
>>>>> --
>>>>> /**
>>>>>    * Daryl Herzmann
>>>>>    * Assistant Scientist -- Iowa Environmental Mesonet
>>>>>    * http://mesonet.agron.iastate.edu
>>>>>    */
>>>>> _______________________________________________
>>>>> Ncep.list.idp.mrms.users mailing list
>>>>> Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
>>>>>
>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
>>>>>
>>>>>
>> --
>> Benjamin R. J. Schwedler
>> NWS Aviation Weather Testbed - NextGen/AWT Development Meteorologist
>> CIRA/Colorado State University
>> NOAA/Aviation Weather Center
>> 7220 NW 101st Terrace
>> Kansas City, MO 64153-2371
>> 816.584.7268
>>
>>
> _______________________________________________
> Ncep.list.idp.mrms.users mailing list
> Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.idp.mrms.users/attachments/20150805/bd6b1f7c/attachment-0001.html 