[Ncep.list.idp.mrms.users] MRMS product degraded -- 20150802

Wed Aug 5 12:44:11 UTC 2015

All,

FYI - Still encountering some missing files, specifically the 1-HR Gauge-Corrected QPE @ 7,8,10 UTC today and all hail related products (MESH,MESH-30,MESH-60,POSH,…) between ~9:10UTC and ~10:16UTC today. Missing over LDM as well as the HTTP server.

V/R,

Stefan
Stefan Cecelski, Ph.D.
Weather Analytics
Director of Scientific Modeling
www.weatheranalytics.com<http://www.weatheranalytics.com/>

On Aug 4, 2015, at 6:47 AM, Holly Uhlenhake - NOAA Affiliate <holly.uhlenhake at noaa.gov<mailto:holly.uhlenhake at noaa.gov>> wrote:

Ben, Daryl,

We have alerts when any product hasn't been updated in 1.5x the frequency we expect.  So for a 2 minute product we expect to see a delay of no more than 3 minutes.  So yes we have flags for all of this, but depending on the issue there might not be much we can do simply because of the high expiration of the products we are producing.  If we get a CPU flag on one of our radar machines and the data propagates slowly through the system or it's late there is little to nothing we can do except try and find out what caused the CPU issue and add resources accordingly if that is the problem.

It appears the problems were/are related to something in VMware.  I think our hardware team is still working with them on a solution, but I have not gotten all the details yet since I was out yesterday.   I'm afraid you will probably continue to see this until it is resolved.  Also, when we have problems like this the SDM calls us and we try to fix it, but this was not application specific so any attempts that were made to correct the system was simply not working due to the VMware problems.

Thanks,
Holly

On Mon, Aug 3, 2015 at 9:45 PM, Benjamin Schwedler <benjamin.schwedler at noaa.gov<mailto:benjamin.schwedler at noaa.gov>> wrote:
My basic monitoring based on file age also indicates that we have been missing multiple files today, particularly precipitation accumulations.  Things were semi-stable by about 15Z, but it appears that there may still be lingering issues.

Ben

On 08/03/2015 08:18 PM, daryl herzmann wrote:
Hello,

I have noticed a number of files missing in the LDM feed today (Monday)
and even now (this evening, 4 August UTC).  For example these two files
are missing

PrecipRate_00.00_20150804-003800.grib2.gz
PrecipRate_00.00_20150804-004000.grib2.gz

I checked this http://140.172.25.182/OPERATIONAL_NCEP_MRMS_GRIB2 and they
are missing there as well.  Do you folks not have alerting of when this
happens?

daryl

On Mon, 3 Aug 2015, Jaysankar Iyer - NOAA Affiliate wrote:

Hi Daryl,

We had numerous hardware and system related issues (hopefully all resolved)
over the weekend that led to IDP MRMS systems degrade in performance.

Let us know if you continue to see issues.

Thanks much

-Jay Iyer (MRMS Support Team)

On Mon, Aug 3, 2015 at 4:51 PM, daryl herzmann <akrherz at iastate.edu<mailto:akrherz at iastate.edu>> wrote:

Hello,

Any updates on this issue?  I have noticed that problems persisted into
this morning (Monday).

daryl

On Sun, 2 Aug 2015, Holly Uhlenhake - NOAA Affiliate wrote:

It looks like it's hardware related right now.  Maybe Derek will get more
info soon :-\

On Sun, Aug 2, 2015 at 10:07 AM, Karen Cooper <karen.cooper at noaa.gov<mailto:karen.cooper at noaa.gov>>
wrote:
Any idea what's causing the delays?

Sent from my iPhone

On Aug 2, 2015, at 8:46 AM, Derek VanPelt - NOAA Affiliate <
derek.vanpelt at noaa.gov<mailto:derek.vanpelt at noaa.gov>> wrote:
The IDP MRMS systems have been suffering from slowdowns in processing
this morning since ~10Z.  Some MRMS products may be missing or delayed
until we can get a fix in place.
We are working the issue, and will send out an update when we have one.

Thank you,

Derek Van Pelt
NCEP/NCO/IDP
_______________________________________________
Ncep.list.idp.mrms.users mailing list
Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov<mailto:Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov>

https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
_______________________________________________
Ncep.list.idp.mrms.users mailing list
Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov<mailto:Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov>

https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
--
/**
   * Daryl Herzmann
   * Assistant Scientist -- Iowa Environmental Mesonet
   * http://mesonet.agron.iastate.edu<http://mesonet.agron.iastate.edu/>
   */
_______________________________________________
Ncep.list.idp.mrms.users mailing list
Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov<mailto:Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov>
https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users

--
Benjamin R. J. Schwedler
NWS Aviation Weather Testbed - NextGen/AWT Development Meteorologist
CIRA/Colorado State University
NOAA/Aviation Weather Center
7220 NW 101st Terrace
Kansas City, MO 64153-2371
816.584.7268<tel:816.584.7268>

_______________________________________________
Ncep.list.idp.mrms.users mailing list
Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov<mailto:Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov>
https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.idp.mrms.users/attachments/20150805/18344170/attachment.html 