<div dir="ltr"><div><div><div>Hi Stefan,<br><br></div>We are hoping to get an update at our operations meeting this morning (13Z) and then we'll update our users. Expect to hear from one of us later this morning.<br><br></div>Thanks,<br></div>Holly<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 5, 2015 at 8:44 AM, Cecelski, Stefan <span dir="ltr"><<a href="mailto:Stefan.Cecelski@weatheranalytics.com" target="_blank">Stefan.Cecelski@weatheranalytics.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
All,
<div><br>
</div>
<div>FYI - Still encountering some missing files, specifically the 1-HR Gauge-Corrected QPE @ 7,8,10 UTC today and all hail related products (MESH,MESH-30,MESH-60,POSH,…) between ~9:10UTC and ~10:16UTC today. Missing over LDM as well as the HTTP server.</div>
<div><br>
</div>
<div>V/R,</div>
<div><br>
</div>
<div>Stefan<br>
<div>
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="color:rgb(0,0,0);letter-spacing:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;word-wrap:break-word">
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif">
<b><span style="font-size:12pt;font-family:Tahoma,sans-serif;background-color:white">Stefan Cecelski, Ph.D.<u></u><u></u></span></b></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif;background-color:white">
<b><span style="font-size:14pt;font-family:Tahoma,sans-serif;color:rgb(0,176,240)">Weather Analytics<u></u><u></u></span></b></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif;background-color:white">
<span style="font-size:12pt;font-family:Tahoma,sans-serif;color:rgb(59,56,56)">Director of Scientific Modeling<u></u><u></u></span></div>
<div style="margin:0in 0in 0.0001pt;font-size:11pt;font-family:Calibri,sans-serif;background-color:white">
<span style="font-size:12pt;font-family:Tahoma,sans-serif"><span style="color:rgb(51,102,255)"><a href="http://www.weatheranalytics.com/" style="color:rgb(149,79,114)" target="_blank">www.weatheranalytics.com</a></span></span></div>
</div>
</div>
</div><div><div class="h5">
<br>
<div>
<blockquote type="cite">
<div>On Aug 4, 2015, at 6:47 AM, Holly Uhlenhake - NOAA Affiliate <<a href="mailto:holly.uhlenhake@noaa.gov" target="_blank">holly.uhlenhake@noaa.gov</a>> wrote:</div>
<br>
<div>
<div dir="ltr">Ben, Daryl,<br>
<div><br>
We have alerts when any product hasn't been updated in 1.5x the frequency we expect. So for a 2 minute product we expect to see a delay of no more than 3 minutes. So yes we have flags for all of this, but depending on the issue there might not be much we
can do simply because of the high expiration of the products we are producing. If we get a CPU flag on one of our radar machines and the data propagates slowly through the system or it's late there is little to nothing we can do except try and find out what
caused the CPU issue and add resources accordingly if that is the problem.<br>
<br>
</div>
<div>It appears the problems were/are related to something in VMware. I think our hardware team is still working with them on a solution, but I have not gotten all the details yet since I was out yesterday. I'm afraid you will probably continue
to see this until it is resolved. Also, when we have problems like this the SDM calls us and we try to fix it, but this was not application specific so any attempts that were made to correct the system was simply not working due to the VMware problems.
<br>
<br>
</div>
<div>Thanks,<br>
</div>
<div>Holly<br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Aug 3, 2015 at 9:45 PM, Benjamin Schwedler <span dir="ltr">
<<a href="mailto:benjamin.schwedler@noaa.gov" target="_blank">benjamin.schwedler@noaa.gov</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
My basic monitoring based on file age also indicates that we have been missing multiple files today, particularly precipitation accumulations. Things were semi-stable by about 15Z, but it appears that there may still be lingering issues.<br>
<br>
Ben
<div>
<div><br>
<br>
On 08/03/2015 08:18 PM, daryl herzmann wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
I have noticed a number of files missing in the LDM feed today (Monday)<br>
and even now (this evening, 4 August UTC). For example these two files<br>
are missing<br>
<br>
PrecipRate_00.00_20150804-003800.grib2.gz<br>
PrecipRate_00.00_20150804-004000.grib2.gz<br>
<br>
I checked this <a href="http://140.172.25.182/OPERATIONAL_NCEP_MRMS_GRIB2" rel="noreferrer" target="_blank">
http://140.172.25.182/OPERATIONAL_NCEP_MRMS_GRIB2</a> and they<br>
are missing there as well. Do you folks not have alerting of when this<br>
happens?<br>
<br>
daryl<br>
<br>
On Mon, 3 Aug 2015, Jaysankar Iyer - NOAA Affiliate wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Daryl,<br>
<br>
We had numerous hardware and system related issues (hopefully all resolved)<br>
over the weekend that led to IDP MRMS systems degrade in performance.<br>
<br>
Let us know if you continue to see issues.<br>
<br>
Thanks much<br>
<br>
-Jay Iyer (MRMS Support Team)<br>
<br>
On Mon, Aug 3, 2015 at 4:51 PM, daryl herzmann <<a href="mailto:akrherz@iastate.edu" target="_blank">akrherz@iastate.edu</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
<br>
Any updates on this issue? I have noticed that problems persisted into<br>
this morning (Monday).<br>
<br>
daryl<br>
<br>
On Sun, 2 Aug 2015, Holly Uhlenhake - NOAA Affiliate wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It looks like it's hardware related right now. Maybe Derek will get more<br>
info soon :-\<br>
<br>
On Sun, Aug 2, 2015 at 10:07 AM, Karen Cooper <<a href="mailto:karen.cooper@noaa.gov" target="_blank">karen.cooper@noaa.gov</a>><br>
</blockquote>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Any idea what's causing the delays?<br>
<br>
Sent from my iPhone<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Aug 2, 2015, at 8:46 AM, Derek VanPelt - NOAA Affiliate <<br>
</blockquote>
<a href="mailto:derek.vanpelt@noaa.gov" target="_blank">derek.vanpelt@noaa.gov</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
The IDP MRMS systems have been suffering from slowdowns in processing<br>
</blockquote>
this morning since ~10Z. Some MRMS products may be missing or delayed<br>
until we can get a fix in place.<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
We are working the issue, and will send out an update when we have one.<br>
<br>
Thank you,<br>
<br>
Derek Van Pelt<br>
NCEP/NCO/IDP<br>
_______________________________________________<br>
Ncep.list.idp.mrms.users mailing list<br>
<a href="mailto:Ncep.list.idp.mrms.users@lstsrv.ncep.noaa.gov" target="_blank">Ncep.list.idp.mrms.users@lstsrv.ncep.noaa.gov</a><br>
<br>
</blockquote>
</blockquote>
</blockquote>
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users" rel="noreferrer" target="_blank">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users</a><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
_______________________________________________<br>
Ncep.list.idp.mrms.users mailing list<br>
<a href="mailto:Ncep.list.idp.mrms.users@lstsrv.ncep.noaa.gov" target="_blank">Ncep.list.idp.mrms.users@lstsrv.ncep.noaa.gov</a><br>
<br>
</blockquote>
</blockquote>
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users" rel="noreferrer" target="_blank">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users</a><br>
--<br>
/**<br>
* Daryl Herzmann<br>
* Assistant Scientist -- Iowa Environmental Mesonet<br>
* <a href="http://mesonet.agron.iastate.edu/" rel="noreferrer" target="_blank">
http://mesonet.agron.iastate.edu</a><br>
*/<br>
_______________________________________________<br>
Ncep.list.idp.mrms.users mailing list<br>
<a href="mailto:Ncep.list.idp.mrms.users@lstsrv.ncep.noaa.gov" target="_blank">Ncep.list.idp.mrms.users@lstsrv.ncep.noaa.gov</a><br>
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users" rel="noreferrer" target="_blank">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users</a><br>
<br>
</blockquote>
</blockquote>
</blockquote>
<br>
-- <br>
</div>
</div>
Benjamin R. J. Schwedler<br>
NWS Aviation Weather Testbed - NextGen/AWT Development Meteorologist<br>
CIRA/Colorado State University<br>
NOAA/Aviation Weather Center<br>
7220 NW 101st Terrace<br>
Kansas City, MO 64153-2371<br>
<a href="tel:816.584.7268" value="+18165847268" target="_blank">816.584.7268</a><br>
<br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
Ncep.list.idp.mrms.users mailing list<br>
<a href="mailto:Ncep.list.idp.mrms.users@lstsrv.ncep.noaa.gov" target="_blank">Ncep.list.idp.mrms.users@lstsrv.ncep.noaa.gov</a><br>
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users" target="_blank">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users</a><br>
</div>
</blockquote>
</div>
<br>
</div></div></div>
</div>
</blockquote></div><br></div>