<div dir="ltr"><div><div><div><div>MRMS customers,<br><br></div>The MRMS application endured latencies, and produced some incomplete products over ~3 hours starting this morning @1530Z.<br><br></div>We were able to identify the issue quickly, but it took longer than expected to enact a full fix.<br><br></div>Apologies. I believe we have learned from this to the point that we should be able to avoid future issues. We will continue enhanced monitoring throughout the afternoon. <br><br></div>Derek<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 10, 2017 at 2:51 PM, Karen Cooper - NOAA Affiliate <span dir="ltr"><<a href="mailto:karen.cooper@noaa.gov" target="_blank">karen.cooper@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div>Hi Steve,<br><br></div><div>I'm including the NCEP IDP support group here, as this involves them. <br></div><br></div><div>It does appear that from about 15:46 to 18:30 that a lot of the products that are created by stitching the tiles were missing data. This is from looking at the College Park system (<a href="http://vm-lnx-mrms-convert1a.ncep.noaa.gov" target="_blank">vm-lnx-mrms-convert1a.ncep.<wbr>noaa.gov</a> specfically). <br><br></div><div>I'm not sure what would have caused this, but it also looks like there delays in the processing of some data. Our processing has timing controls to keep data for a short time -- typically twice the frequency of the product, but it will disregard data that is too old so that we mitigate the risk of including old data, as it could be misleading to analysts. <br><br></div><div>I'm going to defer to the NCO team as to what may have caused the slowdown during that time. <br><br></div><div>There are multiple CREFs some being done as 2D products which are not tiled, and some done from 3D tiled data, so depending on which one you were looking at there might not have been tiles missing. <br></div><div><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 10, 2017 at 11:48 AM, Steven Lack - NOAA Federal <span dir="ltr"><<a href="mailto:steven.a.lack@noaa.gov" target="_blank">steven.a.lack@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Karen,<div><br></div><div>It looks like starting after 1530 or so we started dropping tiles and we have had intermittent data coming in since then. This has been a problem with at least RALA and ET-18, somehow CREF looks okay. This is both on our AWIPS2 feed and our NIDS feed for NAWIPS.</div><div><br></div><div>Thanks for any info,</div><div><br></div><div>Steve<span class="HOEnZb"><font color="#888888"><span class="m_-5978782087035497304m_6148797766624821168HOEnZb"><font color="#888888"><br><div><div><br></div>-- <br><div class="m_-5978782087035497304m_6148797766624821168m_5945406950517494106gmail_signature" data-smartmail="gmail_signature"><div dir="ltr">Steven A. Lack, Ph.D.<br>Techniques Development Meteorologist<br>Aviation Weather Center<br>7220 NW 101st Terrace<br>Kansas City, MO 64153<br><br>office: <a href="tel:(816)%20584-7281" value="+18165847281" target="_blank">816-584-7281</a><br>cell: <a href="tel:(303)%20882-9395" value="+13038829395" target="_blank">303-882-9395</a><br><br>The views or opinions expressed in this email do not necessarily reflect those of NOAA or the United States Government. <br></div></div>
</div></font></span></font></span></div></div><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></div><span class="HOEnZb"><font color="#888888"><br><br clear="all"><br>-- <br><div class="m_-5978782087035497304m_6148797766624821168gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr">“The presence of those seeking the truth is infinitely to be preferred to the presence of those who think they’ve found it."<br></div><div> -- Terry Prachett<br></div><div dir="ltr"><br>------------------------------<wbr>-------------<br><a href="mailto:Karen.Cooper@noaa.gov" target="_blank">Karen.Cooper@noaa.gov</a><br><br><span style="color:rgb(0,0,0)">Phone#: <a href="tel:(405)%20325-6456" value="+14053256456" target="_blank">405-325-6456</a></span><br>Cell: <a href="tel:(405)%20834-8559" value="+14058348559" target="_blank">405-834-8559</a><br>National Severe Storms Laboratory<br><br></div></div></div></div></div></div></div></div></div></div>
</font></span></div></div>
<br>______________________________<wbr>_________________<br>
Ncep.list.idp_support mailing list<br>
<a href="mailto:Ncep.list.idp_support@lstsrv.ncep.noaa.gov">Ncep.list.idp_support@lstsrv.<wbr>ncep.noaa.gov</a><br>
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp_support" rel="noreferrer" target="_blank">https://www.lstsrv.ncep.noaa.<wbr>gov/mailman/listinfo/ncep.<wbr>list.idp_support</a><br>
<br></blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div>Derek Van Pelt<br></div>IDP Onboarding<br></div>NOAA/NCEP/NCO<br></div></div>
</div>