<div dir="ltr"><div>Curt,</div><div><br></div><div>Jay added PAH back into MRMS last Monday (September 17th, 2018).</div><div><br></div><div>Justin<br></div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Sep 27, 2018 at 8:37 AM NCO <<a href="mailto:toc.nwstg@noaa.gov">toc.nwstg@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Ticket Properties<br>
Ticket Number: 809080283<br>
Opened: 2018-09-08 19:54:10<br>
Status: OPEN<br>
Category: IDP<br>
<br>
Description: Paducah radar data not being used in MRMS (since at least12z)? <br>
<br>
2018-09-27 12:34:49 GMT @ 1234Z OMB sent a group Google Chat to several members of the IDP on-boarding team requesting a status on adding PUH back into MRMS. (CJJ)<br>
<br>
<br>
<br>
===================<br>
2018-09-12 16:36:21 GMT <a href="mailto:jonathan.case-1@nasa.gov" target="_blank">jonathan.case-1@nasa.gov</a> sent out the following email: "Hello All,<br>
<br>
Here at SPoRT, we run an experimental land surface model product over CONUS at ~3km resolution. We utilize the MRMS hourly gauge-corrected radar QPE product in the 3 most recent days for driving the soil model. We saw the same type of polygon blockage pattern in our web graphics from Saturday 8 September, as the remnants of tropical storm Gordon traversed western Kentucky/southern Illinois. The degraded MRMS product clearly affected out output quality as well. (see an animation of hourly near-surface soil moisture with MRMS contours overlaid from 8 September at <a href="https://weather.msfc.nasa.gov/cgi-bin/basicLooper.pl?category=lis_CONUS&initialize=first&regex=rsoim0-10_20180908*.gif" rel="noreferrer" target="_blank">https://weather.msfc.nasa.gov/cgi-bin/basicLooper.pl?category=lis_CONUS&initialize=first&regex=rsoim0-10_20180908*.gif</a>). <br>
<br>
Sincerely,<br>
<br>
Jonathan Case @ NASA/SPoRT Center" (ZT) <br>
<br>
<br>
===================<br>
2018-09-12 12:25:35 GMT @ 1222Z On Boarding Team (Jay Iyer) sent an email stating...<br>
we are moving to [putting KPAH back into MRMS] this morning, if we get the approvals. We got mail confirmation that the site is now fully fixed. (CJJ) <br>
<br>
<br>
===================<br>
2018-09-12 12:20:47 GMT @ 1217Z On Boarding Team (Jay Iyer) sent an email stating...<br>
This was an ARFC that we implemented over the weekend to remove KPAH from MRMS processing to address a hardware failure at KPAH. We will put this back in MRMS once we receive notification that the issues at the site has been resolved. (CJJ) <br>
<br>
<br>
===================<br>
2018-09-12 12:20:47 GMT @ 1216Z Ken Ludington sent an email stating...<br>
The ESA (added to this thread) confirms repairs are completed and the radar is now operating normally. It can be returned to MRMS at your discretion. (CJJ) <br>
<br>
<br>
===================<br>
2018-09-12 10:03:19 GMT ATTN: Jay Iyer - please advise:<br>
1) verification that yesterday's removal of KPAH from MRMS processing<br>
2) if the removal of KPAH from MRMS is the end state or just a temporary fix (CJJ) <br>
<br>
<br>
===================<br>
2018-09-10 17:51:54 GMT 1616Z - Received email notification from Jay Iyer: Completed - we will continue to monitor.<br>
<br>
Thanks much<br>
<br>
-Jay<br>
<br>
(SEC) <br>
<br>
<br>
===================<br>
2018-09-10 16:09:24 GMT Distro updated. //JOHN// <br>
<br>
<br>
===================<br>
2018-09-10 16:09:24 GMT At 1600z, received e-mail from IDP System Eng, Jay Lyer:<br>
DP MRMS: ARFC - Remove KPAH from MRMS processing<br>
Starting now.. //JOHN// <br>
<br>
<br>
===================<br>
2018-09-09 15:34:13 GMT [NCO-MCL] @(1524)z (<a href="mailto:jaysankar.iyer@noaa.gov" target="_blank">jaysankar.iyer@noaa.gov</a>) reported the following via email:<br>
SDM,<br>
<br>
Based on our phone conversation, I have removed KPAH from MRMS processing on the active MRMS site in CPRK. Processing has resumed and things should improve soon.<br>
<br>
<br>
<br>
===================<br>
2018-09-09 15:16:53 GMT Distro list updated. //JOHN// <br>
<br>
<br>
===================<br>
2018-09-09 15:16:52 GMT At 1456z, National Severe Storms Laboratory Meteorologist, Kenneth H, e-mailed:<br>
>From the NSSL perspective the best option to handle the situation would be to remove KPAH from MRMS processes, and let other neighboring radars (KLSX,KVWX, KHPX, and KNQA) fill in. Yes, there would be some slight degradation of products in the KPAH area but this may be significantly better then the current 'hole', <br>
<br>
For MRMS v12, we'll have an automated ARJ failure identification scheme and a corresponding QC mitigation to avoid the issue being currently experienced. <br>
<br>
Our apology this is currently occurring in the MRMS operational system. //JOHN// <br>
<br>
<br>
===================<br>
2018-09-09 14:26:58 GMT [NCO-MCL] @(1423)z (<a href="mailto:jaysankar.iyer@noaa.gov" target="_blank">jaysankar.iyer@noaa.gov</a>) reported the following via email:<br>
Hello,<br>
<br>
The config file edit we proposed would remove all inputs to MRMS from KPAH. MRMS would produce products with input from adjacent radar sites. We are not certain of the quality of products that are produced with inputs from nearby stations.<br>
<br>
<br>
<br>
===================<br>
2018-09-09 14:22:35 GMT [NCO-MCL] @(1405)z (<a href="mailto:mark.fenbers@noaa.gov" target="_blank">mark.fenbers@noaa.gov</a>) reported the following via email:<br>
Randy, good info! Thank you! But something really puzzles me about what was said. Namely, that editing a config file for the MRMS processing would prevent a WFO from getting data from their local radar?? I admittedly don't know how the whole MRMS process works, but the notion of a simple config file change at headquarters having a side effect of stopping a WFO from getting their own local radar products seems alarming to me -- like something is very amiss with how the MRMS process is designed, and very much a security concern. Can you verify that this is accurate and that there isn't any other way to have MRMS fill-in data without such grave side-effects?<br>
<br>
<br>
<br>
<br>
===================<br>
2018-09-09 13:42:43 GMT Updated distro listing by adding WFO-Paducah, ITO, ESA, MIC. //JOHN// <br>
<br>
<br>
===================<br>
2018-09-09 13:42:42 GMT At 1329z, SDM-Randy, e-mailed:<br>
To all,<br>
<br>
A summary of what IDP Onboarding has provided concerning this urgent ticket.<br>
<br>
This is a recap of this morning's phone conversations. Initially we thought this was an MRMS ingest issue so we restarted processing on KPAH radar (vm-lnx-mrms-sr17a). This is before we got the email saying it is a hardware failure at the site. Since some of the data is coming in for the KPAH radar, MRMS does not know it has to fill in from the other radars. In order for them to fill in we need to remove KPAH from MRMS completely. This requires a config file change and a restart of MRMS processing on that VM. The problem with this is that it would cause WFO Paducah to lose the local radars products for KPAH which are still coming in. Based on this I understand you do not want us to make the change and we will stand down. //JOHN// <br>
<br>
<br>
===================<br>
2018-09-09 11:52:28 GMT At 1146z, Mark F, e-mailed further clarification:<br>
But even with the PAH down for maintenance UFN, this shouldn't leave a hole in the product. Radars go down throughout the country fairly often, and when this happens, MRMS data is filled in from surrounding radars. But with the PAH outage, this fill-in from other radars is not happening like it usually does, and THIS is where the problem exists -- where the MRMS product is created. I hope this clarifies things. //JOHN// <br>
<br>
<br>
===================<br>
2018-09-09 11:37:31 GMT [NCO-MCL] NCO called the Lead Forecaster at PAH and he reported that the issue is with the Radar itself. Since there is weather in the area, technician would like to wait until weather cleared before working on the radar further. <br>
<br>
<br>
===================<br>
2018-09-09 11:32:06 GMT At 1120z, On-Call IDP Sys Eng, Vanessa called with update that she is researching to see what options may exist, if any, at the systems level, to enhance MRMS output. At this time, it remains uncertain. Vanessa is also going to consult with Senior IDP Eng for additional input. //JOHN// <br>
<br>
<br>
===================<br>
2018-09-09 11:01:42 GMT At 1050z, NCO-Tech Control consulted with SDM, and then called IDP oOn-Call System Eng, Vanessa M. Informed Vanessa this ticket has been upgraded to URGENT, based on input from Sr Meteorologist from Wilmington OH, RFC, and we are interested in determining what type of action, if any, can be performed on MRMS system, to allow whole to be filled by surrounding Radar's, short of Radar repair outside of RADAR repair, at KPAH. Vanessa will take a look to determine what actions can be performed and will report back. //JOHN// <br>
<br>
<br>
===================<br>
2018-09-09 10:44:08 GMT At 1028z, received e-mail update from RFC Senior Meteorologist, Mark, F:<br>
Well, I don't know who Scott talked to, but it IS critical to OHRFC!! We've got the biggest event we've had in a very long time and this is affecting one of our most important products that multiple agencies depend on. How can this not be considered critical??! //JOHN// <br>
<br>
<br>
===================<br>
2018-09-08 20:54:42 GMT 2021Z - Received email notification from the SDM: NCF called and also reported the same to the SDM. There are no MRMS issues in BB. Sent an E-mail to IDP on boarding support. 2007Z - Talked to Scott at NCF and he reported that after receiving feedback from several RFC sites it was deemed non-critical and could wait until Monday.<br>
(SEC) <br>
<br>
<br>
===================<br>
2018-09-08 20:02:46 GMT 1857Z - Received email notification from Ray Davis: i there,<br>
I bet that someone has already reported this, but just in case... The KPAH dual-pole radar products seem to be missing, and MRMS seems to be preventing the surrounding radars to fill in the void. The problem appears to have started sometime after 12z this morning. Could you check into this?<br>
<br>
The KPAH hole didn't appear after we processed our grids up through 12z, but it appears that MRMS is now backfilling previously saved grids with bad data. The Following FTM was issued from the WFO in reference to this site: KPAH 062106 FTMPAH Message Date: Sep 06 2018 21:10:57 THE KPAH RADAR IS OPERATIONAL WITH DEGRADED DUAL POL PRODUCTS. DUAL POL PRODUCTS SHOULD BE USED WITH CAUTION. REPAIRS WILL BE MADE TO THE RADAR EARLY NEXT WEEK.<br>
<br>
(SEC) <br>
<br>
<br>
===================<br>
_______________________________________________<br>
Ncep.list.idp_support mailing list<br>
<a href="mailto:Ncep.list.idp_support@lstsrv.ncep.noaa.gov" target="_blank">Ncep.list.idp_support@lstsrv.ncep.noaa.gov</a><br>
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp_support" rel="noreferrer" target="_blank">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp_support</a><br>
</blockquote></div>