[Ncep.list.idp.mrms.users] OPEN - TID: 809080283 - IDP - Paducah radar data not being used in MRMS (since at least12z)?

Wed Sep 12 12:16:47 UTC 2018

The ESA (added to this thread) confirms repairs are completed and the radar
is now operating normally.  It can be returned to MRMS at your discretion.

Ken Ludington
Information Technology Officer (ITO)
National Weather Service/NOAA
Paducah, KY

Office: 270.744.6440
E-Mail: Ken.Ludington at noaa.gov

On Wed, Sep 12, 2018 at 5:06 AM NCO <toc.nwstg at noaa.gov> wrote:

> Ticket Properties
> Ticket Number: 809080283
> Opened: 2018-09-08 19:54:10
> Status: OPEN
> Category: IDP
>
> Description:  Paducah radar data not being used in MRMS (since at
> least12z)?
>
> 2018-09-12 10:03:19 GMT ATTN: Jay Iyer - please advise:
> 1) verification that yesterday's removal of KPAH from MRMS processing
> 2) if the removal of KPAH from MRMS is the end state or just a temporary
> fix (CJJ)
>
>
> ===================
> 2018-09-10 17:51:54 GMT 1616Z - Received email notification from Jay Iyer:
> Completed - we will continue to monitor.
>
> Thanks much
>
> -Jay
>
> (SEC)
>
>
> ===================
> 2018-09-10 16:09:24 GMT Distro updated. //JOHN//
>
>
> ===================
> 2018-09-10 16:09:24 GMT At 1600z, received e-mail from IDP System Eng, Jay
> Lyer:
> DP MRMS: ARFC - Remove KPAH from MRMS processing
> Starting now.. //JOHN//
>
>
> ===================
> 2018-09-09 15:34:13 GMT [NCO-MCL] @(1524)z (jaysankar.iyer at noaa.gov)
> reported the following via email:
> SDM,
>
> Based on our phone conversation, I have removed KPAH from MRMS processing
> on the active MRMS site in CPRK. Processing has resumed and things should
> improve soon.
>
>
>
> ===================
> 2018-09-09 15:16:53 GMT Distro list updated. //JOHN//
>
>
> ===================
> 2018-09-09 15:16:52 GMT At 1456z, National Severe Storms Laboratory
> Meteorologist, Kenneth H, e-mailed:
> From the NSSL perspective the best option to handle the situation would be
> to remove KPAH from MRMS processes, and let other neighboring radars
> (KLSX,KVWX, KHPX, and KNQA) fill in. Yes, there would be some slight
> degradation of products in the KPAH area but this may be significantly
> better then the current 'hole',
>
> For MRMS v12, we'll have an automated ARJ failure identification scheme
> and a corresponding QC mitigation to avoid the issue being currently
> experienced.
>
> Our apology this is currently occurring in the MRMS operational system.
> //JOHN//
>
>
> ===================
> 2018-09-09 14:26:58 GMT [NCO-MCL] @(1423)z (jaysankar.iyer at noaa.gov)
> reported the following via email:
> Hello,
>
> The config file edit we proposed would remove all inputs to MRMS from
> KPAH. MRMS would produce products with input from adjacent radar sites. We
> are not certain of the quality of products that are produced with inputs
> from nearby stations.
>
>
>
> ===================
> 2018-09-09 14:22:35 GMT [NCO-MCL] @(1405)z (mark.fenbers at noaa.gov)
> reported the following via email:
> Randy, good info! Thank you! But something really puzzles me about what
> was said. Namely, that editing a config file for the MRMS processing would
> prevent a WFO from getting data from their local radar?? I admittedly don't
> know how the whole MRMS process works, but the notion of a simple config
> file change at headquarters having a side effect of stopping a WFO from
> getting their own local radar products seems alarming to me -- like
> something is very amiss with how the MRMS process is designed, and very
> much a security concern. Can you verify that this is accurate and that
> there isn't any other way to have MRMS fill-in data without such grave
> side-effects?
>
>
>
>
> ===================
> 2018-09-09 13:42:43 GMT Updated distro listing by adding WFO-Paducah, ITO,
> ESA, MIC. //JOHN//
>
>
> ===================
> 2018-09-09 13:42:42 GMT At 1329z, SDM-Randy, e-mailed:
> To all,
>
> A summary of what IDP Onboarding has provided concerning this urgent
> ticket.
>
> This is a recap of this morning's phone conversations. Initially we
> thought this was an MRMS ingest issue so we restarted processing on KPAH
> radar (vm-lnx-mrms-sr17a). This is before we got the email saying it is a
> hardware failure at the site. Since some of the data is coming in for the
> KPAH radar, MRMS does not know it has to fill in from the other radars. In
> order for them to fill in we need to remove KPAH from MRMS completely. This
> requires a config file change and a restart of MRMS processing on that VM.
> The problem with this is that it would cause WFO Paducah to lose the local
> radars products for KPAH which are still coming in. Based on this I
> understand you do not want us to make the change and we will stand down.
> //JOHN//
>
>
> ===================
> 2018-09-09 11:52:28 GMT At 1146z, Mark F, e-mailed further clarification:
> But even with the PAH down for maintenance UFN, this shouldn't leave a
> hole in the product. Radars go down throughout the country fairly often,
> and when this happens, MRMS data is filled in from surrounding radars. But
> with the PAH outage, this fill-in from other radars is not happening like
> it usually does, and THIS is where the problem exists -- where the MRMS
> product is created. I hope this clarifies things. //JOHN//
>
>
> ===================
> 2018-09-09 11:37:31 GMT [NCO-MCL] NCO called the Lead Forecaster at PAH
> and he reported that the issue is with the Radar itself. Since there is
> weather in the area, technician would like to wait until weather cleared
> before working on the radar further.
>
>
> ===================
> 2018-09-09 11:32:06 GMT At 1120z, On-Call IDP Sys Eng, Vanessa called with
> update that she is researching to see what options may exist, if any, at
> the systems level, to enhance MRMS output. At this time, it remains
> uncertain. Vanessa is also going to consult with Senior IDP Eng for
> additional input. //JOHN//
>
>
> ===================
> 2018-09-09 11:01:42 GMT At 1050z, NCO-Tech Control consulted with SDM, and
> then called IDP oOn-Call System Eng, Vanessa M. Informed Vanessa this
> ticket has been upgraded to URGENT, based on input from Sr Meteorologist
> from Wilmington OH, RFC, and we are interested in determining what type of
> action, if any, can be performed on MRMS system, to allow whole to be
> filled by surrounding Radar's, short of Radar repair outside of RADAR
> repair, at KPAH. Vanessa will take a look to determine what actions can be
> performed and will report back. //JOHN//
>
>
> ===================
> 2018-09-09 10:44:08 GMT At 1028z, received e-mail update from RFC Senior
> Meteorologist, Mark, F:
> Well, I don't know who Scott talked to, but it IS critical to OHRFC!!
> We've got the biggest event we've had in a very long time and this is
> affecting one of our most important products that multiple agencies depend
> on. How can this not be considered critical??! //JOHN//
>
>
> ===================
> 2018-09-08 20:54:42 GMT 2021Z - Received email notification from the SDM:
> NCF called and also reported the same to the SDM. There are no MRMS issues
> in BB. Sent an E-mail to IDP on boarding support. 2007Z - Talked to Scott
> at NCF and he reported that after receiving feedback from several RFC sites
> it was deemed non-critical and could wait until Monday.
> (SEC)
>
>
> ===================
> 2018-09-08 20:02:46 GMT 1857Z - Received email notification from Ray
> Davis: i there,
> I bet that someone has already reported this, but just in case... The KPAH
> dual-pole radar products seem to be missing, and MRMS seems to be
> preventing the surrounding radars to fill in the void. The problem appears
> to have started sometime after 12z this morning. Could you check into this?
>
> The KPAH hole didn't appear after we processed our grids up through 12z,
> but it appears that MRMS is now backfilling previously saved grids with bad
> data. The Following FTM was issued from the WFO in reference to this site:
> KPAH 062106 FTMPAH Message Date: Sep 06 2018 21:10:57 THE KPAH RADAR IS
> OPERATIONAL WITH DEGRADED DUAL POL PRODUCTS. DUAL POL PRODUCTS SHOULD BE
> USED WITH CAUTION. REPAIRS WILL BE MADE TO THE RADAR EARLY NEXT WEEK.
>
> (SEC)
>
>
> ===================
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.idp.mrms.users/attachments/20180912/74a8a992/attachment-0001.html 