[Ncep.list.idp.mrms.users] [Ncep.list.idp_support] OPEN - TID: 809080283 - IDP - Paducah radar data not being used in MRMS (since at least12z)?

Thu Sep 27 12:53:13 UTC 2018

Curt,

Jay added PAH back into MRMS last Monday (September 17th, 2018).

Justin

On Thu, Sep 27, 2018 at 8:37 AM NCO <toc.nwstg at noaa.gov> wrote:

> Ticket Properties
> Ticket Number: 809080283
> Opened: 2018-09-08 19:54:10
> Status: OPEN
> Category: IDP
>
> Description:  Paducah radar data not being used in MRMS (since at
> least12z)?
>
> 2018-09-27 12:34:49 GMT @ 1234Z OMB sent a group Google Chat to several
> members of the IDP on-boarding team requesting a status on adding PUH back
> into MRMS. (CJJ)
>
>
>
> ===================
> 2018-09-12 16:36:21 GMT jonathan.case-1 at nasa.gov sent out the following
> email: "Hello All,
>
> Here at SPoRT, we run an experimental land surface model product over
> CONUS at ~3km resolution. We utilize the MRMS hourly gauge-corrected radar
> QPE product in the 3 most recent days for driving the soil model. We saw
> the same type of polygon blockage pattern in our web graphics from Saturday
> 8 September, as the remnants of tropical storm Gordon traversed western
> Kentucky/southern Illinois. The degraded MRMS product clearly affected out
> output quality as well. (see an animation of hourly near-surface soil
> moisture with MRMS contours overlaid from 8 September at
> https://weather.msfc.nasa.gov/cgi-bin/basicLooper.pl?category=lis_CONUS&initialize=first&regex=rsoim0-10_20180908*.gif).
>
>
> Sincerely,
>
> Jonathan Case @ NASA/SPoRT Center" (ZT)
>
>
> ===================
> 2018-09-12 12:25:35 GMT @ 1222Z On Boarding Team (Jay Iyer) sent an email
> stating...
> we are moving to [putting KPAH back into MRMS] this morning, if we get the
> approvals. We got mail confirmation that the site is now fully fixed. (CJJ)
>
>
> ===================
> 2018-09-12 12:20:47 GMT @ 1217Z On Boarding Team (Jay Iyer) sent an email
> stating...
> This was an ARFC that we implemented over the weekend to remove KPAH from
> MRMS processing to address a hardware failure at KPAH. We will put this
> back in MRMS once we receive notification that the issues at the site has
> been resolved. (CJJ)
>
>
> ===================
> 2018-09-12 12:20:47 GMT @ 1216Z Ken Ludington sent an email stating...
> The ESA (added to this thread) confirms repairs are completed and the
> radar is now operating normally. It can be returned to MRMS at your
> discretion. (CJJ)
>
>
> ===================
> 2018-09-12 10:03:19 GMT ATTN: Jay Iyer - please advise:
> 1) verification that yesterday's removal of KPAH from MRMS processing
> 2) if the removal of KPAH from MRMS is the end state or just a temporary
> fix (CJJ)
>
>
> ===================
> 2018-09-10 17:51:54 GMT 1616Z - Received email notification from Jay Iyer:
> Completed - we will continue to monitor.
>
> Thanks much
>
> -Jay
>
> (SEC)
>
>
> ===================
> 2018-09-10 16:09:24 GMT Distro updated. //JOHN//
>
>
> ===================
> 2018-09-10 16:09:24 GMT At 1600z, received e-mail from IDP System Eng, Jay
> Lyer:
> DP MRMS: ARFC - Remove KPAH from MRMS processing
> Starting now.. //JOHN//
>
>
> ===================
> 2018-09-09 15:34:13 GMT [NCO-MCL] @(1524)z (jaysankar.iyer at noaa.gov)
> reported the following via email:
> SDM,
>
> Based on our phone conversation, I have removed KPAH from MRMS processing
> on the active MRMS site in CPRK. Processing has resumed and things should
> improve soon.
>
>
>
> ===================
> 2018-09-09 15:16:53 GMT Distro list updated. //JOHN//
>
>
> ===================
> 2018-09-09 15:16:52 GMT At 1456z, National Severe Storms Laboratory
> Meteorologist, Kenneth H, e-mailed:
> >From the NSSL perspective the best option to handle the situation would
> be to remove KPAH from MRMS processes, and let other neighboring radars
> (KLSX,KVWX, KHPX, and KNQA) fill in. Yes, there would be some slight
> degradation of products in the KPAH area but this may be significantly
> better then the current 'hole',
>
> For MRMS v12, we'll have an automated ARJ failure identification scheme
> and a corresponding QC mitigation to avoid the issue being currently
> experienced.
>
> Our apology this is currently occurring in the MRMS operational system.
> //JOHN//
>
>
> ===================
> 2018-09-09 14:26:58 GMT [NCO-MCL] @(1423)z (jaysankar.iyer at noaa.gov)
> reported the following via email:
> Hello,
>
> The config file edit we proposed would remove all inputs to MRMS from
> KPAH. MRMS would produce products with input from adjacent radar sites. We
> are not certain of the quality of products that are produced with inputs
> from nearby stations.
>
>
>
> ===================
> 2018-09-09 14:22:35 GMT [NCO-MCL] @(1405)z (mark.fenbers at noaa.gov)
> reported the following via email:
> Randy, good info! Thank you! But something really puzzles me about what
> was said. Namely, that editing a config file for the MRMS processing would
> prevent a WFO from getting data from their local radar?? I admittedly don't
> know how the whole MRMS process works, but the notion of a simple config
> file change at headquarters having a side effect of stopping a WFO from
> getting their own local radar products seems alarming to me -- like
> something is very amiss with how the MRMS process is designed, and very
> much a security concern. Can you verify that this is accurate and that
> there isn't any other way to have MRMS fill-in data without such grave
> side-effects?
>
>
>
>
> ===================
> 2018-09-09 13:42:43 GMT Updated distro listing by adding WFO-Paducah, ITO,
> ESA, MIC. //JOHN//
>
>
> ===================
> 2018-09-09 13:42:42 GMT At 1329z, SDM-Randy, e-mailed:
> To all,
>
> A summary of what IDP Onboarding has provided concerning this urgent
> ticket.
>
> This is a recap of this morning's phone conversations. Initially we
> thought this was an MRMS ingest issue so we restarted processing on KPAH
> radar (vm-lnx-mrms-sr17a). This is before we got the email saying it is a
> hardware failure at the site. Since some of the data is coming in for the
> KPAH radar, MRMS does not know it has to fill in from the other radars. In
> order for them to fill in we need to remove KPAH from MRMS completely. This
> requires a config file change and a restart of MRMS processing on that VM.
> The problem with this is that it would cause WFO Paducah to lose the local
> radars products for KPAH which are still coming in. Based on this I
> understand you do not want us to make the change and we will stand down.
> //JOHN//
>
>
> ===================
> 2018-09-09 11:52:28 GMT At 1146z, Mark F, e-mailed further clarification:
> But even with the PAH down for maintenance UFN, this shouldn't leave a
> hole in the product. Radars go down throughout the country fairly often,
> and when this happens, MRMS data is filled in from surrounding radars. But
> with the PAH outage, this fill-in from other radars is not happening like
> it usually does, and THIS is where the problem exists -- where the MRMS
> product is created. I hope this clarifies things. //JOHN//
>
>
> ===================
> 2018-09-09 11:37:31 GMT [NCO-MCL] NCO called the Lead Forecaster at PAH
> and he reported that the issue is with the Radar itself. Since there is
> weather in the area, technician would like to wait until weather cleared
> before working on the radar further.
>
>
> ===================
> 2018-09-09 11:32:06 GMT At 1120z, On-Call IDP Sys Eng, Vanessa called with
> update that she is researching to see what options may exist, if any, at
> the systems level, to enhance MRMS output. At this time, it remains
> uncertain. Vanessa is also going to consult with Senior IDP Eng for
> additional input. //JOHN//
>
>
> ===================
> 2018-09-09 11:01:42 GMT At 1050z, NCO-Tech Control consulted with SDM, and
> then called IDP oOn-Call System Eng, Vanessa M. Informed Vanessa this
> ticket has been upgraded to URGENT, based on input from Sr Meteorologist
> from Wilmington OH, RFC, and we are interested in determining what type of
> action, if any, can be performed on MRMS system, to allow whole to be
> filled by surrounding Radar's, short of Radar repair outside of RADAR
> repair, at KPAH. Vanessa will take a look to determine what actions can be
> performed and will report back. //JOHN//
>
>
> ===================
> 2018-09-09 10:44:08 GMT At 1028z, received e-mail update from RFC Senior
> Meteorologist, Mark, F:
> Well, I don't know who Scott talked to, but it IS critical to OHRFC!!
> We've got the biggest event we've had in a very long time and this is
> affecting one of our most important products that multiple agencies depend
> on. How can this not be considered critical??! //JOHN//
>
>
> ===================
> 2018-09-08 20:54:42 GMT 2021Z - Received email notification from the SDM:
> NCF called and also reported the same to the SDM. There are no MRMS issues
> in BB. Sent an E-mail to IDP on boarding support. 2007Z - Talked to Scott
> at NCF and he reported that after receiving feedback from several RFC sites
> it was deemed non-critical and could wait until Monday.
> (SEC)
>
>
> ===================
> 2018-09-08 20:02:46 GMT 1857Z - Received email notification from Ray
> Davis: i there,
> I bet that someone has already reported this, but just in case... The KPAH
> dual-pole radar products seem to be missing, and MRMS seems to be
> preventing the surrounding radars to fill in the void. The problem appears
> to have started sometime after 12z this morning. Could you check into this?
>
> The KPAH hole didn't appear after we processed our grids up through 12z,
> but it appears that MRMS is now backfilling previously saved grids with bad
> data. The Following FTM was issued from the WFO in reference to this site:
> KPAH 062106 FTMPAH Message Date: Sep 06 2018 21:10:57 THE KPAH RADAR IS
> OPERATIONAL WITH DEGRADED DUAL POL PRODUCTS. DUAL POL PRODUCTS SHOULD BE
> USED WITH CAUTION. REPAIRS WILL BE MADE TO THE RADAR EARLY NEXT WEEK.
>
> (SEC)
>
>
> ===================
> _______________________________________________
> Ncep.list.idp_support mailing list
> Ncep.list.idp_support at lstsrv.ncep.noaa.gov
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp_support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.idp.mrms.users/attachments/20180927/173f9409/attachment.html 