[Ncep.list.idp.mrms.users] [Ncep.list.idp_support] MRMS issue today

[Daryl Herzmann]

Hello,

I am noticing an uptick in missing files today.  Are known troubles
still afoot?  Here's a printout of some missing files in CPRK in the
past 24 hours for the products I care about :)

MISS cprk 2017-03-10T22:00Z RadarOnly_QPE_01H
MISS cprk 2017-03-11T07:00Z RadarOnly_QPE_01H
MISS cprk 2017-03-11T12:00Z RadarOnly_QPE_01H
MISS cprk 2017-03-11T13:00Z RadarOnly_QPE_01H
MISS cprk 2017-03-10T19:26Z PrecipFlag
MISS cprk 2017-03-10T22:00Z PrecipFlag
MISS cprk 2017-03-11T02:00Z PrecipFlag
MISS cprk 2017-03-11T04:10Z PrecipFlag
MISS cprk 2017-03-11T07:00Z PrecipFlag
MISS cprk 2017-03-11T09:54Z PrecipFlag
MISS cprk 2017-03-11T12:00Z PrecipFlag
MISS cprk 2017-03-11T13:00Z PrecipFlag
MISS cprk 2017-03-10T21:00Z SeamlessHSR
MISS cprk 2017-03-10T22:00Z SeamlessHSR
MISS cprk 2017-03-10T23:00Z SeamlessHSR
MISS cprk 2017-03-11T02:00Z SeamlessHSR
MISS cprk 2017-03-11T04:16Z SeamlessHSR
MISS cprk 2017-03-11T07:00Z SeamlessHSR
MISS cprk 2017-03-11T09:00Z SeamlessHSR
MISS cprk 2017-03-11T12:00Z SeamlessHSR
MISS cprk 2017-03-11T13:00Z SeamlessHSR
MISS cprk 2017-03-11T14:00Z SeamlessHSR
MISS cprk 2017-03-10T19:26Z PrecipRate
MISS cprk 2017-03-10T19:56Z PrecipRate
MISS cprk 2017-03-10T22:00Z PrecipRate
MISS cprk 2017-03-11T01:04Z PrecipRate
MISS cprk 2017-03-11T02:00Z PrecipRate
MISS cprk 2017-03-11T03:32Z PrecipRate
MISS cprk 2017-03-11T03:46Z PrecipRate
MISS cprk 2017-03-11T04:10Z PrecipRate
MISS cprk 2017-03-11T07:00Z PrecipRate
MISS cprk 2017-03-11T07:26Z PrecipRate
MISS cprk 2017-03-11T07:46Z PrecipRate
MISS cprk 2017-03-11T12:00Z PrecipRate
MISS cprk 2017-03-11T12:40Z PrecipRate
MISS cprk 2017-03-11T13:00Z PrecipRate
MISS cprk 2017-03-11T13:20Z PrecipRate
MISS cprk 2017-03-11T14:48Z PrecipRate


daryl

On Fri, Mar 10, 2017 at 1:58 PM, Derek VanPelt - NOAA Affiliate
<derek.vanpelt at noaa.gov> wrote:
> MRMS customers,
>
> The MRMS application endured latencies, and produced some incomplete
> products over ~3 hours starting this morning @1530Z.
>
> We were able to identify the issue quickly, but it took longer than expected
> to enact a full fix.
>
> Apologies.  I believe we have learned from this to the point that we should
> be able to avoid future issues.  We will continue enhanced monitoring
> throughout the afternoon.
>
> Derek
>
> On Fri, Mar 10, 2017 at 2:51 PM, Karen Cooper - NOAA Affiliate
> <karen.cooper at noaa.gov> wrote:
>>
>> Hi Steve,
>>
>> I'm including the NCEP IDP support group here, as this involves them.
>>
>> It does appear that from about 15:46 to 18:30 that a lot of the products
>> that are created by stitching the tiles were missing data.   This is from
>> looking at the College Park system (vm-lnx-mrms-convert1a.ncep.noaa.gov
>> specfically).
>>
>> I'm not sure what would have caused this, but it also looks like there
>> delays in the processing of some data.   Our processing has timing controls
>> to keep data for a short time -- typically twice the frequency of the
>> product, but it will disregard data that is too old so that we mitigate the
>> risk of including old data, as it could be misleading to analysts.
>>
>> I'm going to defer to the NCO team as to what may have caused the slowdown
>> during that time.
>>
>> There are multiple CREFs some being done as 2D products which are not
>> tiled, and some done from 3D tiled data, so depending on which one you were
>> looking at there might not have been tiles missing.
>>
>>
>> On Fri, Mar 10, 2017 at 11:48 AM, Steven Lack - NOAA Federal
>> <steven.a.lack at noaa.gov> wrote:
>>>
>>> Karen,
>>>
>>> It looks like starting after 1530 or so we started dropping tiles and we
>>> have had intermittent data coming in since then.  This has been a problem
>>> with at least RALA and ET-18, somehow CREF looks okay.  This is both on our
>>> AWIPS2 feed and our NIDS feed for NAWIPS.
>>>
>>> Thanks for any info,
>>>
>>> Steve
>>>
>>> --
>>> Steven A. Lack, Ph.D.
>>> Techniques Development Meteorologist
>>> Aviation Weather Center
>>> 7220 NW 101st Terrace
>>> Kansas City, MO 64153
>>>
>>> office: 816-584-7281
>>> cell:  303-882-9395
>>>
>>> The views or opinions expressed in this email do not necessarily reflect
>>> those of NOAA or the United States Government.
>>
>>
>>
>>
>> --
>> “The presence of those seeking the truth is infinitely to be preferred to
>> the presence of those who think they’ve found it."
>>  -- Terry Prachett
>>
>> -------------------------------------------
>> Karen.Cooper at noaa.gov
>>
>> Phone#:  405-325-6456
>> Cell:   405-834-8559
>> National Severe Storms Laboratory
>>
>>
>> _______________________________________________
>> Ncep.list.idp_support mailing list
>> Ncep.list.idp_support at lstsrv.ncep.noaa.gov
>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp_support
>>
>
>
>
> --
> Derek Van Pelt
> IDP Onboarding
> NOAA/NCEP/NCO
>
> _______________________________________________
> Ncep.list.idp.mrms.users mailing list
> Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
>