[Ncep.list.idp.mrms.users] [Ncep.list.idp_support] MRMS issue today

Sat Mar 11 18:56:39 UTC 2017

IWSB on-call is still investigating issues with this.

NCEP/SDM

On 11 March 2017 at 18:53, Paul Kirkwood - NOAA Federal <
paul.kirkwood at noaa.gov> wrote:

> 2017.03.11:
>
> SR RFC's are reporting missing Q3 data.
>
>
> Thanks,
> Paul Kirkwood - NOAA Federal
> Chief, Technology Infusion Branch (TIB)
> Science and Technology Services Division (STSD)
> Southern Region Headquarters
> Email: Paul.Kirkwood at noaa.gov
> Phone: 682.703.3745 <(682)%20703-3745>
> Web: http://weather.gov/srh
> "Science Is Organized Knowledge" - Library of Congress
>
>
> On Sat, Mar 11, 2017 at 10:22 AM, Daryl Herzmann <akrherz at iastate.edu>
> wrote:
>
>> Hello,
>>
>> I am noticing an uptick in missing files today.  Are known troubles
>> still afoot?  Here's a printout of some missing files in CPRK in the
>> past 24 hours for the products I care about :)
>>
>> MISS cprk 2017-03-10T22:00Z RadarOnly_QPE_01H
>> MISS cprk 2017-03-11T07:00Z RadarOnly_QPE_01H
>> MISS cprk 2017-03-11T12:00Z RadarOnly_QPE_01H
>> MISS cprk 2017-03-11T13:00Z RadarOnly_QPE_01H
>> MISS cprk 2017-03-10T19:26Z PrecipFlag
>> MISS cprk 2017-03-10T22:00Z PrecipFlag
>> MISS cprk 2017-03-11T02:00Z PrecipFlag
>> MISS cprk 2017-03-11T04:10Z PrecipFlag
>> MISS cprk 2017-03-11T07:00Z PrecipFlag
>> MISS cprk 2017-03-11T09:54Z PrecipFlag
>> MISS cprk 2017-03-11T12:00Z PrecipFlag
>> MISS cprk 2017-03-11T13:00Z PrecipFlag
>> MISS cprk 2017-03-10T21:00Z SeamlessHSR
>> MISS cprk 2017-03-10T22:00Z SeamlessHSR
>> MISS cprk 2017-03-10T23:00Z SeamlessHSR
>> MISS cprk 2017-03-11T02:00Z SeamlessHSR
>> MISS cprk 2017-03-11T04:16Z SeamlessHSR
>> MISS cprk 2017-03-11T07:00Z SeamlessHSR
>> MISS cprk 2017-03-11T09:00Z SeamlessHSR
>> MISS cprk 2017-03-11T12:00Z SeamlessHSR
>> MISS cprk 2017-03-11T13:00Z SeamlessHSR
>> MISS cprk 2017-03-11T14:00Z SeamlessHSR
>> MISS cprk 2017-03-10T19:26Z PrecipRate
>> MISS cprk 2017-03-10T19:56Z PrecipRate
>> MISS cprk 2017-03-10T22:00Z PrecipRate
>> MISS cprk 2017-03-11T01:04Z PrecipRate
>> MISS cprk 2017-03-11T02:00Z PrecipRate
>> MISS cprk 2017-03-11T03:32Z PrecipRate
>> MISS cprk 2017-03-11T03:46Z PrecipRate
>> MISS cprk 2017-03-11T04:10Z PrecipRate
>> MISS cprk 2017-03-11T07:00Z PrecipRate
>> MISS cprk 2017-03-11T07:26Z PrecipRate
>> MISS cprk 2017-03-11T07:46Z PrecipRate
>> MISS cprk 2017-03-11T12:00Z PrecipRate
>> MISS cprk 2017-03-11T12:40Z PrecipRate
>> MISS cprk 2017-03-11T13:00Z PrecipRate
>> MISS cprk 2017-03-11T13:20Z PrecipRate
>> MISS cprk 2017-03-11T14:48Z PrecipRate
>>
>>
>> daryl
>>
>> On Fri, Mar 10, 2017 at 1:58 PM, Derek VanPelt - NOAA Affiliate
>> <derek.vanpelt at noaa.gov> wrote:
>> > MRMS customers,
>> >
>> > The MRMS application endured latencies, and produced some incomplete
>> > products over ~3 hours starting this morning @1530Z.
>> >
>> > We were able to identify the issue quickly, but it took longer than
>> expected
>> > to enact a full fix.
>> >
>> > Apologies.  I believe we have learned from this to the point that we
>> should
>> > be able to avoid future issues.  We will continue enhanced monitoring
>> > throughout the afternoon.
>> >
>> > Derek
>> >
>> > On Fri, Mar 10, 2017 at 2:51 PM, Karen Cooper - NOAA Affiliate
>> > <karen.cooper at noaa.gov> wrote:
>> >>
>> >> Hi Steve,
>> >>
>> >> I'm including the NCEP IDP support group here, as this involves them.
>> >>
>> >> It does appear that from about 15:46 to 18:30 that a lot of the
>> products
>> >> that are created by stitching the tiles were missing data.   This is
>> from
>> >> looking at the College Park system (vm-lnx-mrms-convert1a.ncep.no
>> aa.gov
>> >> specfically).
>> >>
>> >> I'm not sure what would have caused this, but it also looks like there
>> >> delays in the processing of some data.   Our processing has timing
>> controls
>> >> to keep data for a short time -- typically twice the frequency of the
>> >> product, but it will disregard data that is too old so that we
>> mitigate the
>> >> risk of including old data, as it could be misleading to analysts.
>> >>
>> >> I'm going to defer to the NCO team as to what may have caused the
>> slowdown
>> >> during that time.
>> >>
>> >> There are multiple CREFs some being done as 2D products which are not
>> >> tiled, and some done from 3D tiled data, so depending on which one you
>> were
>> >> looking at there might not have been tiles missing.
>> >>
>> >>
>> >> On Fri, Mar 10, 2017 at 11:48 AM, Steven Lack - NOAA Federal
>> >> <steven.a.lack at noaa.gov> wrote:
>> >>>
>> >>> Karen,
>> >>>
>> >>> It looks like starting after 1530 or so we started dropping tiles and
>> we
>> >>> have had intermittent data coming in since then.  This has been a
>> problem
>> >>> with at least RALA and ET-18, somehow CREF looks okay.  This is both
>> on our
>> >>> AWIPS2 feed and our NIDS feed for NAWIPS.
>> >>>
>> >>> Thanks for any info,
>> >>>
>> >>> Steve
>> >>>
>> >>> --
>> >>> Steven A. Lack, Ph.D.
>> >>> Techniques Development Meteorologist
>> >>> Aviation Weather Center
>> >>> 7220 NW 101st Terrace
>> >>> Kansas City, MO 64153
>> >>>
>> >>> office: 816-584-7281
>> >>> cell:  303-882-9395
>> >>>
>> >>> The views or opinions expressed in this email do not necessarily
>> reflect
>> >>> those of NOAA or the United States Government.
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >> “The presence of those seeking the truth is infinitely to be preferred
>> to
>> >> the presence of those who think they’ve found it."
>> >>  -- Terry Prachett
>> >>
>> >> -------------------------------------------
>> >> Karen.Cooper at noaa.gov
>> >>
>> >> Phone#:  405-325-6456
>> >> Cell:   405-834-8559
>> >> National Severe Storms Laboratory
>> >>
>> >>
>> >> _______________________________________________
>> >> Ncep.list.idp_support mailing list
>> >> Ncep.list.idp_support at lstsrv.ncep.noaa.gov
>> >> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>> idp_support
>> >>
>> >
>> >
>> >
>> > --
>> > Derek Van Pelt
>> > IDP Onboarding
>> > NOAA/NCEP/NCO
>> >
>> > _______________________________________________
>> > Ncep.list.idp.mrms.users mailing list
>> > Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
>> > https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>> idp.mrms.users
>> >
>>
>> _______________________________________________
>> Ncep.list.idp.mrms.users mailing list
>> Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>> idp.mrms.users
>>
>
>
> _______________________________________________
> Ncep.list.idp.mrms.users mailing list
> Ncep.list.idp.mrms.users at lstsrv.ncep.noaa.gov
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.idp.mrms.users
>
>

-- 
Senior Duty Meteorologist
NOAA/NWS/NCEP/NCO/OMB
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.idp.mrms.users/attachments/20170311/d61ea2b3/attachment.html 