[Ncep.list.ntbn_nbuild] 16.2.1n1 - NTBN

Stephen Gilbert stephen.gilbert at noaa.gov
Tue May 17 19:11:51 UTC 2016


Hey Chris,

We did apply the spring configuration changes for NSBN GRIB ingest last 
Friday.
Raytheon Omaha is looking at the GRIB ingest on NTBN to see why the 
performance does
not match what they saw on CTBN a few weeks ago.

I looked at some nam conusnest processing and also saw ~20 min 
latencies.  The file itself
was decoded and processed in little over a minute, which means that it 
sat in an ingest queue
for over 18 minutes, probably due to other GRIB files that came in 
before it.  The last I checked,
I saw over 20,000 GRIB messages waiting in the grib.Ingest queue.

As for the GRIB ingest crash, there is reason to believe it may be due 
to a bug
that was recently discovered in 16.2.1.  Omaha has a fix for this, but 
it has not yet made it to the
16.2.1 baseline.  When it does, we can merge it in with our -n builds.  
Meanwhile, they gave us instructions
on what to capture if this happens again, and they can tell us if their 
fix will solve the problem, or
if it is a new issue.

-steve



On 05/17/2016 12:04 PM, Christopher Juckins - NOAA Federal wrote:
> We do see it running but also see high latencies again.  Do you know 
> if the patch to decode data quickly has been applied and configured?  
> The NAM is most problematic.
>
> INFO  2016-05-17 14:43:58,362 [GribPersist-6] Ingest: EDEX: Ingest - 
> grib2:: 
> /nsbn_store/grib/nam_nam.20160517_nam.t12z.conusnest.hiresf36.tm00.grib2 
> processed in: 5.6300 (sec) Latency: 1,210.1510 (sec)
> INFO  2016-05-17 14:43:58,362 [GribPersist-6] Ingest: EDEX: Ingest - 
> grib2:: 
> /nsbn_store/grib/nam_nam.20160517_nam.t12z.conusnest.hiresf36.tm00.grib2 
> processed in: 5.5770 (sec) Latency: 1,210.1510 (sec)
> INFO  2016-05-17 14:43:59,580 [GribPersist-5] Ingest: EDEX: Ingest - 
> grib2:: 
> /nsbn_store/grib/nam_nam.20160517_nam.t12z.conusnest.hiresf36.tm00.grib2 
> processed in: 6.8230 (sec) Latency: 1,211.3690 (sec)
> INFO  2016-05-17 14:43:59,580 [GribPersist-5] Ingest: EDEX: Ingest - 
> grib2:: 
> /nsbn_store/grib/nam_nam.20160517_nam.t12z.conusnest.hiresf36.tm00.grib2 
> processed in: 5.5370 (sec) Latency: 1,211.3690 (sec)
>
> Christopher Juckins
> Meteorologist/Programmer
> Ocean Prediction Center - College Park, MD
> www.opc.ncep.noaa.gov
>
> On 05/17/2016 04:00 PM, Joshua Huber - NOAA Affiliate wrote:
>> and processing data.
>>
>>
>>
>> Joshua Huber
>> Software Engineer
>> NCEP Central Operations/Software Development Branch
>> 5830 University Research Court #1145
>> College Park, MD 20740
>> 301.683.3913
>>
>>
>>
>> On Tue, May 17, 2016 at 4:00 PM, Joshua Huber - NOAA Affiliate 
>> <joshua.huber at noaa.gov> wrote:
>>
>>     I just checked it about 30 minutes ago and it was running.
>>
>>
>>
>>     Joshua Huber
>>     Software Engineer
>>     NCEP Central Operations/Software Development Branch
>>     5830 University Research Court #1145
>>     College Park, MD 20740
>>     301.683.3913 <tel:301.683.3913>
>>
>>
>>
>>     On Tue, May 17, 2016 at 3:46 PM, Christopher Juckins - NOAA
>>     Federal <christopher.juckins at noaa.gov
>>     <mailto:christopher.juckins at noaa.gov>> wrote:
>>
>>         We wanted to check on the grib decoder again - we noticed
>>         latencies will go up to 1300 seconds.
>>
>>         Also saw these messages in the logs:
>>         INFO  2016-05-17 15:37:21,941 [Ingest.GribDecode-12]
>>         GridPersister: EDEX - Max Grids in memory for GridPersister
>>         exceeded.  Waiting for grids to process.
>>         INFO  2016-05-17 15:37:21,941 [Ingest.GribDecode-10]
>>         GridPersister: EDEX - Max Grids in memory for GridPersister
>>         exceeded.  Waiting for grids to process.
>>         INFO  2016-05-17 15:37:21,942 [Ingest.GribDecode-3]
>>         GridPersister: EDEX - Max Grids in memory for GridPersister
>>         exceeded.  Waiting for grids to process.
>>         INFO  2016-05-17 15:37:21,966 [Ingest.GribDecode-5]
>>         GridPersister: EDEX - Max Grids in memory for GridPersister
>>         exceeded.  Waiting for grids to process.
>>         INFO  2016-05-17 15:37:21,966 [Ingest.GribDecode-4]
>>         GridPersister: EDEX - Max Grids in memory for GridPersister
>>         exceeded.  Waiting for grids to process.
>>         INFO  2016-05-17 15:37:21,966 [Ingest.GribDecode-6]
>>         GridPersister: EDEX - Max Grids in memory for GridPersister
>>         exceeded.  Waiting for grids to process.
>>
>>         Do you know if the gribDecoder is still being checked out
>>         after its crash yesterday?
>>
>>         Thanks,
>>         Chris
>>
>>
>>         On Tue, May 10, 2016 at 3:28 PM, David Plummer - NOAA Federal
>>         <david.plummer at noaa.gov> wrote:
>>
>>             the fix will be applied in the next n build, now
>>             scheduled for Friday install.
>>
>>             On Tue, May 10, 2016 at 9:46 AM, Christopher Juckins -
>>             NOAA Federal <christopher.juckins at noaa.gov> wrote:
>>
>>                 Hi again,
>>
>>                 We wanted to find out if the latency fix has been
>>                 applied to NTBN?
>>
>>                 Thanks,
>>                 Chris
>>
>>                 On Thu, May 5, 2016 at 2:24 PM, Joshua Huber - NOAA
>>                 Affiliate <joshua.huber at noaa.gov> wrote:
>>
>>                     Chris--
>>                     We have the necessary configuration files from
>>                     Raytheon to apply to our ncgrib plugin.
>>
>>                     Paul--
>>                     Steve G has the files and should be able to
>>                     supply them to Shawn G on this.
>>
>>
>>                     On Thursday, May 5, 2016, Christopher Juckins -
>>                     NOAA Federal <christopher.juckins at noaa.gov> wrote:
>>
>>                         Paul,
>>
>>                         I am not sure who to direct this observation
>>                         to...but we noticed the latency for decoding
>>                         model data is high again.  Could someone
>>                         check to see if the fix that was applied to
>>                         CTBN has been applied to NTBN?
>>
>>                         Here is how I am looking for the latency
>>                         values in the EDEX logs:
>>
>>                         [cjuckins at dx3-ntbn: ~]$ cat
>>                         /awips2/edex/logs/edex-ingestGrib-20160505.log |grep
>>                         processed | awk -F " " '{print $16}' | sort
>>                         -n | uniq | tail -10
>>                         2,935.5000
>>                         2,936.8120
>>                         2,937.8580
>>                         2,940.7790
>>                         2,946.4040
>>                         2,947.0450
>>                         2,948.7720
>>                         2,949.9740
>>                         2,951.3670
>>                         2,953.0760
>>
>>                         The NAM is particularly troublesome:
>>
>>                         [cjuckins at dx3-ntbn: ~]$ cat
>>                         /awips2/edex/logs/edex-ingestGrib-20160505.log |grep
>>                         processed | grep conusnest |grep 00z | tail -5
>>                         INFO 2016-05-05 03:11:02,992
>>                         [Ingest.ncGrib-8] Ingest: EDEX: Ingest -
>>                         grib2::
>>                         /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
>>                         processed in: 0.4170 (sec) Latency:
>>                         1,582.2760 (sec)
>>                         INFO 2016-05-05 03:11:03,929
>>                         [Ingest.ncGrib-8] Ingest: EDEX: Ingest -
>>                         grib2::
>>                         /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
>>                         processed in: 0.9360 (sec) Latency:
>>                         1,583.2130 (sec)
>>                         INFO 2016-05-05 03:11:04,466
>>                         [Ingest.ncGrib-8] Ingest: EDEX: Ingest -
>>                         grib2::
>>                         /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
>>                         processed in: 0.5360 (sec) Latency:
>>                         1,583.7500 (sec)
>>                         INFO 2016-05-05 03:11:05,028
>>                         [Ingest.ncGrib-8] Ingest: EDEX: Ingest -
>>                         grib2::
>>                         /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
>>                         processed in: 0.5620 (sec) Latency:
>>                         1,584.3120 (sec)
>>                         INFO 2016-05-05 03:11:06,339
>>                         [Ingest.ncGrib-8] Ingest: EDEX: Ingest -
>>                         grib2::
>>                         /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
>>                         processed in: 1.3100 (sec) Latency:
>>                         1,585.6230 (sec)
>>
>>                         They should be on the order of a few seconds,
>>                         like this on CTBN:
>>
>>                         [cjuckins at dx3-ctbn: ~]$ cat
>>                         /awips2/edex/logs/edex-ingestGrib-20160505.log |grep
>>                         processed | grep conusnest |grep 00z | tail -5
>>                         INFO 2016-05-05 02:44:38,806 [GribPersist-6]
>>                         Ingest: EDEX: Ingest - grib2::
>>                         /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
>>                         processed in: 2.5800 (sec) Latency: 4.4080 (sec)
>>                         INFO 2016-05-05 02:44:38,843 [GribPersist-1]
>>                         Ingest: EDEX: Ingest - grib2::
>>                         /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
>>                         processed in: 1.4310 (sec) Latency: 4.4440 (sec)
>>                         INFO 2016-05-05 02:44:38,851 [GribPersist-4]
>>                         Ingest: EDEX: Ingest - grib2::
>>                         /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
>>                         processed in: 2.6270 (sec) Latency: 4.4530 (sec)
>>                         INFO 2016-05-05 02:44:39,462 [GribPersist-4]
>>                         Ingest: EDEX: Ingest - grib2::
>>                         /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
>>                         processed in: 3.2490 (sec) Latency: 5.0640 (sec)
>>                         INFO 2016-05-05 02:44:39,462 [GribPersist-4]
>>                         Ingest: EDEX: Ingest - grib2::
>>                         /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
>>                         processed in: 3.2490 (sec) Latency: 5.0640 (sec)
>>
>>                         Thanks,
>>                         Chris
>>
>>                         Christopher Juckins
>>                         Meteorologist/Programmer
>>                         Ocean Prediction Center - College Park, MD
>>                         www.opc.ncep.noaa.gov
>>                         <http://www.opc.ncep.noaa.gov>
>>
>>                         On 05/04/2016 08:45 PM, Paul Iwugo - NOAA
>>                         Federal wrote:
>>>                         Thanks Shawn.
>>>
>>>                         Paul Iwugo, PMP
>>>                         Chief, Software Development Branch
>>>                         NOAA | NWS | NCEP | NCO
>>>                         office - 301.683.1303 <tel:301.683.1303>
>>>                         mobile - 301.543.0408 <tel:301.543.0408>
>>>                         paul.iwugo at noaa.gov
>>>
>>>                         On Wed, May 4, 2016 at 2:15 PM, Shawn
>>>                         Gindhart - NOAA Affiliate
>>>                         <shawn.gindhart at noaa.gov> wrote:
>>>
>>>                             It's complete.
>>>
>>>                             Thanks,
>>>                             Shawn
>>>
>>>                             On Wed, May 4, 2016 at 1:29 PM, Shawn
>>>                             Gindhart - NOAA Affiliate
>>>                             <shawn.gindhart at noaa.gov> wrote:
>>>
>>>                                 Installing 16.2.1-29n1 now.
>>>
>>>                                 Thanks,
>>>                                 Shawn
>>>
>>>                                 On Wed, May 4, 2016 at 12:30 PM,
>>>                                 Shawn Gindhart - NOAA Affiliate
>>>                                 <shawn.gindhart at noaa.gov> wrote:
>>>
>>>                                     Yes it passed. I confirmed with
>>>                                     ENV team.
>>>
>>>                                     Thanks,
>>>                                     Shawn
>>>
>>>                                     On Wed, May 4, 2016 at 12:19 PM,
>>>                                     Tiros Lee - NOAA Federal
>>>                                     <tiros.lee at noaa.gov> wrote:
>>>
>>>                                         Have -29 passed the test?
>>>
>>>                                         Tiros Lee
>>>                                         Software Release Lead
>>>                                         NOAA/NWS/NCEP/NCO #1040
>>>                                         301-683-3843
>>>                                         <tel:301-683-3843> (W)
>>>
>>>                                         On Wed, May 4, 2016 at 4:10
>>>                                         PM, Shawn Gindhart - NOAA
>>>                                         Affiliate
>>>                                         <shawn.gindhart at noaa.gov> wrote:
>>>
>>>                                             There was a problem with
>>>                                             CAVE portion of the
>>>                                             install. SS CM Team is
>>>                                             rebuilding with -29n1
>>>
>>>                                             Thanks for your patience.
>>>                                             -Shawn
>>>
>>>                                             On Wed, May 4, 2016 at
>>>                                             10:48 AM, Shawn Gindhart
>>>                                             - NOAA Affiliate
>>>                                             <shawn.gindhart at noaa.gov> wrote:
>>>
>>>                                                 Beginning
>>>                                                 16.2.1-28n1 build on
>>>                                                 NTBN.
>>>
>>>                                                 Thanks,
>>>                                                 Shawn
>>>
>>>                                                 -- 
>>>                                                 Shawn Gindhart
>>>                                                 NCEP Central Operations
>>>                                                 301-683-3919
>>>                                                 <tel:301-683-3919>
>>>
>>>
>>>
>>>
>>>                                             -- 
>>>                                             Shawn Gindhart
>>>                                             NCEP Central Operations
>>>                                             301-683-3919
>>>                                             <tel:301-683-3919>
>>>
>>>
>>>
>>>
>>>
>>>                                     -- 
>>>                                     Shawn Gindhart
>>>                                     NCEP Central Operations
>>>                                     301-683-3919 <tel:301-683-3919>
>>>
>>>
>>>
>>>
>>>                                 -- 
>>>                                 Shawn Gindhart
>>>                                 NCEP Central Operations
>>>                                 301-683-3919 <tel:301-683-3919>
>>>
>>>
>>>
>>>
>>>                             -- 
>>>                             Shawn Gindhart
>>>                             NCEP Central Operations
>>>                             301-683-3919 <tel:301-683-3919>
>>>
>>>
>>
>>
>>
>>                     -- 
>>
>>
>>                     Joshua Huber
>>                     Software Engineer
>>                     NCEP Central Operations/Software Development Branch
>>                     5830 University Research Court #1145
>>                     College Park, MD 20740
>>                     301.683.3913 <tel:301.683.3913>
>>
>>
>>
>>
>>
>>
>>                 -- 
>>                 Christopher Juckins
>>                 Meteorologist/Programmer
>>                 NCEP Ocean Prediction Center
>>                 www.opc.ncep.noaa.gov
>>
>>
>>
>>
>>             -- 
>>             NCEP developers and customers visit the VLAB AWIPS II
>>             NCEP Community
>>             <https://vlab.ncep.noaa.gov/group/ncep-a2cp/home>
>>             (newcomers will need to be added)
>>
>>             W. David Plummer
>>             National Centers AWIPS Team Lead
>>             NCEP Central Operations / Systems Integration Branch
>>             (301) 683‐3917 <tel:%28301%29%C2%A0683%E2%80%903917>
>>
>>
>>
>>             Department of Commerce
>>             National Oceanic and Atmospheric Administration
>>             *NCWCP (*W/NP1)
>>             5830 University Research Court, 1150
>>             College Park, MD  20740-3818
>>
>>
>>
>>
>>
>>         -- 
>>         Christopher Juckins
>>         Meteorologist/Programmer
>>         NCEP Ocean Prediction Center
>>         www.opc.ncep.noaa.gov <http://www.opc.ncep.noaa.gov>
>>
>>
>>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.ntbn_nbuild/attachments/20160517/0e6b049a/attachment-0001.html 


More information about the Ncep.list.ntbn_nbuild mailing list