<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Hey Chris,<br>
      <br>
      We did apply the spring configuration changes for NSBN GRIB ingest
      last Friday.  <br>
      Raytheon Omaha is looking at the GRIB ingest on NTBN to see why
      the performance does<br>
      not match what they saw on CTBN a few weeks ago.<br>
      <br>
      I looked at some nam conusnest processing and also saw ~20 min
      latencies.  The file itself<br>
      was decoded and processed in little over a minute, which means
      that it sat in an ingest queue<br>
      for over 18 minutes, probably due to other GRIB files that came in
      before it.  The last I checked, <br>
      I saw over 20,000 GRIB messages waiting in the grib.Ingest queue.
      <br>
      <br>
      As for the GRIB ingest crash, there is reason to believe it may be
      due to a bug<br>
      that was recently discovered in 16.2.1.  Omaha has a fix for this,
      but it has not yet made it to the<br>
      16.2.1 baseline.  When it does, we can merge it in with our -n
      builds.  Meanwhile, they gave us instructions<br>
      on what to capture if this happens again, and they can tell us if
      their fix will solve the problem, or<br>
      if it is a new issue.<br>
      <br>
      -steve<br>
      <br>
      <br>
      <br>
      On 05/17/2016 12:04 PM, Christopher Juckins - NOAA Federal wrote:<br>
    </div>
    <blockquote cite="mid:573B40F5.1090501@noaa.gov" type="cite">
      <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
      We do see it running but also see high latencies again.  Do you
      know if the patch to decode data quickly has been applied and
      configured?  The NAM is most problematic.<br>
      <br>
      INFO  2016-05-17 14:43:58,362 [GribPersist-6] Ingest: EDEX: Ingest
      - grib2::
      /nsbn_store/grib/nam_nam.20160517_nam.t12z.conusnest.hiresf36.tm00.grib2
      processed in: 5.6300 (sec) Latency: 1,210.1510 (sec)<br>
      INFO  2016-05-17 14:43:58,362 [GribPersist-6] Ingest: EDEX: Ingest
      - grib2::
      /nsbn_store/grib/nam_nam.20160517_nam.t12z.conusnest.hiresf36.tm00.grib2
      processed in: 5.5770 (sec) Latency: 1,210.1510 (sec)<br>
      INFO  2016-05-17 14:43:59,580 [GribPersist-5] Ingest: EDEX: Ingest
      - grib2::
      /nsbn_store/grib/nam_nam.20160517_nam.t12z.conusnest.hiresf36.tm00.grib2
      processed in: 6.8230 (sec) Latency: 1,211.3690 (sec)<br>
      INFO  2016-05-17 14:43:59,580 [GribPersist-5] Ingest: EDEX: Ingest
      - grib2::
      /nsbn_store/grib/nam_nam.20160517_nam.t12z.conusnest.hiresf36.tm00.grib2
      processed in: 5.5370 (sec) Latency: 1,211.3690 (sec)<br>
      <br>
      <pre class="moz-signature" cols="72">Christopher Juckins
Meteorologist/Programmer
Ocean Prediction Center - College Park, MD
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="http://www.opc.ncep.noaa.gov">www.opc.ncep.noaa.gov</a>

</pre>
      <div class="moz-cite-prefix">On 05/17/2016 04:00 PM, Joshua Huber
        - NOAA Affiliate wrote:<br>
      </div>
      <blockquote
cite="mid:CAPYNhibcBggBnr3YZJo_YxJUavxGzcB3DjjZTLJpYKE2Apzsiw@mail.gmail.com"
        type="cite">
        <div dir="ltr">and processing data.<br>
        </div>
        <div class="gmail_extra"><br clear="all">
          <div>
            <div class="gmail_signature">
              <div dir="ltr">
                <div>
                  <div dir="ltr">
                    <div dir="ltr">
                      <div dir="ltr">
                        <div dir="ltr">
                          <div dir="ltr">
                            <div dir="ltr">
                              <div>
                                <div><br>
                                  <img moz-do-not-send="true"
                                    src="http://i61.tinypic.com/2s8k8z7.png"><br>
                                </div>
                                Joshua Huber<br>
                                Software Engineer</div>
                              <div>NCEP Central Operations/Software
                                Development Branch</div>
                              5830 University Research Court #1145<br>
                              <div>College Park, MD 20740<br>
                                301.683.3913<br>
                                <br>
                                <br>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                </div>
              </div>
            </div>
          </div>
          <br>
          <div class="gmail_quote">On Tue, May 17, 2016 at 4:00 PM,
            Joshua Huber - NOAA Affiliate <span dir="ltr">&lt;<a
                moz-do-not-send="true" class="moz-txt-link-abbreviated"
                href="mailto:joshua.huber@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:joshua.huber@noaa.gov">joshua.huber@noaa.gov</a></a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">
              <div dir="ltr">I just checked it about 30 minutes ago and
                it was running.<br>
              </div>
              <div class="gmail_extra"><span class=""><br clear="all">
                  <div>
                    <div>
                      <div dir="ltr">
                        <div>
                          <div dir="ltr">
                            <div dir="ltr">
                              <div dir="ltr">
                                <div dir="ltr">
                                  <div dir="ltr">
                                    <div dir="ltr">
                                      <div>
                                        <div><br>
                                          <img moz-do-not-send="true"
                                            src="http://i61.tinypic.com/2s8k8z7.png"><br>
                                        </div>
                                        Joshua Huber<br>
                                        Software Engineer</div>
                                      <div>NCEP Central
                                        Operations/Software Development
                                        Branch</div>
                                      5830 University Research Court
                                      #1145<br>
                                      <div>College Park, MD 20740<br>
                                        <a moz-do-not-send="true"
                                          href="tel:301.683.3913"
                                          value="+13016833913"
                                          target="_blank">301.683.3913</a><br>
                                        <br>
                                        <br>
                                      </div>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                      </div>
                    </div>
                  </div>
                  <br>
                </span>
                <div>
                  <div class="h5">
                    <div class="gmail_quote">On Tue, May 17, 2016 at
                      3:46 PM, Christopher Juckins - NOAA Federal <span
                        dir="ltr">&lt;<a moz-do-not-send="true"
                          href="mailto:christopher.juckins@noaa.gov"
                          target="_blank">christopher.juckins@noaa.gov</a>&gt;</span>
                      wrote:<br>
                      <blockquote class="gmail_quote" style="margin:0 0
                        0 .8ex;border-left:1px #ccc
                        solid;padding-left:1ex">
                        <div dir="ltr">
                          <div>
                            <div>
                              <div>
                                <div>We wanted to check on the grib
                                  decoder again - we noticed latencies
                                  will go up to 1300 seconds.  <br>
                                  <br>
                                </div>
                                Also saw these messages in the logs:<br>
                                INFO  2016-05-17 15:37:21,941
                                [Ingest.GribDecode-12] GridPersister:
                                EDEX - Max Grids in memory for
                                GridPersister exceeded.  Waiting for
                                grids to process.<br>
                                INFO  2016-05-17 15:37:21,941
                                [Ingest.GribDecode-10] GridPersister:
                                EDEX - Max Grids in memory for
                                GridPersister exceeded.  Waiting for
                                grids to process.<br>
                                INFO  2016-05-17 15:37:21,942
                                [Ingest.GribDecode-3] GridPersister:
                                EDEX - Max Grids in memory for
                                GridPersister exceeded.  Waiting for
                                grids to process.<br>
                                INFO  2016-05-17 15:37:21,966
                                [Ingest.GribDecode-5] GridPersister:
                                EDEX - Max Grids in memory for
                                GridPersister exceeded.  Waiting for
                                grids to process.<br>
                                INFO  2016-05-17 15:37:21,966
                                [Ingest.GribDecode-4] GridPersister:
                                EDEX - Max Grids in memory for
                                GridPersister exceeded.  Waiting for
                                grids to process.<br>
                                INFO  2016-05-17 15:37:21,966
                                [Ingest.GribDecode-6] GridPersister:
                                EDEX - Max Grids in memory for
                                GridPersister exceeded.  Waiting for
                                grids to process.<br>
                                <br>
                              </div>
                              Do you know if the gribDecoder is still
                              being checked out after its crash
                              yesterday?<br>
                              <br>
                            </div>
                            Thanks,<br>
                          </div>
                          Chris<br>
                          <div>
                            <div>
                              <div>
                                <div>
                                  <div>
                                    <div><br>
                                    </div>
                                  </div>
                                </div>
                              </div>
                            </div>
                          </div>
                        </div>
                        <div>
                          <div>
                            <div class="gmail_extra"><br>
                              <div class="gmail_quote">On Tue, May 10,
                                2016 at 3:28 PM, David Plummer - NOAA
                                Federal <span dir="ltr">&lt;<a
                                    moz-do-not-send="true"
                                    class="moz-txt-link-abbreviated"
                                    href="mailto:david.plummer@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:david.plummer@noaa.gov">david.plummer@noaa.gov</a></a>&gt;</span>
                                wrote:<br>
                                <blockquote class="gmail_quote"
                                  style="margin:0 0 0
                                  .8ex;border-left:1px #ccc
                                  solid;padding-left:1ex">
                                  <div dir="ltr">the fix will be applied
                                    in the next n build, now scheduled
                                    for Friday install.</div>
                                  <div class="gmail_extra">
                                    <div>
                                      <div><br>
                                        <div class="gmail_quote">On Tue,
                                          May 10, 2016 at 9:46 AM,
                                          Christopher Juckins - NOAA
                                          Federal <span dir="ltr">&lt;<a
                                              moz-do-not-send="true"
                                              class="moz-txt-link-abbreviated"
href="mailto:christopher.juckins@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:christopher.juckins@noaa.gov">christopher.juckins@noaa.gov</a></a>&gt;</span>
                                          wrote:<br>
                                          <blockquote
                                            class="gmail_quote"
                                            style="margin:0 0 0
                                            .8ex;border-left:1px #ccc
                                            solid;padding-left:1ex">
                                            <div dir="ltr">
                                              <div>
                                                <div>
                                                  <div>Hi again,<br>
                                                    <br>
                                                  </div>
                                                  We wanted to find out
                                                  if the latency fix has
                                                  been applied to NTBN?<br>
                                                  <br>
                                                </div>
                                                Thanks,<br>
                                              </div>
                                              Chris<br>
                                            </div>
                                            <div class="gmail_extra">
                                              <div>
                                                <div><br>
                                                  <div
                                                    class="gmail_quote">On

                                                    Thu, May 5, 2016 at
                                                    2:24 PM, Joshua
                                                    Huber - NOAA
                                                    Affiliate <span
                                                      dir="ltr">&lt;<a
                                                        moz-do-not-send="true"
class="moz-txt-link-abbreviated" href="mailto:joshua.huber@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:joshua.huber@noaa.gov">joshua.huber@noaa.gov</a></a>&gt;</span>
                                                    wrote:<br>
                                                    <blockquote
                                                      class="gmail_quote"
                                                      style="margin:0 0
                                                      0
                                                      .8ex;border-left:1px
                                                      #ccc
                                                      solid;padding-left:1ex">Chris--

                                                      <div>We have the
                                                        necessary
                                                        configuration
                                                        files from
                                                        Raytheon to
                                                        apply to our
                                                        ncgrib plugin.</div>
                                                      <div><br>
                                                      </div>
                                                      <div>Paul--</div>
                                                      <div>Steve G has
                                                        the files and
                                                        should be able
                                                        to supply them
                                                        to <span></span>Shawn

                                                        G on this.</div>
                                                      <div>
                                                        <div>
                                                          <div><br>
                                                          <br>
                                                          On Thursday,
                                                          May 5, 2016,
                                                          Christopher
                                                          Juckins - NOAA
                                                          Federal &lt;<a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
                                                          href="mailto:christopher.juckins@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:christopher.juckins@noaa.gov">christopher.juckins@noaa.gov</a></a>&gt;

                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div
                                                          text="#000000"
bgcolor="#FFFFFF"> Paul,<br>
                                                          <br>
                                                          I am not sure
                                                          who to direct
                                                          this
                                                          observation
                                                          to...but we
                                                          noticed the
                                                          latency for
                                                          decoding model
                                                          data is high
                                                          again.  Could
                                                          someone check
                                                          to see if the
                                                          fix that was
                                                          applied to
                                                          CTBN has been
                                                          applied to
                                                          NTBN?<br>
                                                          <br>
                                                          Here is how I
                                                          am looking for
                                                          the latency
                                                          values in the
                                                          EDEX logs:<br>
                                                          <br>
                                                          [cjuckins@dx3-ntbn:

                                                          ~]$ cat
                                                          /awips2/edex/logs/edex-ingestGrib-20160505.log
                                                          |grep
                                                          processed |
                                                          awk -F " "
                                                          '{print $16}'
                                                          | sort -n |
                                                          uniq | tail
                                                          -10<br>
                                                          2,935.5000<br>
                                                          2,936.8120<br>
                                                          2,937.8580<br>
                                                          2,940.7790<br>
                                                          2,946.4040<br>
                                                          2,947.0450<br>
                                                          2,948.7720<br>
                                                          2,949.9740<br>
                                                          2,951.3670<br>
                                                          2,953.0760<br>
                                                          <br>
                                                          The NAM is
                                                          particularly
                                                          troublesome:<br>
                                                          <br>
                                                          [cjuckins@dx3-ntbn:

                                                          ~]$ cat
                                                          /awips2/edex/logs/edex-ingestGrib-20160505.log
                                                          |grep
                                                          processed |
                                                          grep conusnest
                                                          |grep 00z |
                                                          tail -5<br>
                                                          INFO 
                                                          2016-05-05
                                                          03:11:02,992
                                                          [Ingest.ncGrib-8]
                                                          Ingest: EDEX:
                                                          Ingest -
                                                          grib2::
                                                          /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
                                                          processed in:
                                                          0.4170 (sec)
                                                          Latency:
                                                          1,582.2760
                                                          (sec)<br>
                                                          INFO 
                                                          2016-05-05
                                                          03:11:03,929
                                                          [Ingest.ncGrib-8]
                                                          Ingest: EDEX:
                                                          Ingest -
                                                          grib2::
                                                          /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
                                                          processed in:
                                                          0.9360 (sec)
                                                          Latency:
                                                          1,583.2130
                                                          (sec)<br>
                                                          INFO 
                                                          2016-05-05
                                                          03:11:04,466
                                                          [Ingest.ncGrib-8]
                                                          Ingest: EDEX:
                                                          Ingest -
                                                          grib2::
                                                          /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
                                                          processed in:
                                                          0.5360 (sec)
                                                          Latency:
                                                          1,583.7500
                                                          (sec)<br>
                                                          INFO 
                                                          2016-05-05
                                                          03:11:05,028
                                                          [Ingest.ncGrib-8]
                                                          Ingest: EDEX:
                                                          Ingest -
                                                          grib2::
                                                          /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
                                                          processed in:
                                                          0.5620 (sec)
                                                          Latency:
                                                          1,584.3120
                                                          (sec)<br>
                                                          INFO 
                                                          2016-05-05
                                                          03:11:06,339
                                                          [Ingest.ncGrib-8]
                                                          Ingest: EDEX:
                                                          Ingest -
                                                          grib2::
                                                          /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
                                                          processed in:
                                                          1.3100 (sec)
                                                          Latency:
                                                          1,585.6230
                                                          (sec)<br>
                                                          <br>
                                                          They should be
                                                          on the order
                                                          of a few
                                                          seconds, like
                                                          this on CTBN:<br>
                                                          <br>
                                                          [cjuckins@dx3-ctbn:

                                                          ~]$ cat
                                                          /awips2/edex/logs/edex-ingestGrib-20160505.log
                                                          |grep
                                                          processed |
                                                          grep conusnest
                                                          |grep 00z |
                                                          tail -5<br>
                                                          INFO 
                                                          2016-05-05
                                                          02:44:38,806
                                                          [GribPersist-6]
                                                          Ingest: EDEX:
                                                          Ingest -
                                                          grib2::
                                                          /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
                                                          processed in:
                                                          2.5800 (sec)
                                                          Latency:
                                                          4.4080 (sec)<br>
                                                          INFO 
                                                          2016-05-05
                                                          02:44:38,843
                                                          [GribPersist-1]
                                                          Ingest: EDEX:
                                                          Ingest -
                                                          grib2::
                                                          /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
                                                          processed in:
                                                          1.4310 (sec)
                                                          Latency:
                                                          4.4440 (sec)<br>
                                                          INFO 
                                                          2016-05-05
                                                          02:44:38,851
                                                          [GribPersist-4]
                                                          Ingest: EDEX:
                                                          Ingest -
                                                          grib2::
                                                          /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
                                                          processed in:
                                                          2.6270 (sec)
                                                          Latency:
                                                          4.4530 (sec)<br>
                                                          INFO 
                                                          2016-05-05
                                                          02:44:39,462
                                                          [GribPersist-4]
                                                          Ingest: EDEX:
                                                          Ingest -
                                                          grib2::
                                                          /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
                                                          processed in:
                                                          3.2490 (sec)
                                                          Latency:
                                                          5.0640 (sec)<br>
                                                          INFO 
                                                          2016-05-05
                                                          02:44:39,462
                                                          [GribPersist-4]
                                                          Ingest: EDEX:
                                                          Ingest -
                                                          grib2::
                                                          /nsbn_store/grib/nam_nam.20160505_nam.t00z.conusnest.hiresf60.tm00.grib2
                                                          processed in:
                                                          3.2490 (sec)
                                                          Latency:
                                                          5.0640 (sec)<br>
                                                          <br>
                                                          Thanks,<br>
                                                          Chris<br>
                                                          <br>
                                                          <pre cols="72">Christopher Juckins
Meteorologist/Programmer
Ocean Prediction Center - College Park, MD
<a moz-do-not-send="true" href="http://www.opc.ncep.noaa.gov" target="_blank">www.opc.ncep.noaa.gov</a>

</pre>
                                                          <div>On
                                                          05/04/2016
                                                          08:45 PM, Paul
                                                          Iwugo - NOAA
                                                          Federal wrote:<br>
                                                          </div>
                                                          <blockquote
                                                          type="cite">
                                                          <div dir="ltr">
                                                          <div
                                                          class="gmail_default"
style="font-family:tahoma,sans-serif">Thanks Shawn.</div>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><br
                                                          clear="all">
                                                          <div>
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr"><font
                                                          face="tahoma,
                                                          sans-serif"
                                                          color="#0b5394"
                                                          size="4">Paul
                                                          Iwugo, PMP</font>
                                                          <div><font
                                                          face="tahoma,
                                                          sans-serif"
                                                          color="#0b5394"
                                                          size="2">Chief,

                                                          Software
                                                          Development
                                                          Branch</font></div>
                                                          <div><font
                                                          face="tahoma,
                                                          sans-serif"
                                                          color="#0b5394"
                                                          size="2">NOAA
                                                          | NWS | NCEP |
                                                          NCO</font></div>
                                                          <div><font
                                                          face="tahoma,
                                                          sans-serif"
                                                          color="#0b5394"
                                                          size="2">office

                                                          - <a
                                                          moz-do-not-send="true"
href="tel:301.683.1303" value="+13016831303" target="_blank">301.683.1303</a></font></div>
                                                          <div><font
                                                          face="tahoma,
                                                          sans-serif"
                                                          color="#0b5394"><span
style="font-size:12.8px">mobile - </span><a moz-do-not-send="true"
                                                          href="tel:301.543.0408"
value="+13015430408" target="_blank">301.543.0408</a></font><font
                                                          face="tahoma,
                                                          sans-serif"
                                                          color="#0b5394"
                                                          size="2"><br>
                                                          <a
                                                          moz-do-not-send="true"
class="moz-txt-link-abbreviated" href="mailto:paul.iwugo@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:paul.iwugo@noaa.gov">paul.iwugo@noaa.gov</a></a></font></div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Wed, May 4,
                                                          2016 at 2:15
                                                          PM, Shawn
                                                          Gindhart -
                                                          NOAA Affiliate
                                                          <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
                                                          href="mailto:shawn.gindhart@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:shawn.gindhart@noaa.gov">shawn.gindhart@noaa.gov</a></a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>It's
                                                          complete.<br>
                                                          <br>
                                                          </div>
                                                          Thanks,<br>
                                                          </div>
                                                          Shawn<br>
                                                          </div>
                                                          <div>
                                                          <div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Wed, May 4,
                                                          2016 at 1:29
                                                          PM, Shawn
                                                          Gindhart -
                                                          NOAA Affiliate
                                                          <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
                                                          href="mailto:shawn.gindhart@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:shawn.gindhart@noaa.gov">shawn.gindhart@noaa.gov</a></a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>Installing

                                                          16.2.1-29n1
                                                          now.<br>
                                                          <br>
                                                          </div>
                                                          Thanks,<br>
                                                          </div>
                                                          Shawn<br>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Wed, May 4,
                                                          2016 at 12:30
                                                          PM, Shawn
                                                          Gindhart -
                                                          NOAA Affiliate
                                                          <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
                                                          href="mailto:shawn.gindhart@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:shawn.gindhart@noaa.gov">shawn.gindhart@noaa.gov</a></a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>Yes it
                                                          passed. I
                                                          confirmed with
                                                          ENV team.<br>
                                                          <br>
                                                          </div>
                                                          Thanks,<br>
                                                          </div>
                                                          Shawn<br>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Wed, May 4,
                                                          2016 at 12:19
                                                          PM, Tiros Lee
                                                          - NOAA Federal
                                                          <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
                                                          href="mailto:tiros.lee@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:tiros.lee@noaa.gov">tiros.lee@noaa.gov</a></a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">Have

                                                          -29 passed the
                                                          test?<br>
                                                          </div>
                                                          <div
                                                          class="gmail_extra"><br
                                                          clear="all">
                                                          <div>
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div>Tiros Lee<br>
                                                          Software
                                                          Release Lead<br>
                                                          NOAA/NWS/NCEP/NCO


                                                          #1040<br>
                                                          </div>
                                                          <div><a
                                                          moz-do-not-send="true"
href="tel:301-683-3843" value="+13016833843" target="_blank">301-683-3843</a>
                                                          (W)<br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                          <div
                                                          class="gmail_quote">On

                                                          Wed, May 4,
                                                          2016 at 4:10
                                                          PM, Shawn
                                                          Gindhart -
                                                          NOAA Affiliate
                                                          <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
                                                          href="mailto:shawn.gindhart@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:shawn.gindhart@noaa.gov">shawn.gindhart@noaa.gov</a></a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>There was
                                                          a problem with
                                                          CAVE portion
                                                          of the
                                                          install. SS CM
                                                          Team is
                                                          rebuilding
                                                          with -29n1<br>
                                                          <br>
                                                          </div>
                                                          Thanks for
                                                          your patience.<span><font
color="#888888"><br>
                                                          </font></span></div>
                                                          <span><font
                                                          color="#888888">-Shawn<br>
                                                          </font></span></div>
                                                          <div>
                                                          <div>
                                                          <div
                                                          class="gmail_extra"><br>
                                                          <div
                                                          class="gmail_quote">On


                                                          Wed, May 4,
                                                          2016 at 10:48
                                                          AM, Shawn
                                                          Gindhart -
                                                          NOAA Affiliate
                                                          <span
                                                          dir="ltr">&lt;<a
moz-do-not-send="true" class="moz-txt-link-abbreviated"
                                                          href="mailto:shawn.gindhart@noaa.gov"><a class="moz-txt-link-abbreviated" href="mailto:shawn.gindhart@noaa.gov">shawn.gindhart@noaa.gov</a></a>&gt;</span>
                                                          wrote:<br>
                                                          <blockquote
                                                          class="gmail_quote"
                                                          style="margin:0

                                                          0 0
                                                          .8ex;border-left:1px
                                                          #ccc
                                                          solid;padding-left:1ex">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div>Beginning
                                                          16.2.1-28n1
                                                          build on NTBN.<br>
                                                          <br>
                                                          </div>
                                                          Thanks,<br>
                                                          </div>
                                                          Shawn<span><font
color="#888888"><span><font color="#888888"><br clear="all">
                                                          <div>
                                                          <div>
                                                          <div><br>
                                                          -- <br>
                                                          <div>
                                                          <div dir="ltr"><span>Shawn


                                                          Gindhart<br>
                                                          </span>NCEP
                                                          Central
                                                          Operations<br>
                                                          <div><a
                                                          moz-do-not-send="true"
href="tel:301-683-3919" value="+13016833919" target="_blank">301-683-3919</a></div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </font></span></font></span></div>
                                                          <span><font
                                                          color="#888888">
                                                          </font></span></blockquote>
                                                          </div>
                                                          <span><font
                                                          color="#888888"><br>
                                                          <br
                                                          clear="all">
                                                          <span><font
                                                          color="#888888"><br>
                                                          -- <br>
                                                          <div>
                                                          <div dir="ltr"><span>Shawn


                                                          Gindhart<br>
                                                          </span>NCEP
                                                          Central
                                                          Operations<br>
                                                          <div><a
                                                          moz-do-not-send="true"
href="tel:301-683-3919" value="+13016833919" target="_blank">301-683-3919</a></div>
                                                          </div>
                                                          </div>
                                                          </font></span></font></span></div>
                                                          <span><font
                                                          color="#888888">
                                                          </font></span></div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <span><font
                                                          color="#888888"><br>
                                                          </font></span></div>
                                                          <span><font
                                                          color="#888888">
                                                          </font></span></blockquote>
                                                          </div>
                                                          <span><font
                                                          color="#888888"><br>
                                                          <br
                                                          clear="all">
                                                          <span><font
                                                          color="#888888"><br>
                                                          -- <br>
                                                          <div>
                                                          <div dir="ltr"><span>Shawn


                                                          Gindhart<br>
                                                          </span>NCEP
                                                          Central
                                                          Operations<br>
                                                          <div><a
                                                          moz-do-not-send="true"
href="tel:301-683-3919" value="+13016833919" target="_blank">301-683-3919</a></div>
                                                          </div>
                                                          </div>
                                                          </font></span></font></span></div>
                                                          <span><font
                                                          color="#888888">
                                                          </font></span></blockquote>
                                                          </div>
                                                          <span><font
                                                          color="#888888"><br>
                                                          <br
                                                          clear="all">
                                                          <br>
                                                          -- <br>
                                                          <div>
                                                          <div dir="ltr"><span>Shawn

                                                          Gindhart<br>
                                                          </span>NCEP
                                                          Central
                                                          Operations<br>
                                                          <div><a
                                                          moz-do-not-send="true"
href="tel:301-683-3919" value="+13016833919" target="_blank">301-683-3919</a></div>
                                                          </div>
                                                          </div>
                                                          </font></span></div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          <br
                                                          clear="all">
                                                          <br>
                                                          -- <br>
                                                          <div>
                                                          <div dir="ltr"><span>Shawn

                                                          Gindhart<br>
                                                          </span>NCEP
                                                          Central
                                                          Operations<br>
                                                          <div><a
                                                          moz-do-not-send="true"
href="tel:301-683-3919" value="+13016833919" target="_blank">301-683-3919</a></div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          <br>
                                                          </div>
                                                          </blockquote>
                                                          </div>
                                                          <br>
                                                          <br>
                                                        </div>
                                                      </div>
                                                      <span><font
                                                          color="#888888">--

                                                          <br>
                                                          <div dir="ltr">
                                                          <div>
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div dir="ltr">
                                                          <div>
                                                          <div><br>
                                                          <img
                                                          moz-do-not-send="true"
src="http://i61.tinypic.com/2s8k8z7.png"><br>
                                                          </div>
                                                          Joshua Huber<br>
                                                          Software
                                                          Engineer</div>
                                                          <div>NCEP
                                                          Central
                                                          Operations/Software
                                                          Development
                                                          Branch</div>
                                                          5830
                                                          University
                                                          Research Court
                                                          #1145<br>
                                                          <div>College
                                                          Park, MD 20740<br>
                                                          <a
                                                          moz-do-not-send="true"
href="tel:301.683.3913" value="+13016833913" target="_blank">301.683.3913</a><br>
                                                          <br>
                                                          <br>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          </div>
                                                          <br>
                                                        </font></span></blockquote>
                                                  </div>
                                                  <br>
                                                  <br clear="all">
                                                  <br>
                                                </div>
                                              </div>
                                              <span>-- <br>
                                                <div>Christopher Juckins<br>
Meteorologist/Programmer<br>
                                                  NCEP Ocean Prediction
                                                  Center<br>
                                                  <a
                                                    moz-do-not-send="true"
class="moz-txt-link-abbreviated" href="http://www.opc.ncep.noaa.gov"><a class="moz-txt-link-abbreviated" href="http://www.opc.ncep.noaa.gov">www.opc.ncep.noaa.gov</a></a></div>
                                              </span></div>
                                          </blockquote>
                                        </div>
                                        <br>
                                        <br clear="all">
                                        <div><br>
                                        </div>
                                      </div>
                                    </div>
                                    -- <br>
                                    <div>
                                      <div dir="ltr">
                                        <div>
                                          <div dir="ltr"><font size="1">NCEP

                                              developers and customers
                                              visit the <a
                                                moz-do-not-send="true"
                                                href="https://vlab.ncep.noaa.gov/group/ncep-a2cp/home"
                                                target="_blank">VLAB
                                                AWIPS II NCEP Community</a> </font></div>
                                          <div><font size="1">(newcomers
                                              will need to be added)</font></div>
                                          <div><font size="1"><br>
                                            </font></div>
                                          <div dir="ltr"><font size="1">W.

                                              David Plummer<br>
                                              National Centers AWIPS
                                              Team Lead </font></div>
                                          <div dir="ltr">
                                            <div><font size="1">NCEP
                                                Central Operations /
                                                Systems Integration
                                                Branch </font></div>
                                            <div>
                                              <div><span
                                                  style="font-size:x-small"><a
moz-do-not-send="true" href="tel:%28301%29%C2%A0683%E2%80%903917"
                                                    value="+13016833917"
                                                    target="_blank">(301) 683‐3917</a></span> <font
                                                  size="1"><br>
                                                  <br>
                                                  <img
                                                    moz-do-not-send="true"
src="http://www.nco.ncep.noaa.gov/images/ncep_logo.gif" height="55"
                                                    width="96"><br>
                                                  <br>
                                                  Department of Commerce</font></div>
                                              <div><font size="1">National

                                                  Oceanic and
                                                  Atmospheric
                                                  Administration<br>
                                                </font>
                                                <div><font size="1"><b>NCWCP

                                                      (</b></font><span
style="font-size:x-small">W/NP1)</span></div>
                                                <div><font size="1">5830
                                                    University Research
                                                    Court, 1150</font></div>
                                                <div><font size="1">College

                                                    Park, MD  20740-3818</font></div>
                                                <div><font size="1"><br>
                                                  </font><br>
                                                </div>
                                              </div>
                                            </div>
                                          </div>
                                        </div>
                                      </div>
                                    </div>
                                  </div>
                                </blockquote>
                              </div>
                              <br>
                              <br clear="all">
                              <br>
                              -- <br>
                              <div>Christopher Juckins<br>
                                Meteorologist/Programmer<br>
                                NCEP Ocean Prediction Center<br>
                                <a moz-do-not-send="true"
                                  href="http://www.opc.ncep.noaa.gov"
                                  target="_blank">www.opc.ncep.noaa.gov</a></div>
                            </div>
                          </div>
                        </div>
                      </blockquote>
                    </div>
                    <br>
                  </div>
                </div>
              </div>
            </blockquote>
          </div>
          <br>
        </div>
      </blockquote>
      <br>
    </blockquote>
    <br>
  </body>
</html>