<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Dusan, Sam,<br>
      <br>
      To generalize the answer, the application driven modulfiles can be
      tracked by module variables  ($LOADEDMODULES, and $MODULE PATH)
      and use entire dictionary of module operations.. That's is  not
      only unloaded or switched between applications env., but you can
      conduct environment analysis and make decisions on the fly: switch
      versions, steps of the workflow, data based on the status. You can
      see an example @ we made for the bufr use (different versions) on
      the module nceplibs trac page:<br>
      "<strong><em>How to Engage NCEPLIBS modules and Assign Yours New
          Variables?..."</em></strong><br>
      <a class="moz-txt-link-freetext" href="https://svnemc.ncep.noaa.gov/trac/nceplibs/wiki/Assign">https://svnemc.ncep.noaa.gov/trac/nceplibs/wiki/Assign</a><br>
      <br>
      On LMOD systems  (next level,  on Theia, Yellowstone, and most
      likely on the upcoming system) there are some new env. variables ,
      like:<br>
      $LMOD_DEFAULT_MODULEPATH which gives additional module "families"
      capability.<br>
      <br>
      However, it is all about convenience and best practice.<br>
      Of course, you can write anything in low level scripts. The the
      realities are we still have to use system, 3rd parties, nceplibs
      etc. modules.<br>
      <br>
      Cheers,<br>
      -Eugene<br>
         <br>
      <br>
      <br>
      <br>
      On 5/12/2017 4:04 PM, Dusan Jovic wrote:<br>
    </div>
    <blockquote type="cite"
      cite="mid:49f45006-ac49-b669-c3fd-bedb8a06b7a5@noaa.gov">Thanks,
      Sam.
      <br>
      <br>
      Unloading the modules is indeed something you can't do with a
      simple text file containing "module load ..." lines.
      <br>
      <br>
      But I do not understand the reasons 2. and 3. File modules.nems is
      not supposed to be load-ed but sourced from either interactive
      shell (bash or tcsh) or from a shell script, which can be either
      *sh or *csh. Why would anyone "source /etc/profile" from
      module.nems file, which only contains simple commands. Interactive
      shell or a shell script must be properly configured to load
      modules, which is also the case with modulefies and that's why you
      have module-setup scripts.
      <br>
      <br>
      Anyway, thank you again for the explanation.
      <br>
      <br>
      Dusan
      <br>
      <br>
      On 05/12/2017 03:10 PM, Samuel Trahan - NOAA Affiliate wrote:
      <br>
      <blockquote type="cite">The reasons for moving to a modulefile:
        <br>
        <br>
        1. You can unload the modulefile.
        <br>
        <br>
        2. The same file works in all shells.  Previously, if you used a
        tcsh login, you could not load modules.nems.  This was a major
        problem for some users.
        <br>
        <br>
        3. It prevents adding "source /etc/profile" to the modulefile,
        which causes problems if /etc/profile has already been sourced.
        <br>
        <br>
        There is now a pair of shell include files,
        NEMS/src/conf/module-setup.*sh.inc, which know how to obtain the
        module command and purge modules on seven clusters and five
        shells.  We also have a test system for it in
        NEMS/tests/modulecmd/.  That replaces "source /etc/profile" and
        "module purge."
        <br>
        <br>
        Sincerely,
        <br>
        Sam Trahan
        <br>
        <br>
        On Fri, May 12, 2017 at 1:59 PM, Dusan Jovic
        &lt;<a class="moz-txt-link-abbreviated" href="mailto:dusan.jovic@noaa.gov">dusan.jovic@noaa.gov</a> <a class="moz-txt-link-rfc2396E" href="mailto:dusan.jovic@noaa.gov">&lt;mailto:dusan.jovic@noaa.gov&gt;</a>&gt;
        wrote:
        <br>
        <br>
            Can some of you 'module' experts please explain why is
        writing
        <br>
            real modulefiles (a file that starts with #%Module) and then
        <br>
            loading that file with 'module load' better then writing a
        text
        <br>
            file with just a bunch of module load commands which is then
        <br>
            "sourced" from a build or run scripts? We used to use the
        second
        <br>
            approach in "the old NEMS" for many years and never had any
        problem.
        <br>
        <br>
            What are the advantages and are there any disadvantages?
        <br>
        <br>
            Thanks,
        <br>
        <br>
            Dusan
        <br>
        <br>
        <br>
            On 05/12/2017 12:40 PM, Jun Wang - NOAA Affiliate wrote:
        <br>
        <blockquote type="cite">    Jim,
          <br>
          <br>
              Sam is working on that, we hope to make it to this
          release. Thanks.
          <br>
          <br>
              Jun
          <br>
          <br>
              On Fri, May 12, 2017 at 12:37 PM, James Rosinski - NOAA
          Affiliate
          <br>
              &lt;<a class="moz-txt-link-abbreviated" href="mailto:james.rosinski@noaa.gov">james.rosinski@noaa.gov</a>
          <a class="moz-txt-link-rfc2396E" href="mailto:james.rosinski@noaa.gov">&lt;mailto:james.rosinski@noaa.gov&gt;</a>&gt; wrote:
          <br>
          <br>
                  Jun and others;
          <br>
          <br>
                  I noticed there is a build capability for fv3gfs on
          jet, but
          <br>
                  not any run instructions or runjob_jet.sh for jet. Is
          jet run
          <br>
                  capability intended, or is that for later?
          <br>
          <br>
                  Regards,
          <br>
                  Jim Rosinski
          <br>
          <br>
          <br>
          <br>
                  On Fri, May 12, 2017 at 10:16 AM, Rusty Benson - NOAA
          Federal
          <br>
                  &lt;<a class="moz-txt-link-abbreviated" href="mailto:rusty.benson@noaa.gov">rusty.benson@noaa.gov</a>
          <a class="moz-txt-link-rfc2396E" href="mailto:rusty.benson@noaa.gov">&lt;mailto:rusty.benson@noaa.gov&gt;</a>&gt; wrote:
          <br>
          <br>
                      Eugene,
          <br>
          <br>
                      Responding only to point 4, what options do you
          see that
          <br>
                      are Cray compilation flags?
          <br>
          <br>
                      Rusty
          <br>
                      --
          <br>
                      Rusty Benson, PhD
          <br>
                      Modeling Systems Group
          <br>
                      NOAA Geophysical Fluid Dynamics Lab
          <br>
                      Princeton, NJ
          <br>
          <br>
                      On Fri, May 12, 2017 at 11:58 AM, Eugene Mirvis
          <br>
                      &lt;<a class="moz-txt-link-abbreviated" href="mailto:eugene.mirvis@noaa.gov">eugene.mirvis@noaa.gov</a>
          <a class="moz-txt-link-rfc2396E" href="mailto:eugene.mirvis@noaa.gov">&lt;mailto:eugene.mirvis@noaa.gov&gt;</a>&gt;
          <br>
                      wrote:
          <br>
          <br>
                          Gerard,
          <br>
          <br>
                          Just several points to clarify.
          <br>
          <br>
                          1. NEMS practice  to call modulefiles what is
          <br>
                          actually the scripts (calling module
          commands),
          <br>
                          that require mostly bash env in order to
          source and
          <br>
                          keep environment was always non standard use.
          <br>
          <br>
                          Therefore,  if the developers will take Sam's
          advice
          <br>
                          and make real modulefiles (starting with
          #%Module)
          <br>
                          from the script, "export" and other scripting
          works
          <br>
                          wouldn't make sense, while Module util
          commands and
          <br>
                          Tcl/Tk  will work.
          <br>
          <br>
                          2. There is another dilemma - to keep needed
          <br>
                          environment and change within a workflow env.
          change.
          <br>
                          btw, *
          <br>
                          *unsetenv, append-path, prepend-path and
          remove-path
          <br>
                          module commands are very useful for
          controlling that,
          <br>
                          but you have to keep
          <br>
                          $LOADEDMODULES, and $MODULE PATH in consistent
          order.
          <br>
          <br>
                          3. Speaking of which,
          <br>
                          module purge
          <br>
                          and
          <br>
                          module switch
          <br>
                          are very useful to unload application driven
          modules.
          <br>
                          a/ You just have to do that not any moment,
          but
          <br>
                          before apps module is loaded, then, unload
          <br>
                          &lt;appsModulefile&gt; - not purge, but
          unload.
          <br>
                          b/ On Crays, the are some internal
          dependencies
          <br>
                          inside of PrgEnv. So you have to keep all
          "module use
          <br>
                          &lt;knowns&gt;" to recover, otherwise you
          might find
          <br>
                          "module not found"
          <br>
          <br>
                          4.
          <br>
                          Compiling on Theia, I'm just wondering why
          Cray's
          <br>
                          compilation flags are utilized... allover:
          <br>
                          See for instance
          <br>
                          ...
          <br>
                          *mpiifort -I/apps/netcdf/4.3.0-intel/include
          <br>
                          -fno-alias -auto -safe-cray-ptr -ftz -assume
          byterecl
          <br>
                          -nowarn -sox -align array64byte -i4 -real-size
          64
          <br>
                          -no-prec-div -no-prec-sqrt
          <br>
                          -xCORE-AVX2**-qno-opt-dynamic-align* -O2
          -debug
          <br>
                          minimal -fp-model source -qoverride-limits
          <br>
                          -qopt-prefetch=3 -qopenmp
          <br>
                         
          -I/apps/esmf/7.0.0/intel/intelmpi/mod/modO/Linux.intel.64.in
          <br>
                          <a class="moz-txt-link-rfc2396E" href="http://Linux.intel.64.in">&lt;http://Linux.intel.64.in&gt;</a>telmpi.default
          <br>
                          -I/apps/esmf/7.0.0/intel/intelmpi/include
          <br>
                          -I/apps/netcdf/4.3.0-intel/include -IENS_Cpl
          -I.
          <br>
                         
-I/scratch4/NCEPDEV/global/noscrub/Eugene.Mirvis/fv3gfs.v0beta/FV3/nems_dir<br>
                          -c module_MEDIATOR_methods.f90
          <br>
                          ...
          <br>
          <br>
                          Thanks,
          <br>
                          -Eugene
          <br>
                          On 5/12/2017 3:27 AM, Samuel Trahan - NOAA
          Affiliate
          <br>
                          wrote:
          <br>
          <blockquote type="cite">                Gerard, The "module
            load module.free-nctools" is
            <br>
                            failing because that file contains commands
            that are
            <br>
                            not valid in a modulefile. You can have as
            many
            <br>
                            "module" commands as you want in a
            modulefile, but
            <br>
                            you cannot have bash code. Also, it needs to
            start
            <br>
                            with #%Module export A=B # bad setenv A B
            <br>
                            prepend-path A (something) No "source
            /etc/profile"
            <br>
                            No "source /.../init/bash" No "module purge"
            because
            <br>
                            that causes infinite loops on some
            platforms. (The
            <br>
                            module is trying to unload itself.)
            Obtaining the
            <br>
                            "module" command and purging modules has to
            be done
            <br>
                            before loading the module. We have a pair of
            scripts
            <br>
                            (one csh, one sh) that do that on all NOAA
            machines.
            <br>
                            They are aware of csh, tcsh, ksh, bash, and
            sh; and
            <br>
                            run the correct initialization for each. The
            master
            <br>
                            copies are in
            NEMS/src/conf/module-setup.*sh.inc
            <br>
                            (csh and sh versions). Those scripts work on
            all
            <br>
                            NOAA machines and have a test suite inside
            <br>
                            NEMS/tests. man 4 modulefile
            <br>
                           
            <a class="moz-txt-link-freetext" href="http://modules.sourceforge.net/c/modulefile.html">http://modules.sourceforge.net/c/modulefile.html</a>
            <br>
                           
            <a class="moz-txt-link-rfc2396E" href="http://modules.sourceforge.net/c/modulefile.html">&lt;http://modules.sourceforge.net/c/modulefile.html&gt;</a>
            <br>
            <br>
            <br>
                            Sincerely,
            <br>
                            Sam Trahan
            <br>
            <br>
                            On Thu, May 11, 2017 at 5:01 PM, Gerard
            Ketefian - NOAA Affiliate &lt;
            <br>
                            <a class="moz-txt-link-abbreviated" href="mailto:gerard.ketefian@noaa.gov">gerard.ketefian@noaa.gov</a>
            <br>
                            <a class="moz-txt-link-rfc2396E" href="mailto:gerard.ketefian@noaa.gov">&lt;mailto:gerard.ketefian@noaa.gov&gt;</a>&gt;
            wrote:
            <br>
            <br>
            <blockquote type="cite">                Hi Jun,
              <br>
              <br>
                              Yes, after the missing "cp" fix, I (and
              Jim too) still had a problem with
              <br>
                              this line in runjob_theia.sh:
              <br>
              <br>
                              module load module.fre-nctools
              <br>
              <br>
                              This syntax (where you specify a file name
              after "module load") is not
              <br>
                              supported in our environment.  So what we
              did was first change the above
              <br>
                              line to:
              <br>
              <br>
                              . ./module.fre-nctools
              <br>
              <br>
                              to just source the module.fre-nctools
              file.  Also, we changed the contents
              <br>
                              of the file
${BASE_DIR}/fv3gfs.v0beta/release/v0/modulefiles/fv3gfs/fre-nctools.theia<br>
                              to:
              <br>
              <br>
                                  module load impi/5.1.2.150
              <a class="moz-txt-link-rfc2396E" href="http://5.1.2.150">&lt;http://5.1.2.150&gt;</a>
              <br>
                                  module load netcdf/4.3.0
              <br>
                                  module load hdf5/1.8.14
              <br>
                                  export HDF5_DIR=$HDF5
              <br>
                                  export NETCDF_DIR=$NETCDF
              <br>
                                  export
              LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
              <br>
              <br>
                              (Btw, should LIBRARY_PATH be changed to
              LD_LIBRARY_PATH in the last line?)
              <br>
                                With this, the remapping worked without
              any error messages.  When I
              <br>
                              compared the output with the baseline run,
              all the files were the same
              <br>
                              EXCEPT the three 1deg files from the
              remapping.  I don't know why those
              <br>
                              were differnt.  They weren't exactly the
              same but had almost the same
              <br>
                              sizes.
              <br>
              <br>
                              It would be nice to be able to use the
              syntax
              <br>
              <br>
                              module load module.fre-nctools
              <br>
              <br>
                              in our environment as well.  When I do a
              "module help", I don't see this
              <br>
                              syntax as an option (where you're
              specifying a file name that contains
              <br>
                              several commands).  Is there a way to get
              this to work on theia?
              <br>
              <br>
                              Thanks,
              <br>
                              Gerard
              <br>
              <br>
              <br>
                              On Thu, May 11, 2017 at 7:00 AM, Jun Wang
              - NOAA Affiliate &lt;
              <br>
                              <a class="moz-txt-link-abbreviated" href="mailto:jun.wang@noaa.gov">jun.wang@noaa.gov</a>
              <a class="moz-txt-link-rfc2396E" href="mailto:jun.wang@noaa.gov">&lt;mailto:jun.wang@noaa.gov&gt;</a>&gt; wrote:
              <br>
              <br>
              <blockquote type="cite">                Gerard,
                <br>
                <br>
                                With the fix last night, do you still
                have problem load the module
                <br>
                                module.fre-nctools?
                <br>
                <br>
                                Thanks for pointing out the forecast
                time specified in runjob_theia.sh, I
                <br>
                                changed it to be consistent with the
                baseline.
                <br>
                <br>
                                Another fix on the forecast executable
                is Jim Abeles are also committed
                <br>
                                to the tag.
                <br>
                <br>
                                I am planning to commit all the changes
                made to the temporary beta tag to
                <br>
                                trunk later today for Sam to add jet
                extension, please send me any
                <br>
                                fix/suggestion. Thanks to all who are
                doing testing.
                <br>
                <br>
                <br>
                                Jun
                <br>
                <br>
                                On Thu, May 11, 2017 at 5:03 AM, Gerard
                Ketefian - NOAA Affiliate &lt;
                <br>
                                <a class="moz-txt-link-abbreviated" href="mailto:gerard.ketefian@noaa.gov">gerard.ketefian@noaa.gov</a>
                <br>
                               
                <a class="moz-txt-link-rfc2396E" href="mailto:gerard.ketefian@noaa.gov">&lt;mailto:gerard.ketefian@noaa.gov&gt;</a>&gt; wrote:
                <br>
                <br>
                <blockquote type="cite">                Hi all,
                  <br>
                  <br>
                                  With Ligia's hints and Jun's last fix,
                  I was able to complete the run
                  <br>
                                  but not the remap.  I think the remap
                  fails because some modules don't get
                  <br>
                                  loaded properly.
                  <br>
                  <br>
                                  To get the remap to also work, I
                  replaced the following line in
                  <br>
                                  runjob_theia.sh
                  <br>
                  <br>
                                  module load module.fre-nctools
                  <br>
                  <br>
                                  with the following block (copied and
                  modified from the file
                  <br>
                                  module.fre-nctools):
                  <br>
                  <br>
                                  module load impi/5.1.2.150
                  <a class="moz-txt-link-rfc2396E" href="http://5.1.2.150">&lt;http://5.1.2.150&gt;</a>
                  <br>
                                  module load netcdf/4.3.0
                  <br>
                                  module load hdf5/1.8.14
                  <br>
                                  export HDF5_DIR=$HDF5
                  <br>
                                  export NETCDF_DIR=$NETCDF
                  <br>
                                  export
                  LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
                  <br>
                  <br>
                                  This change should allow the 1deg
                  remapped netcdf files to be generated.
                  <br>
                  <br>
                                  When I do the comparison of the sample
                  run's netcdf files with baseline,
                  <br>
                                  there is a about factor of 5
                  difference (the sample run files being
                  <br>
                                  larger).  This is because there are
                  only 8 output times in the baseline
                  <br>
                                  files but 40 in the run output.
                  <br>
                  <br>
                                  Gerard
                  <br>
                  <br>
                  <br>
                                  On Wed, May 10, 2017 at 9:56 PM, Jun
                  Wang - NOAA Affiliate &lt;
                  <br>
                                  <a class="moz-txt-link-abbreviated" href="mailto:jun.wang@noaa.gov">jun.wang@noaa.gov</a>
                  <a class="moz-txt-link-rfc2396E" href="mailto:jun.wang@noaa.gov">&lt;mailto:jun.wang@noaa.gov&gt;</a>&gt; wrote:
                  <br>
                  <br>
                  <blockquote type="cite">                Ligia,
                    <br>
                    <br>
                                    Thanks for the feedback.  The
                    suggestion on instruction is put in
                    <br>
                                    readme.txt. It is found that "cp "
                    is missing  in line 124 in
                    <br>
                                    runjob_theia.sh. I committed the
                    changes to the beta test tag:
                    <br>
                                   
                    <a class="moz-txt-link-freetext" href="https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t">https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t</a>
                    <br>
                                   
                    <a class="moz-txt-link-rfc2396E" href="https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t">&lt;https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t&gt;</a>
                    <br>
                                    ags/fv3gfs.v0beta
                    <br>
                    <br>
                                    Please check again. The results
                    differences will need further
                    <br>
                                    investigation. Thanks.
                    <br>
                    <br>
                                    Jun
                    <br>
                    <br>
                                    On Wed, May 10, 2017 at 10:28 PM,
                    Ligia Bernardet - NOAA Affiliate &lt;
                    <br>
                                    <a class="moz-txt-link-abbreviated" href="mailto:ligia.bernardet@noaa.gov">ligia.bernardet@noaa.gov</a>
                    <br>
                                   
                    <a class="moz-txt-link-rfc2396E" href="mailto:ligia.bernardet@noaa.gov">&lt;mailto:ligia.bernardet@noaa.gov&gt;</a>&gt; wrote:
                    <br>
                    <br>
                    <blockquote type="cite">                Folks,
                      <br>
                      <br>
                                      Here is some feedback
                      <br>
                      <br>
                      <br>
                                      *About the instructions*
                      <br>
                      <br>
                                          1. Minor typo. The word
                      "trunk" should be removed: Four executable
                      <br>
                                          files will be created under
                      fv3gfs.v0beta/*trunk*/NEMS/exe
                      <br>
                                          2. runjob_theia.sh:
                      Non-NCEPDEV people need to change directories
                      <br>
                                          DATA and ROTDIR to an area
                      they can write to
                      <br>
                                          3. diff_baseline.sh:
                      <br>
                                             1. It would be helpful to
                      tell users to add arguments
                      <br>
                                             to diff_baseline.sh to set
                      resolution and machine.
                      <br>
                                             2. Non-NCEPDEV people need
                      to change directory dir1 to location
                      <br>
                                             of their output
                      <br>
                      <br>
                                      *Outcome*
                      <br>
                                      It seems I was able to get through
                      the forecast but failed in remap.
                      <br>
                                      Problem seems related to loading
                      modules, I did not fully investigate yet.
                      <br>
                                      Output is in
                      /scratch4/BMC/gmtb/Ligia.Berna
                      <br>
                                      rdet/fv3gfs.v0beta/release/v0/exp
                      <br>
                      <br>
                                      When running diff, NetCDF files
                      differ from the baseline. I noticed
                      <br>
                                      the file sizes are different (mine
                      are larger than the baseline).
                      <br>
                      <br>
                                      Ligia
                      <br>
                      <br>
                                     
                      /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
                      <br>
                                     
                      exp/../modulefiles/fv3gfs/fre-nctools.theia
                      module.fre-nctools
                      <br>
                      <br>
                                     
                      /var/spool/torque/mom_priv/jobs/23566207.bqs3.SC
                      <a class="moz-txt-link-rfc2396E" href="http://23566207.bqs3.SC">&lt;http://23566207.bqs3.SC&gt;</a>: line 126:
                      <br>
                                     
                      /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
                      <br>
                                     
                      exp/../modulefiles/fv3gfs/fre-nctools.theia:
                      *Permission denied*
                      <br>
                      <br>
                                      + module load module.fre-nctools
                      <br>
                      <br>
                                      ++ /apps/lmod/lmod/libexec/lmod
                      bash load module.fre-nctools
                      <br>
                      <br>
                                      Lmod has detected the following
                      error: The following module(s) are
                      <br>
                                      unknown:
                      <br>
                      <br>
                                      "module.fre-nctools"
                      <br>
                      <br>
                                      On Wed, May 10, 2017 at 5:47 PM,
                      James Rosinski - NOAA Affiliate &lt;
                      <br>
                                      <a class="moz-txt-link-abbreviated" href="mailto:james.rosinski@noaa.gov">james.rosinski@noaa.gov</a>
                      <br>
                                     
                      <a class="moz-txt-link-rfc2396E" href="mailto:james.rosinski@noaa.gov">&lt;mailto:james.rosinski@noaa.gov&gt;</a>&gt; wrote:
                      <br>
                      <br>
                      <blockquote type="cite">                Hi Jun;
                        <br>
                        <br>
                                        I am about to head home for the
                        day, but here are my comments so far,
                        <br>
                                        after following the instructions
                        for theia:
                        <br>
                        <br>
                                        o The builds of models and remap
                        codes completed successfully. One
                        <br>
                                        suggestion might be to have the
                        user specify 32 or 64-bit, and nh vs. hydro
                        <br>
                                        in order to cut down compilation
                        time by a factor of 4.
                        <br>
                        <br>
                                        o The batch job attempting to
                        run the model failed. Relevant lines in
                        <br>
                                        err_theia are:
                        <br>
                        <br>
                        <br>
                        <br>
                        <br>
                        <br>
                        <br>
                        <br>
                        <br>
                                        *+ cd
                        /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900+
                        <br>
                                        /bin/cp -p
                        <br>
                                       
/scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp/../../../NEMS/exe/fv3_gfs_nh.prod.32bit.x<br>
                                       
                        /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900/.+
                        -prepend-rank
                        <br>
                                        -np 288
                        ./fv3_gfs_nh.prod.32bit.x+ ERR=127+ export ERR+
                        err=127*
                        <br>
                                        Looks like somehow "mpirun" was
                        not found (note there is nothing in
                        <br>
                                        front of "-prepend-rank"). FYI I
                        use csh for my login shell--not sure if
                        <br>
                                        this is behind the error. I had
                        no modules loaded when submitting the job.
                        <br>
                        <br>
                                        If  you'd like to examine the
                        output you should have read access to
                        <br>
                                        it here on theia:
                        <br>
                        <br>
                                       
                        /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp
                        <br>
                        <br>
                                        More info tomorrow...
                        <br>
                        <br>
                                        Regards,
                        <br>
                                        Jim Rosinski
                        <br>
                        <br>
                        <br>
                                        On Wed, May 10, 2017 at 2:55 PM,
                        Jun Wang - NOAA Affiliate &lt;
                        <br>
                                        <a class="moz-txt-link-abbreviated" href="mailto:jun.wang@noaa.gov">jun.wang@noaa.gov</a>
                        <a class="moz-txt-link-rfc2396E" href="mailto:jun.wang@noaa.gov">&lt;mailto:jun.wang@noaa.gov&gt;</a>&gt; wrote:
                        <br>
                        <br>
                        <blockquote type="cite">                Dear
                          all,
                          <br>
                          <br>
                                          I was noticed that some
                          directory names in the readme.txt are not
                          <br>
                                          correct. I just updated the
                          tag version, please let me know if you have
                          any
                          <br>
                                          further questions. Thanks.
                          <br>
                          <br>
                                          Jun
                          <br>
                          <br>
                                          On Wed, May 10, 2017 at 4:39
                          PM, Jun Wang - NOAA Affiliate &lt;
                          <br>
                                          <a class="moz-txt-link-abbreviated" href="mailto:jun.wang@noaa.gov">jun.wang@noaa.gov</a>
                          <a class="moz-txt-link-rfc2396E" href="mailto:jun.wang@noaa.gov">&lt;mailto:jun.wang@noaa.gov&gt;</a>&gt; wrote:
                          <br>
                          <br>
                          <blockquote type="cite">                Rusty,
                            <br>
                            <br>
                                            Thanks for quick feedback.
                            Today we just had a VLAB meeting on how
                            <br>
                                            to provide information for
                            the public release. Vijay mentioned that EMC
                            <br>
                                            will be setting up an FV3GFS
                            community web page through VLAB, some basic
                            <br>
                                            document will be provided
                            there. A formal instruction on how to get
                            release
                            <br>
                                            code, and to compile and run
                            experiment will be on that web page too. For
                            <br>
                                            questions/feedback,  a forum
                            will be set up for users to post questions
                            and
                            <br>
                                            to provide answers
                            &amp;feedback, the purpose is that all the
                            developers will
                            <br>
                                            see the questions/answers,
                            it is suggested not to send
                            questions/feedback
                            <br>
                                            to any individual's personal
                            email (If people receive questions from
                            users,
                            <br>
                                            we suggest that they post
                            the questions along with their answers to
                            the
                            <br>
                                            forum). The readme.txt is a
                            temporary solution to get the testing
                            started,
                            <br>
                                            it may be changed in the
                            final release.
                            <br>
                            <br>
                                            Kate Howard
                            (<a class="moz-txt-link-abbreviated" href="mailto:kate.howard@noaa.gov">kate.howard@noaa.gov</a>
                            <br>
                                           
                            <a class="moz-txt-link-rfc2396E" href="mailto:kate.howard@noaa.gov">&lt;mailto:kate.howard@noaa.gov&gt;</a>) is
                            working on the VLAB fv3gfs
                            <br>
                                            web page, she can add the
                            gfdl fv3gfs support email on the web page
                            too, if
                            <br>
                                            you have any fv3 document
                            for general developers, please send to her.
                            <br>
                            <br>
                                            Thanks.
                            <br>
                            <br>
                            <br>
                                            Jun
                            <br>
                            <br>
                                            On Wed, May 10, 2017 at 4:02
                            PM, Rusty Benson - NOAA Federal &lt;
                            <br>
                                            <a class="moz-txt-link-abbreviated" href="mailto:rusty.benson@noaa.gov">rusty.benson@noaa.gov</a>
                            <br>
                                           
                            <a class="moz-txt-link-rfc2396E" href="mailto:rusty.benson@noaa.gov">&lt;mailto:rusty.benson@noaa.gov&gt;</a>&gt;
                            wrote:
                            <br>
                            <br>
                            <blockquote type="cite">                Hi
                              Jun and Vijay,
                              <br>
                              <br>
                                              In the readme.txt Q&amp;A,
                              you mention where to get help.  Has there
                              <br>
                                              been any thought to
                              putting together a single email for
                              tracking all
                              <br>
                                              questions/requests that
                              can be used as a basis for creating a
                              knowledgebase
                              <br>
                                              via a wiki or other
                              forum?  By segmenting FV3 and physics
                              support, I think
                              <br>
                                              we are missing an
                              opportunity for personnel to get exposure
                              to and learn
                              <br>
                                              about system pieces for
                              which they may not necessarily be
                              responsible.
                              <br>
                              <br>
                                              If we do go the route of a
                              single support email, we have an
                              <br>
                                              existing email for FV3
                              support which could be used as an alias
                              member.
                              <br>
                                              Otherwise, we would want
                              to publish the email inside of the
                              readme.txt and
                              <br>
                                              not have individual team
                              members being contacted directly
                              <br>
                              <br>
                                             
                              <a class="moz-txt-link-rfc2396E" href="mailto:oar.gfdl.fvgfs_support@noaa.gov">&lt;oar.gfdl.fvgfs_support@noaa.gov&gt;</a>
                              <br>
                                             
                              <a class="moz-txt-link-rfc2396E" href="mailto:oar.gfdl.fvgfs_support@noaa.gov">&lt;mailto:oar.gfdl.fvgfs_support@noaa.gov&gt;</a>
                              <br>
                              <br>
                              <br>
                              <br>
                                              Rusty
                              <br>
                                              --
                              <br>
                                              Rusty Benson, PhD
                              <br>
                                              Modeling Systems Group
                              <br>
                                              NOAA Geophysical Fluid
                              Dynamics Lab
                              <br>
                                              Princeton, NJ
                              <br>
                              <br>
                                              On Wed, May 10, 2017 at
                              2:08 PM, Jun Wang - NOAA Affiliate &lt;
                              <br>
                                              <a class="moz-txt-link-abbreviated" href="mailto:jun.wang@noaa.gov">jun.wang@noaa.gov</a>
                              <a class="moz-txt-link-rfc2396E" href="mailto:jun.wang@noaa.gov">&lt;mailto:jun.wang@noaa.gov&gt;</a>&gt;
                              wrote:
                              <br>
                              <br>
                              <blockquote type="cite">               
                                Dear all,
                                <br>
                                <br>
                                                The following email is
                                for people who are willing to do beta
                                <br>
                                                testing for the fv3gfs
                                May 15 release. Please ignore the email
                                if you are
                                <br>
                                                not going to run the
                                test.
                                <br>
                                <br>
                                                The svn tag for beta
                                testing is located at:
                                <br>
                                <br>
                                                 
                                <a class="moz-txt-link-freetext" href="https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/">https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/</a>
                                <br>
                                               
                                <a class="moz-txt-link-rfc2396E" href="https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/">&lt;https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/&gt;</a>
                                <br>
                                                tags/fv3gfs.v0beta
                                <br>
                                <br>
                                                The instruction file on
                                how to get and compile the code and to
                                <br>
                                                run an experiment is at:
                                <br>
                                <br>
                                               
                                <a class="moz-txt-link-freetext" href="https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t">https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t</a>
                                <br>
                                               
                                <a class="moz-txt-link-rfc2396E" href="https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t">&lt;https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t&gt;</a>
                                <br>
                                               
                                ags/fv3gfs.v0beta/release/v0/readme.txt
                                <br>
                                <br>
                                                Please follow the
                                instructions to see if you can run an
                                <br>
                                                experiment.
                                <br>
                                <br>
                                                  Thanks.
                                <br>
                                <br>
                                <br>
                                                Jun
                                <br>
                                <br>
                                <br>
                                <br>
                                               
                                _______________________________________________
                                <br>
                                                Ncep.list.fv3-announce
                                mailing list
                                <br>
                                               
                                <a class="moz-txt-link-abbreviated" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov</a>
                                <br>
                                               
                                <a class="moz-txt-link-rfc2396E" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">&lt;mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov&gt;</a>
                                <br>
                                               
                                <a class="moz-txt-link-freetext" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list</a>
                                <br>
                                               
                                <a class="moz-txt-link-rfc2396E" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list">&lt;https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list&gt;</a>.
                                <br>
                                                fv3-announce
                                <br>
                                <br>
                                <br>
                              </blockquote>
                            </blockquote>
                          </blockquote>
                                         
                          _______________________________________________
                          <br>
                                          Ncep.list.fv3-announce mailing
                          list
                          <br>
                                         
                          <a class="moz-txt-link-abbreviated" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov</a>
                          <br>
                                         
                          <a class="moz-txt-link-rfc2396E" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">&lt;mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov&gt;</a>
                          <br>
                                         
                          <a class="moz-txt-link-freetext" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list</a>
                          <br>
                                         
                          <a class="moz-txt-link-rfc2396E" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list">&lt;https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list&gt;</a>.
                          <br>
                                          fv3-announce
                          <br>
                          <br>
                          <br>
                        </blockquote>
                                       
                        _______________________________________________
                        <br>
                                        Ncep.list.fv3-announce mailing
                        list
                        <br>
                                       
                        <a class="moz-txt-link-abbreviated" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov</a>
                        <br>
                                       
                        <a class="moz-txt-link-rfc2396E" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">&lt;mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov&gt;</a>
                        <br>
                                       
                        <a class="moz-txt-link-freetext" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list</a>
                        <br>
                                       
                        <a class="moz-txt-link-rfc2396E" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list">&lt;https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list&gt;</a>.
                        <br>
                                        fv3-announce
                        <br>
                        <br>
                        <br>
                      </blockquote>
                    </blockquote>
                  </blockquote>
                                  --
                  <br>
                                  Gerard Ketefian
                  <br>
                                  Research Scientist
                  <br>
                                  NOAA/OAR/ESRL/GSD/EMB, R/GSD1
                  <br>
                                  325 Broadway
                  <br>
                                  Boulder, CO 80305
                  <br>
                                  phone:303-497-6209
                  &lt;tel:%28303%29%20497-6209&gt; 
                  &lt;(303)%20497-6209&gt;
                  <br>
                  <br>
                  <br>
                </blockquote>
              </blockquote>
                              --
              <br>
                              Gerard Ketefian
              <br>
                              Research Scientist
              <br>
                              NOAA/OAR/ESRL/GSD/EMB, R/GSD1
              <br>
                              325 Broadway
              <br>
                              Boulder, CO 80305
              <br>
                              phone:303-497-6209
              &lt;tel:%28303%29%20497-6209&gt;
              <br>
              <br>
              <br>
                             
              _______________________________________________
              <br>
                              Ncep.list.fv3-announce mailing list
              <br>
                             
              <a class="moz-txt-link-abbreviated" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov</a>
              <br>
                             
              <a class="moz-txt-link-rfc2396E" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">&lt;mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov&gt;</a>
              <br>
                             
              <a class="moz-txt-link-freetext" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce</a>
              <br>
                             
<a class="moz-txt-link-rfc2396E" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">&lt;https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce&gt;</a><br>
              <br>
              <br>
            </blockquote>
          </blockquote>
          <br>
                          --                 EUGENE MIRVIS, Tech Lead,
          <br>
                          Senior Computational Scientist, IMSG @
          <br>
                            Global Climate &amp; Weather Modeling Branch
          of
          <br>
                             NOAA/NCEP Environmental Modeling Center
          <br>
                                       NCWCP,  rm  2183
          <br>
                                5830 University Research Ct.
          <br>
                                   College Park, MD 20740
          <br>
                                       Ph.301.683.3809
          &lt;tel:%28301%29%20683-3809&gt;
          <br>
          <br>
          <br>
                         
          _______________________________________________
          <br>
                          Ncep.list.fv3-announce mailing list
          <br>
                          <a class="moz-txt-link-abbreviated" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov</a>
          <br>
                         
          <a class="moz-txt-link-rfc2396E" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">&lt;mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov&gt;</a>
          <br>
                         
          <a class="moz-txt-link-freetext" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce</a>
          <br>
                         
<a class="moz-txt-link-rfc2396E" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">&lt;https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce&gt;</a><br>
          <br>
          <br>
          <br>
                      _______________________________________________
          <br>
                      Ncep.list.fv3-announce mailing list
          <br>
                      <a class="moz-txt-link-abbreviated" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov</a>
          <br>
                     
          <a class="moz-txt-link-rfc2396E" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">&lt;mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov&gt;</a>
          <br>
                     
          <a class="moz-txt-link-freetext" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce</a>
          <br>
                     
<a class="moz-txt-link-rfc2396E" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">&lt;https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce&gt;</a><br>
          <br>
          <br>
          <br>
          <br>
          <br>
              _______________________________________________
          <br>
              Ncep.list.fv3-announce mailing list
          <br>
              <a class="moz-txt-link-abbreviated" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov</a>
          <br>
              <a class="moz-txt-link-rfc2396E" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">&lt;mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov&gt;</a>
          <br>
             
          <a class="moz-txt-link-freetext" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce</a>
          <br>
             
<a class="moz-txt-link-rfc2396E" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">&lt;https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce&gt;</a><br>
        </blockquote>
        <br>
        <br>
        <br>
            _______________________________________________
        <br>
            Ncep.list.fv3-announce mailing list
        <br>
            <a class="moz-txt-link-abbreviated" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov</a>
        <br>
            <a class="moz-txt-link-rfc2396E" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">&lt;mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov&gt;</a>
        <br>
           
        <a class="moz-txt-link-freetext" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce</a>
        <br>
           
<a class="moz-txt-link-rfc2396E" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">&lt;https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce&gt;</a><br>
        <br>
        <br>
      </blockquote>
      <br>
      <br>
      <br>
      <fieldset class="mimeAttachmentHeader"></fieldset>
      <br>
      <pre wrap="">_______________________________________________
Ncep.list.fv3-announce mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov</a>
<a class="moz-txt-link-freetext" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce">https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce</a>
</pre>
    </blockquote>
    <br>
  </body>
</html>