[Ncep.list.fv3-announce] fv3gfs release beta test
Samuel Trahan - NOAA Affiliate
samuel.trahan at noaa.gov
Fri May 12 20:34:46 UTC 2017
Dusan,
The "source /etc/profile" was in the modules.nems because some of the
scripts that ran modules.nems did not already have the modulecmd
available. Also, the "module purge" command does not always work
properly. Due to this, there were some tweaks in modules.nems and the
environment_*() functions on platform-by-platform basis. Splitting the
system into module-setup.*sh.inc and modulefiles lets us work with all use
cases in a single modulefile.
Sincerely,
Sam Trahan
On Fri, May 12, 2017 at 4:04 PM, Dusan Jovic <dusan.jovic at noaa.gov> wrote:
> Thanks, Sam.
>
> Unloading the modules is indeed something you can't do with a simple text
> file containing "module load ..." lines.
>
> But I do not understand the reasons 2. and 3. File modules.nems is not
> supposed to be load-ed but sourced from either interactive shell (bash or
> tcsh) or from a shell script, which can be either *sh or *csh. Why would
> anyone "source /etc/profile" from module.nems file, which only contains
> simple commands. Interactive shell or a shell script must be properly
> configured to load modules, which is also the case with modulefies and
> that's why you have module-setup scripts.
>
> Anyway, thank you again for the explanation.
>
> Dusan
>
>
> On 05/12/2017 03:10 PM, Samuel Trahan - NOAA Affiliate wrote:
>
> The reasons for moving to a modulefile:
>
> 1. You can unload the modulefile.
>
> 2. The same file works in all shells. Previously, if you used a tcsh
> login, you could not load modules.nems. This was a major problem for some
> users.
>
> 3. It prevents adding "source /etc/profile" to the modulefile, which
> causes problems if /etc/profile has already been sourced.
>
> There is now a pair of shell include files, NEMS/src/conf/module-setup.*sh.inc,
> which know how to obtain the module command and purge modules on seven
> clusters and five shells. We also have a test system for it in
> NEMS/tests/modulecmd/. That replaces "source /etc/profile" and "module
> purge."
>
> Sincerely,
> Sam Trahan
>
> On Fri, May 12, 2017 at 1:59 PM, Dusan Jovic <dusan.jovic at noaa.gov> wrote:
>
>> Can some of you 'module' experts please explain why is writing real
>> modulefiles (a file that starts with #%Module) and then loading that file
>> with 'module load' better then writing a text file with just a bunch of
>> module load commands which is then "sourced" from a build or run scripts?
>> We used to use the second approach in "the old NEMS" for many years and
>> never had any problem.
>>
>> What are the advantages and are there any disadvantages?
>>
>> Thanks,
>>
>> Dusan
>>
>> On 05/12/2017 12:40 PM, Jun Wang - NOAA Affiliate wrote:
>>
>> Jim,
>>
>> Sam is working on that, we hope to make it to this release. Thanks.
>>
>> Jun
>>
>> On Fri, May 12, 2017 at 12:37 PM, James Rosinski - NOAA Affiliate <
>> james.rosinski at noaa.gov> wrote:
>>
>>> Jun and others;
>>>
>>> I noticed there is a build capability for fv3gfs on jet, but not any run
>>> instructions or runjob_jet.sh for jet. Is jet run capability intended, or
>>> is that for later?
>>>
>>> Regards,
>>> Jim Rosinski
>>>
>>>
>>>
>>> On Fri, May 12, 2017 at 10:16 AM, Rusty Benson - NOAA Federal <
>>> rusty.benson at noaa.gov> wrote:
>>>
>>>> Eugene,
>>>>
>>>> Responding only to point 4, what options do you see that are Cray
>>>> compilation flags?
>>>>
>>>> Rusty
>>>> --
>>>> Rusty Benson, PhD
>>>> Modeling Systems Group
>>>> NOAA Geophysical Fluid Dynamics Lab
>>>> Princeton, NJ
>>>>
>>>> On Fri, May 12, 2017 at 11:58 AM, Eugene Mirvis <eugene.mirvis at noaa.gov
>>>> > wrote:
>>>>
>>>>> Gerard,
>>>>>
>>>>> Just several points to clarify.
>>>>>
>>>>> 1. NEMS practice to call modulefiles what is actually the scripts
>>>>> (calling module commands),
>>>>> that require mostly bash env in order to source and keep environment
>>>>> was always non standard use.
>>>>>
>>>>> Therefore, if the developers will take Sam's advice and make real
>>>>> modulefiles (starting with #%Module) from the script, "export" and other
>>>>> scripting works
>>>>> wouldn't make sense, while Module util commands and Tcl/Tk will work.
>>>>>
>>>>> 2. There is another dilemma - to keep needed environment and change
>>>>> within a workflow env. change.
>>>>> btw,
>>>>> unsetenv, append-path, prepend-path and remove-path module commands
>>>>> are very useful for controlling that, but you have to keep
>>>>> $LOADEDMODULES, and $MODULE PATH in consistent order.
>>>>>
>>>>> 3. Speaking of which,
>>>>> module purge
>>>>> and
>>>>> module switch
>>>>> are very useful to unload application driven modules.
>>>>> a/ You just have to do that not any moment, but before apps module is
>>>>> loaded, then, unload <appsModulefile> - not purge, but unload.
>>>>> b/ On Crays, the are some internal dependencies inside of PrgEnv. So
>>>>> you have to keep all "module use <knowns>" to recover, otherwise you might
>>>>> find "module not found"
>>>>>
>>>>> 4.
>>>>> Compiling on Theia, I'm just wondering why Cray's compilation flags
>>>>> are utilized... allover:
>>>>> See for instance
>>>>> ...
>>>>> *mpiifort -I/apps/netcdf/4.3.0-intel/include -fno-alias -auto
>>>>> -safe-cray-ptr -ftz -assume byterecl -nowarn -sox -align array64byte -i4
>>>>> -real-size 64 -no-prec-div -no-prec-sqrt -xCORE-AVX2**
>>>>> -qno-opt-dynamic-align* -O2 -debug minimal -fp-model source
>>>>> -qoverride-limits -qopt-prefetch=3 -qopenmp -I/apps/esmf/7.0.0/intel/intel
>>>>> mpi/mod/modO/Linux.intel.64.intelmpi.default
>>>>> -I/apps/esmf/7.0.0/intel/intelmpi/include
>>>>> -I/apps/netcdf/4.3.0-intel/include -IENS_Cpl -I.
>>>>> -I/scratch4/NCEPDEV/global/noscrub/Eugene.Mirvis/fv3gfs.v0beta/FV3/nems_dir
>>>>> -c module_MEDIATOR_methods.f90
>>>>> ...
>>>>>
>>>>> Thanks,
>>>>> -Eugene
>>>>> On 5/12/2017 3:27 AM, Samuel Trahan - NOAA Affiliate wrote:
>>>>>
>>>>> Gerard,
>>>>>
>>>>> The "module load module.free-nctools" is failing because that file contains
>>>>> commands that are not valid in a modulefile. You can have as many "module"
>>>>> commands as you want in a modulefile, but you cannot have bash code. Also,
>>>>> it needs to start with #%Module
>>>>>
>>>>> export A=B # bad
>>>>> setenv A B
>>>>> prepend-path A (something)
>>>>>
>>>>> No "source /etc/profile"
>>>>> No "source /.../init/bash"
>>>>> No "module purge" because that causes infinite loops on some platforms.
>>>>> (The module is trying to unload itself.)
>>>>>
>>>>> Obtaining the "module" command and purging modules has to be done before
>>>>> loading the module. We have a pair of scripts (one csh, one sh) that do
>>>>> that on all NOAA machines. They are aware of csh, tcsh, ksh, bash, and sh;
>>>>> and run the correct initialization for each. The master copies are in
>>>>> NEMS/src/conf/module-setup.*sh.inc (csh and sh versions). Those scripts
>>>>> work on all NOAA machines and have a test suite inside NEMS/tests.
>>>>>
>>>>> man 4 modulefilehttp://modules.sourceforge.net/c/modulefile.html
>>>>>
>>>>>
>>>>> Sincerely,
>>>>> Sam Trahan
>>>>>
>>>>> On Thu, May 11, 2017 at 5:01 PM, Gerard Ketefian - NOAA Affiliate <gerard.ketefian at noaa.gov> wrote:
>>>>>
>>>>>
>>>>> Hi Jun,
>>>>>
>>>>> Yes, after the missing "cp" fix, I (and Jim too) still had a problem with
>>>>> this line in runjob_theia.sh:
>>>>>
>>>>> module load module.fre-nctools
>>>>>
>>>>> This syntax (where you specify a file name after "module load") is not
>>>>> supported in our environment. So what we did was first change the above
>>>>> line to:
>>>>>
>>>>> . ./module.fre-nctools
>>>>>
>>>>> to just source the module.fre-nctools file. Also, we changed the contents
>>>>> of the file ${BASE_DIR}/fv3gfs.v0beta/release/v0/modulefiles/fv3gfs/fre-nctools.theia
>>>>> to:
>>>>>
>>>>> module load impi/5.1.2.150
>>>>> module load netcdf/4.3.0
>>>>> module load hdf5/1.8.14
>>>>> export HDF5_DIR=$HDF5
>>>>> export NETCDF_DIR=$NETCDF
>>>>> export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
>>>>>
>>>>> (Btw, should LIBRARY_PATH be changed to LD_LIBRARY_PATH in the last line?)
>>>>> With this, the remapping worked without any error messages. When I
>>>>> compared the output with the baseline run, all the files were the same
>>>>> EXCEPT the three 1deg files from the remapping. I don't know why those
>>>>> were differnt. They weren't exactly the same but had almost the same
>>>>> sizes.
>>>>>
>>>>> It would be nice to be able to use the syntax
>>>>>
>>>>> module load module.fre-nctools
>>>>>
>>>>> in our environment as well. When I do a "module help", I don't see this
>>>>> syntax as an option (where you're specifying a file name that contains
>>>>> several commands). Is there a way to get this to work on theia?
>>>>>
>>>>> Thanks,
>>>>> Gerard
>>>>>
>>>>>
>>>>> On Thu, May 11, 2017 at 7:00 AM, Jun Wang - NOAA Affiliate <jun.wang at noaa.gov> wrote:
>>>>>
>>>>>
>>>>> Gerard,
>>>>>
>>>>> With the fix last night, do you still have problem load the module
>>>>> module.fre-nctools?
>>>>>
>>>>> Thanks for pointing out the forecast time specified in runjob_theia.sh, I
>>>>> changed it to be consistent with the baseline.
>>>>>
>>>>> Another fix on the forecast executable is Jim Abeles are also committed
>>>>> to the tag.
>>>>>
>>>>> I am planning to commit all the changes made to the temporary beta tag to
>>>>> trunk later today for Sam to add jet extension, please send me any
>>>>> fix/suggestion. Thanks to all who are doing testing.
>>>>>
>>>>>
>>>>> Jun
>>>>>
>>>>> On Thu, May 11, 2017 at 5:03 AM, Gerard Ketefian - NOAA Affiliate <gerard.ketefian at noaa.gov> wrote:
>>>>>
>>>>>
>>>>> Hi all,
>>>>>
>>>>> With Ligia's hints and Jun's last fix, I was able to complete the run
>>>>> but not the remap. I think the remap fails because some modules don't get
>>>>> loaded properly.
>>>>>
>>>>> To get the remap to also work, I replaced the following line in
>>>>> runjob_theia.sh
>>>>>
>>>>> module load module.fre-nctools
>>>>>
>>>>> with the following block (copied and modified from the file
>>>>> module.fre-nctools):
>>>>>
>>>>> module load impi/5.1.2.150
>>>>> module load netcdf/4.3.0
>>>>> module load hdf5/1.8.14
>>>>> export HDF5_DIR=$HDF5
>>>>> export NETCDF_DIR=$NETCDF
>>>>> export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
>>>>>
>>>>> This change should allow the 1deg remapped netcdf files to be generated.
>>>>>
>>>>> When I do the comparison of the sample run's netcdf files with baseline,
>>>>> there is a about factor of 5 difference (the sample run files being
>>>>> larger). This is because there are only 8 output times in the baseline
>>>>> files but 40 in the run output.
>>>>>
>>>>> Gerard
>>>>>
>>>>>
>>>>> On Wed, May 10, 2017 at 9:56 PM, Jun Wang - NOAA Affiliate <jun.wang at noaa.gov> wrote:
>>>>>
>>>>>
>>>>> Ligia,
>>>>>
>>>>> Thanks for the feedback. The suggestion on instruction is put in
>>>>> readme.txt. It is found that "cp " is missing in line 124 in
>>>>> runjob_theia.sh. I committed the changes to the beta test tag:https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t
>>>>> ags/fv3gfs.v0beta
>>>>>
>>>>> Please check again. The results differences will need further
>>>>> investigation. Thanks.
>>>>>
>>>>> Jun
>>>>>
>>>>> On Wed, May 10, 2017 at 10:28 PM, Ligia Bernardet - NOAA Affiliate <ligia.bernardet at noaa.gov> wrote:
>>>>>
>>>>>
>>>>> Folks,
>>>>>
>>>>> Here is some feedback
>>>>>
>>>>>
>>>>> *About the instructions*
>>>>>
>>>>> 1. Minor typo. The word "trunk" should be removed: Four executable
>>>>> files will be created under fv3gfs.v0beta/*trunk*/NEMS/exe
>>>>> 2. runjob_theia.sh: Non-NCEPDEV people need to change directories
>>>>> DATA and ROTDIR to an area they can write to
>>>>> 3. diff_baseline.sh:
>>>>> 1. It would be helpful to tell users to add arguments
>>>>> to diff_baseline.sh to set resolution and machine.
>>>>> 2. Non-NCEPDEV people need to change directory dir1 to location
>>>>> of their output
>>>>>
>>>>> *Outcome*
>>>>> It seems I was able to get through the forecast but failed in remap.
>>>>> Problem seems related to loading modules, I did not fully investigate yet.
>>>>> Output is in /scratch4/BMC/gmtb/Ligia.Berna
>>>>> rdet/fv3gfs.v0beta/release/v0/exp
>>>>>
>>>>> When running diff, NetCDF files differ from the baseline. I noticed
>>>>> the file sizes are different (mine are larger than the baseline).
>>>>>
>>>>> Ligia
>>>>>
>>>>> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
>>>>> exp/../modulefiles/fv3gfs/fre-nctools.theia module.fre-nctools
>>>>>
>>>>> /var/spool/torque/mom_priv/jobs/23566207.bqs3.SC: line 126:
>>>>> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
>>>>> exp/../modulefiles/fv3gfs/fre-nctools.theia: *Permission denied*
>>>>>
>>>>> + module load module.fre-nctools
>>>>>
>>>>> ++ /apps/lmod/lmod/libexec/lmod bash load module.fre-nctools
>>>>>
>>>>> Lmod has detected the following error: The following module(s) are
>>>>> unknown:
>>>>>
>>>>> "module.fre-nctools"
>>>>>
>>>>> On Wed, May 10, 2017 at 5:47 PM, James Rosinski - NOAA Affiliate <james.rosinski at noaa.gov> wrote:
>>>>>
>>>>>
>>>>> Hi Jun;
>>>>>
>>>>> I am about to head home for the day, but here are my comments so far,
>>>>> after following the instructions for theia:
>>>>>
>>>>> o The builds of models and remap codes completed successfully. One
>>>>> suggestion might be to have the user specify 32 or 64-bit, and nh vs. hydro
>>>>> in order to cut down compilation time by a factor of 4.
>>>>>
>>>>> o The batch job attempting to run the model failed. Relevant lines in
>>>>> err_theia are:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *+ cd /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900+
>>>>> /bin/cp -p
>>>>> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp/../../../NEMS/exe/fv3_gfs_nh.prod.32bit.x
>>>>> /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900/.+ -prepend-rank
>>>>> -np 288 ./fv3_gfs_nh.prod.32bit.x+ ERR=127+ export ERR+ err=127*
>>>>> Looks like somehow "mpirun" was not found (note there is nothing in
>>>>> front of "-prepend-rank"). FYI I use csh for my login shell--not sure if
>>>>> this is behind the error. I had no modules loaded when submitting the job.
>>>>>
>>>>> If you'd like to examine the output you should have read access to
>>>>> it here on theia:
>>>>>
>>>>> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp
>>>>>
>>>>> More info tomorrow...
>>>>>
>>>>> Regards,
>>>>> Jim Rosinski
>>>>>
>>>>>
>>>>> On Wed, May 10, 2017 at 2:55 PM, Jun Wang - NOAA Affiliate <jun.wang at noaa.gov> wrote:
>>>>>
>>>>>
>>>>> Dear all,
>>>>>
>>>>> I was noticed that some directory names in the readme.txt are not
>>>>> correct. I just updated the tag version, please let me know if you have any
>>>>> further questions. Thanks.
>>>>>
>>>>> Jun
>>>>>
>>>>> On Wed, May 10, 2017 at 4:39 PM, Jun Wang - NOAA Affiliate <jun.wang at noaa.gov> wrote:
>>>>>
>>>>>
>>>>> Rusty,
>>>>>
>>>>> Thanks for quick feedback. Today we just had a VLAB meeting on how
>>>>> to provide information for the public release. Vijay mentioned that EMC
>>>>> will be setting up an FV3GFS community web page through VLAB, some basic
>>>>> document will be provided there. A formal instruction on how to get release
>>>>> code, and to compile and run experiment will be on that web page too. For
>>>>> questions/feedback, a forum will be set up for users to post questions and
>>>>> to provide answers &feedback, the purpose is that all the developers will
>>>>> see the questions/answers, it is suggested not to send questions/feedback
>>>>> to any individual's personal email (If people receive questions from users,
>>>>> we suggest that they post the questions along with their answers to the
>>>>> forum). The readme.txt is a temporary solution to get the testing started,
>>>>> it may be changed in the final release.
>>>>>
>>>>> Kate Howard (kate.howard at noaa.gov) is working on the VLAB fv3gfs
>>>>> web page, she can add the gfdl fv3gfs support email on the web page too, if
>>>>> you have any fv3 document for general developers, please send to her.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> Jun
>>>>>
>>>>> On Wed, May 10, 2017 at 4:02 PM, Rusty Benson - NOAA Federal <rusty.benson at noaa.gov> wrote:
>>>>>
>>>>>
>>>>> Hi Jun and Vijay,
>>>>>
>>>>> In the readme.txt Q&A, you mention where to get help. Has there
>>>>> been any thought to putting together a single email for tracking all
>>>>> questions/requests that can be used as a basis for creating a knowledgebase
>>>>> via a wiki or other forum? By segmenting FV3 and physics support, I think
>>>>> we are missing an opportunity for personnel to get exposure to and learn
>>>>> about system pieces for which they may not necessarily be responsible.
>>>>>
>>>>> If we do go the route of a single support email, we have an
>>>>> existing email for FV3 support which could be used as an alias member.
>>>>> Otherwise, we would want to publish the email inside of the readme.txt and
>>>>> not have individual team members being contacted directly
>>>>> <oar.gfdl.fvgfs_support at noaa.gov> <oar.gfdl.fvgfs_support at noaa.gov>
>>>>>
>>>>>
>>>>>
>>>>> Rusty
>>>>> --
>>>>> Rusty Benson, PhD
>>>>> Modeling Systems Group
>>>>> NOAA Geophysical Fluid Dynamics Lab
>>>>> Princeton, NJ
>>>>>
>>>>> On Wed, May 10, 2017 at 2:08 PM, Jun Wang - NOAA Affiliate <jun.wang at noaa.gov> wrote:
>>>>>
>>>>>
>>>>> Dear all,
>>>>>
>>>>> The following email is for people who are willing to do beta
>>>>> testing for the fv3gfs May 15 release. Please ignore the email if you are
>>>>> not going to run the test.
>>>>>
>>>>> The svn tag for beta testing is located at:
>>>>>
>>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/
>>>>> tags/fv3gfs.v0beta
>>>>>
>>>>> The instruction file on how to get and compile the code and to
>>>>> run an experiment is at:
>>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t
>>>>> ags/fv3gfs.v0beta/release/v0/readme.txt
>>>>>
>>>>> Please follow the instructions to see if you can run an
>>>>> experiment.
>>>>>
>>>>> Thanks.
>>>>>
>>>>>
>>>>> Jun
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Ncep.list.fv3-announce mailing listNcep.list.fv3-announce at lstsrv.ncep.noaa.govhttps://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>>> fv3-announce
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Ncep.list.fv3-announce mailing listNcep.list.fv3-announce at lstsrv.ncep.noaa.govhttps://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>>> fv3-announce
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Ncep.list.fv3-announce mailing listNcep.list.fv3-announce at lstsrv.ncep.noaa.govhttps://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>>> fv3-announce
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gerard Ketefian
>>>>> Research Scientist
>>>>> NOAA/OAR/ESRL/GSD/EMB, R/GSD1
>>>>> 325 Broadway
>>>>> Boulder, CO 80305
>>>>> phone: 303-497-6209 <%28303%29%20497-6209> <(303)%20497-6209>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Gerard Ketefian
>>>>> Research Scientist
>>>>> NOAA/OAR/ESRL/GSD/EMB, R/GSD1
>>>>> 325 Broadway
>>>>> Boulder, CO 80305
>>>>> phone: 303-497-6209 <%28303%29%20497-6209>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Ncep.list.fv3-announce mailing listNcep.list.fv3-announce at lstsrv.ncep.noaa.govhttps://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
>>>>>
>>>>>
>>>>> --
>>>>> EUGENE MIRVIS, Tech Lead,
>>>>> Senior Computational Scientist, IMSG @
>>>>> Global Climate & Weather Modeling Branch of
>>>>> NOAA/NCEP Environmental Modeling Center
>>>>> NCWCP, rm 2183
>>>>> 5830 University Research Ct.
>>>>> College Park, MD 20740
>>>>> Ph. 301.683.3809 <%28301%29%20683-3809>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Ncep.list.fv3-announce mailing list
>>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>>> fv3-announce
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Ncep.list.fv3-announce mailing list
>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>> fv3-announce
>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> Ncep.list.fv3-announce mailing listNcep.list.fv3-announce at lstsrv.ncep.noaa.govhttps://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
>>
>>
>>
>> _______________________________________________
>> Ncep.list.fv3-announce mailing list
>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.fv3-announce/attachments/20170512/88bcd3c2/attachment-0001.html
More information about the Ncep.list.fv3-announce
mailing list