[Ncep.list.fv3-announce] fv3gfs release beta test
Dusan Jovic
dusan.jovic at noaa.gov
Fri May 12 17:59:54 UTC 2017
Can some of you 'module' experts please explain why is writing real
modulefiles (a file that starts with #%Module) and then loading that
file with 'module load' better then writing a text file with just a
bunch of module load commands which is then "sourced" from a build or
run scripts? We used to use the second approach in "the old NEMS" for
many years and never had any problem.
What are the advantages and are there any disadvantages?
Thanks,
Dusan
On 05/12/2017 12:40 PM, Jun Wang - NOAA Affiliate wrote:
> Jim,
>
> Sam is working on that, we hope to make it to this release. Thanks.
>
> Jun
>
> On Fri, May 12, 2017 at 12:37 PM, James Rosinski - NOAA Affiliate
> <james.rosinski at noaa.gov <mailto:james.rosinski at noaa.gov>> wrote:
>
> Jun and others;
>
> I noticed there is a build capability for fv3gfs on jet, but not
> any run instructions or runjob_jet.sh for jet. Is jet run
> capability intended, or is that for later?
>
> Regards,
> Jim Rosinski
>
>
>
> On Fri, May 12, 2017 at 10:16 AM, Rusty Benson - NOAA Federal
> <rusty.benson at noaa.gov <mailto:rusty.benson at noaa.gov>> wrote:
>
> Eugene,
>
> Responding only to point 4, what options do you see that are
> Cray compilation flags?
>
> Rusty
> --
> Rusty Benson, PhD
> Modeling Systems Group
> NOAA Geophysical Fluid Dynamics Lab
> Princeton, NJ
>
> On Fri, May 12, 2017 at 11:58 AM, Eugene Mirvis
> <eugene.mirvis at noaa.gov <mailto:eugene.mirvis at noaa.gov>> wrote:
>
> Gerard,
>
> Just several points to clarify.
>
> 1. NEMS practice to call modulefiles what is actually the
> scripts (calling module commands),
> that require mostly bash env in order to source and keep
> environment was always non standard use.
>
> Therefore, if the developers will take Sam's advice and
> make real modulefiles (starting with #%Module) from the
> script, "export" and other scripting works
> wouldn't make sense, while Module util commands and
> Tcl/Tk will work.
>
> 2. There is another dilemma - to keep needed environment
> and change within a workflow env. change.
> btw, *
> *unsetenv, append-path, prepend-path and remove-path
> module commands are very useful for controlling that, but
> you have to keep
> $LOADEDMODULES, and $MODULE PATH in consistent order.
>
> 3. Speaking of which,
> module purge
> and
> module switch
> are very useful to unload application driven modules.
> a/ You just have to do that not any moment, but before
> apps module is loaded, then, unload <appsModulefile> - not
> purge, but unload.
> b/ On Crays, the are some internal dependencies inside of
> PrgEnv. So you have to keep all "module use <knowns>" to
> recover, otherwise you might find "module not found"
>
> 4.
> Compiling on Theia, I'm just wondering why Cray's
> compilation flags are utilized... allover:
> See for instance
> ...
> *mpiifort -I/apps/netcdf/4.3.0-intel/include -fno-alias
> -auto -safe-cray-ptr -ftz -assume byterecl -nowarn -sox
> -align array64byte -i4 -real-size 64 -no-prec-div
> -no-prec-sqrt -xCORE-AVX2**-qno-opt-dynamic-align* -O2
> -debug minimal -fp-model source -qoverride-limits
> -qopt-prefetch=3 -qopenmp
> -I/apps/esmf/7.0.0/intel/intelmpi/mod/modO/Linux.intel.64.in
> <http://Linux.intel.64.in>telmpi.default
> -I/apps/esmf/7.0.0/intel/intelmpi/include
> -I/apps/netcdf/4.3.0-intel/include -IENS_Cpl -I.
> -I/scratch4/NCEPDEV/global/noscrub/Eugene.Mirvis/fv3gfs.v0beta/FV3/nems_dir
> -c module_MEDIATOR_methods.f90
> ...
>
> Thanks,
> -Eugene
> On 5/12/2017 3:27 AM, Samuel Trahan - NOAA Affiliate wrote:
>> Gerard, The "module load module.free-nctools" is failing
>> because that file contains commands that are not valid in
>> a modulefile. You can have as many "module" commands as
>> you want in a modulefile, but you cannot have bash code.
>> Also, it needs to start with #%Module export A=B # bad
>> setenv A B prepend-path A (something) No "source
>> /etc/profile" No "source /.../init/bash" No "module
>> purge" because that causes infinite loops on some
>> platforms. (The module is trying to unload itself.)
>> Obtaining the "module" command and purging modules has to
>> be done before loading the module. We have a pair of
>> scripts (one csh, one sh) that do that on all NOAA
>> machines. They are aware of csh, tcsh, ksh, bash, and sh;
>> and run the correct initialization for each. The master
>> copies are in NEMS/src/conf/module-setup.*sh.inc (csh and
>> sh versions). Those scripts work on all NOAA machines and
>> have a test suite inside NEMS/tests. man 4 modulefile
>> http://modules.sourceforge.net/c/modulefile.html
>> <http://modules.sourceforge.net/c/modulefile.html>
>>
>>
>> Sincerely,
>> Sam Trahan
>>
>> On Thu, May 11, 2017 at 5:01 PM, Gerard Ketefian - NOAA Affiliate <
>> gerard.ketefian at noaa.gov <mailto:gerard.ketefian at noaa.gov>> wrote:
>>
>>> Hi Jun,
>>>
>>> Yes, after the missing "cp" fix, I (and Jim too) still had a problem with
>>> this line in runjob_theia.sh:
>>>
>>> module load module.fre-nctools
>>>
>>> This syntax (where you specify a file name after "module load") is not
>>> supported in our environment. So what we did was first change the above
>>> line to:
>>>
>>> . ./module.fre-nctools
>>>
>>> to just source the module.fre-nctools file. Also, we changed the contents
>>> of the file ${BASE_DIR}/fv3gfs.v0beta/release/v0/modulefiles/fv3gfs/fre-nctools.theia
>>> to:
>>>
>>> module load impi/5.1.2.150 <http://5.1.2.150>
>>> module load netcdf/4.3.0
>>> module load hdf5/1.8.14
>>> export HDF5_DIR=$HDF5
>>> export NETCDF_DIR=$NETCDF
>>> export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
>>>
>>> (Btw, should LIBRARY_PATH be changed to LD_LIBRARY_PATH in the last line?)
>>> With this, the remapping worked without any error messages. When I
>>> compared the output with the baseline run, all the files were the same
>>> EXCEPT the three 1deg files from the remapping. I don't know why those
>>> were differnt. They weren't exactly the same but had almost the same
>>> sizes.
>>>
>>> It would be nice to be able to use the syntax
>>>
>>> module load module.fre-nctools
>>>
>>> in our environment as well. When I do a "module help", I don't see this
>>> syntax as an option (where you're specifying a file name that contains
>>> several commands). Is there a way to get this to work on theia?
>>>
>>> Thanks,
>>> Gerard
>>>
>>>
>>> On Thu, May 11, 2017 at 7:00 AM, Jun Wang - NOAA Affiliate <
>>> jun.wang at noaa.gov <mailto:jun.wang at noaa.gov>> wrote:
>>>
>>>> Gerard,
>>>>
>>>> With the fix last night, do you still have problem load the module
>>>> module.fre-nctools?
>>>>
>>>> Thanks for pointing out the forecast time specified in runjob_theia.sh, I
>>>> changed it to be consistent with the baseline.
>>>>
>>>> Another fix on the forecast executable is Jim Abeles are also committed
>>>> to the tag.
>>>>
>>>> I am planning to commit all the changes made to the temporary beta tag to
>>>> trunk later today for Sam to add jet extension, please send me any
>>>> fix/suggestion. Thanks to all who are doing testing.
>>>>
>>>>
>>>> Jun
>>>>
>>>> On Thu, May 11, 2017 at 5:03 AM, Gerard Ketefian - NOAA Affiliate <
>>>> gerard.ketefian at noaa.gov <mailto:gerard.ketefian at noaa.gov>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> With Ligia's hints and Jun's last fix, I was able to complete the run
>>>>> but not the remap. I think the remap fails because some modules don't get
>>>>> loaded properly.
>>>>>
>>>>> To get the remap to also work, I replaced the following line in
>>>>> runjob_theia.sh
>>>>>
>>>>> module load module.fre-nctools
>>>>>
>>>>> with the following block (copied and modified from the file
>>>>> module.fre-nctools):
>>>>>
>>>>> module load impi/5.1.2.150 <http://5.1.2.150>
>>>>> module load netcdf/4.3.0
>>>>> module load hdf5/1.8.14
>>>>> export HDF5_DIR=$HDF5
>>>>> export NETCDF_DIR=$NETCDF
>>>>> export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
>>>>>
>>>>> This change should allow the 1deg remapped netcdf files to be generated.
>>>>>
>>>>> When I do the comparison of the sample run's netcdf files with baseline,
>>>>> there is a about factor of 5 difference (the sample run files being
>>>>> larger). This is because there are only 8 output times in the baseline
>>>>> files but 40 in the run output.
>>>>>
>>>>> Gerard
>>>>>
>>>>>
>>>>> On Wed, May 10, 2017 at 9:56 PM, Jun Wang - NOAA Affiliate <
>>>>> jun.wang at noaa.gov <mailto:jun.wang at noaa.gov>> wrote:
>>>>>
>>>>>> Ligia,
>>>>>>
>>>>>> Thanks for the feedback. The suggestion on instruction is put in
>>>>>> readme.txt. It is found that "cp " is missing in line 124 in
>>>>>> runjob_theia.sh. I committed the changes to the beta test tag:
>>>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t
>>>>>> <https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t>
>>>>>> ags/fv3gfs.v0beta
>>>>>>
>>>>>> Please check again. The results differences will need further
>>>>>> investigation. Thanks.
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>> On Wed, May 10, 2017 at 10:28 PM, Ligia Bernardet - NOAA Affiliate <
>>>>>> ligia.bernardet at noaa.gov
>>>>>> <mailto:ligia.bernardet at noaa.gov>> wrote:
>>>>>>
>>>>>>> Folks,
>>>>>>>
>>>>>>> Here is some feedback
>>>>>>>
>>>>>>>
>>>>>>> *About the instructions*
>>>>>>>
>>>>>>> 1. Minor typo. The word "trunk" should be removed: Four executable
>>>>>>> files will be created under fv3gfs.v0beta/*trunk*/NEMS/exe
>>>>>>> 2. runjob_theia.sh: Non-NCEPDEV people need to change directories
>>>>>>> DATA and ROTDIR to an area they can write to
>>>>>>> 3. diff_baseline.sh:
>>>>>>> 1. It would be helpful to tell users to add arguments
>>>>>>> to diff_baseline.sh to set resolution and machine.
>>>>>>> 2. Non-NCEPDEV people need to change directory dir1 to location
>>>>>>> of their output
>>>>>>>
>>>>>>> *Outcome*
>>>>>>> It seems I was able to get through the forecast but failed in remap.
>>>>>>> Problem seems related to loading modules, I did not fully investigate yet.
>>>>>>> Output is in /scratch4/BMC/gmtb/Ligia.Berna
>>>>>>> rdet/fv3gfs.v0beta/release/v0/exp
>>>>>>>
>>>>>>> When running diff, NetCDF files differ from the baseline. I noticed
>>>>>>> the file sizes are different (mine are larger than the baseline).
>>>>>>>
>>>>>>> Ligia
>>>>>>>
>>>>>>> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
>>>>>>> exp/../modulefiles/fv3gfs/fre-nctools.theia module.fre-nctools
>>>>>>>
>>>>>>> /var/spool/torque/mom_priv/jobs/23566207.bqs3.SC <http://23566207.bqs3.SC>: line 126:
>>>>>>> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
>>>>>>> exp/../modulefiles/fv3gfs/fre-nctools.theia: *Permission denied*
>>>>>>>
>>>>>>> + module load module.fre-nctools
>>>>>>>
>>>>>>> ++ /apps/lmod/lmod/libexec/lmod bash load module.fre-nctools
>>>>>>>
>>>>>>> Lmod has detected the following error: The following module(s) are
>>>>>>> unknown:
>>>>>>>
>>>>>>> "module.fre-nctools"
>>>>>>>
>>>>>>> On Wed, May 10, 2017 at 5:47 PM, James Rosinski - NOAA Affiliate <
>>>>>>> james.rosinski at noaa.gov <mailto:james.rosinski at noaa.gov>> wrote:
>>>>>>>
>>>>>>>> Hi Jun;
>>>>>>>>
>>>>>>>> I am about to head home for the day, but here are my comments so far,
>>>>>>>> after following the instructions for theia:
>>>>>>>>
>>>>>>>> o The builds of models and remap codes completed successfully. One
>>>>>>>> suggestion might be to have the user specify 32 or 64-bit, and nh vs. hydro
>>>>>>>> in order to cut down compilation time by a factor of 4.
>>>>>>>>
>>>>>>>> o The batch job attempting to run the model failed. Relevant lines in
>>>>>>>> err_theia are:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> *+ cd /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900+
>>>>>>>> /bin/cp -p
>>>>>>>> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp/../../../NEMS/exe/fv3_gfs_nh.prod.32bit.x
>>>>>>>> /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900/.+ -prepend-rank
>>>>>>>> -np 288 ./fv3_gfs_nh.prod.32bit.x+ ERR=127+ export ERR+ err=127*
>>>>>>>> Looks like somehow "mpirun" was not found (note there is nothing in
>>>>>>>> front of "-prepend-rank"). FYI I use csh for my login shell--not sure if
>>>>>>>> this is behind the error. I had no modules loaded when submitting the job.
>>>>>>>>
>>>>>>>> If you'd like to examine the output you should have read access to
>>>>>>>> it here on theia:
>>>>>>>>
>>>>>>>> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp
>>>>>>>>
>>>>>>>> More info tomorrow...
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Jim Rosinski
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, May 10, 2017 at 2:55 PM, Jun Wang - NOAA Affiliate <
>>>>>>>> jun.wang at noaa.gov <mailto:jun.wang at noaa.gov>> wrote:
>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> I was noticed that some directory names in the readme.txt are not
>>>>>>>>> correct. I just updated the tag version, please let me know if you have any
>>>>>>>>> further questions. Thanks.
>>>>>>>>>
>>>>>>>>> Jun
>>>>>>>>>
>>>>>>>>> On Wed, May 10, 2017 at 4:39 PM, Jun Wang - NOAA Affiliate <
>>>>>>>>> jun.wang at noaa.gov <mailto:jun.wang at noaa.gov>> wrote:
>>>>>>>>>
>>>>>>>>>> Rusty,
>>>>>>>>>>
>>>>>>>>>> Thanks for quick feedback. Today we just had a VLAB meeting on how
>>>>>>>>>> to provide information for the public release. Vijay mentioned that EMC
>>>>>>>>>> will be setting up an FV3GFS community web page through VLAB, some basic
>>>>>>>>>> document will be provided there. A formal instruction on how to get release
>>>>>>>>>> code, and to compile and run experiment will be on that web page too. For
>>>>>>>>>> questions/feedback, a forum will be set up for users to post questions and
>>>>>>>>>> to provide answers &feedback, the purpose is that all the developers will
>>>>>>>>>> see the questions/answers, it is suggested not to send questions/feedback
>>>>>>>>>> to any individual's personal email (If people receive questions from users,
>>>>>>>>>> we suggest that they post the questions along with their answers to the
>>>>>>>>>> forum). The readme.txt is a temporary solution to get the testing started,
>>>>>>>>>> it may be changed in the final release.
>>>>>>>>>>
>>>>>>>>>> Kate Howard (kate.howard at noaa.gov <mailto:kate.howard at noaa.gov>) is working on the VLAB fv3gfs
>>>>>>>>>> web page, she can add the gfdl fv3gfs support email on the web page too, if
>>>>>>>>>> you have any fv3 document for general developers, please send to her.
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Jun
>>>>>>>>>>
>>>>>>>>>> On Wed, May 10, 2017 at 4:02 PM, Rusty Benson - NOAA Federal <
>>>>>>>>>> rusty.benson at noaa.gov <mailto:rusty.benson at noaa.gov>> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Jun and Vijay,
>>>>>>>>>>>
>>>>>>>>>>> In the readme.txt Q&A, you mention where to get help. Has there
>>>>>>>>>>> been any thought to putting together a single email for tracking all
>>>>>>>>>>> questions/requests that can be used as a basis for creating a knowledgebase
>>>>>>>>>>> via a wiki or other forum? By segmenting FV3 and physics support, I think
>>>>>>>>>>> we are missing an opportunity for personnel to get exposure to and learn
>>>>>>>>>>> about system pieces for which they may not necessarily be responsible.
>>>>>>>>>>>
>>>>>>>>>>> If we do go the route of a single support email, we have an
>>>>>>>>>>> existing email for FV3 support which could be used as an alias member.
>>>>>>>>>>> Otherwise, we would want to publish the email inside of the readme.txt and
>>>>>>>>>>> not have individual team members being contacted directly
>>>>>>>>>>>
>>>>>>>>>>> <oar.gfdl.fvgfs_support at noaa.gov>
>>>>>>>>>>> <mailto:oar.gfdl.fvgfs_support at noaa.gov>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Rusty
>>>>>>>>>>> --
>>>>>>>>>>> Rusty Benson, PhD
>>>>>>>>>>> Modeling Systems Group
>>>>>>>>>>> NOAA Geophysical Fluid Dynamics Lab
>>>>>>>>>>> Princeton, NJ
>>>>>>>>>>>
>>>>>>>>>>> On Wed, May 10, 2017 at 2:08 PM, Jun Wang - NOAA Affiliate <
>>>>>>>>>>> jun.wang at noaa.gov <mailto:jun.wang at noaa.gov>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Dear all,
>>>>>>>>>>>>
>>>>>>>>>>>> The following email is for people who are willing to do beta
>>>>>>>>>>>> testing for the fv3gfs May 15 release. Please ignore the email if you are
>>>>>>>>>>>> not going to run the test.
>>>>>>>>>>>>
>>>>>>>>>>>> The svn tag for beta testing is located at:
>>>>>>>>>>>>
>>>>>>>>>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/
>>>>>>>>>>>> <https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/>
>>>>>>>>>>>> tags/fv3gfs.v0beta
>>>>>>>>>>>>
>>>>>>>>>>>> The instruction file on how to get and compile the code and to
>>>>>>>>>>>> run an experiment is at:
>>>>>>>>>>>>
>>>>>>>>>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t
>>>>>>>>>>>> <https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t>
>>>>>>>>>>>> ags/fv3gfs.v0beta/release/v0/readme.txt
>>>>>>>>>>>>
>>>>>>>>>>>> Please follow the instructions to see if you can run an
>>>>>>>>>>>> experiment.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks.
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Jun
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Ncep.list.fv3-announce mailing list
>>>>>>>>>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>>>>>>>>>> <mailto:Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov>
>>>>>>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list
>>>>>>>>>>>> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list>.
>>>>>>>>>>>> fv3-announce
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Ncep.list.fv3-announce mailing list
>>>>>>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>>>>>>> <mailto:Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov>
>>>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list
>>>>>>>>> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list>.
>>>>>>>>> fv3-announce
>>>>>>>>>
>>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Ncep.list.fv3-announce mailing list
>>>>>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>>>>>> <mailto:Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov>
>>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list
>>>>>>>> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list>.
>>>>>>>> fv3-announce
>>>>>>>>
>>>>>>>>
>>>>> --
>>>>> Gerard Ketefian
>>>>> Research Scientist
>>>>> NOAA/OAR/ESRL/GSD/EMB, R/GSD1
>>>>> 325 Broadway
>>>>> Boulder, CO 80305
>>>>> phone:303-497-6209 <tel:%28303%29%20497-6209> <(303)%20497-6209>
>>>>>
>>>>>
>>> --
>>> Gerard Ketefian
>>> Research Scientist
>>> NOAA/OAR/ESRL/GSD/EMB, R/GSD1
>>> 325 Broadway
>>> Boulder, CO 80305
>>> phone:303-497-6209 <tel:%28303%29%20497-6209>
>>>
>>>
>>> _______________________________________________
>>> Ncep.list.fv3-announce mailing list
>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>> <mailto:Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov>
>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
>>> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce>
>>>
>>>
>
> --
> EUGENE MIRVIS, Tech Lead,
> Senior Computational Scientist, IMSG @
> Global Climate & Weather Modeling Branch of
> NOAA/NCEP Environmental Modeling Center
> NCWCP, rm 2183
> 5830 University Research Ct.
> College Park, MD 20740
> Ph.301.683.3809 <tel:%28301%29%20683-3809>
>
>
> _______________________________________________
> Ncep.list.fv3-announce mailing list
> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
> <mailto:Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov>
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce>
>
>
>
> _______________________________________________
> Ncep.list.fv3-announce mailing list
> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
> <mailto:Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov>
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce>
>
>
>
>
>
> _______________________________________________
> Ncep.list.fv3-announce mailing list
> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.fv3-announce/attachments/20170512/5b9a0722/attachment-0001.html
More information about the Ncep.list.fv3-announce
mailing list