[Ncep.list.fv3-announce] fv3gfs release beta test
Samuel Trahan - NOAA Affiliate
samuel.trahan at noaa.gov
Fri May 12 19:10:22 UTC 2017
The reasons for moving to a modulefile:
1. You can unload the modulefile.
2. The same file works in all shells. Previously, if you used a tcsh
login, you could not load modules.nems. This was a major problem for some
users.
3. It prevents adding "source /etc/profile" to the modulefile, which causes
problems if /etc/profile has already been sourced.
There is now a pair of shell include files,
NEMS/src/conf/module-setup.*sh.inc, which know how to obtain the module
command and purge modules on seven clusters and five shells. We also have
a test system for it in NEMS/tests/modulecmd/. That replaces "source
/etc/profile" and "module purge."
Sincerely,
Sam Trahan
On Fri, May 12, 2017 at 1:59 PM, Dusan Jovic <dusan.jovic at noaa.gov> wrote:
> Can some of you 'module' experts please explain why is writing real
> modulefiles (a file that starts with #%Module) and then loading that file
> with 'module load' better then writing a text file with just a bunch of
> module load commands which is then "sourced" from a build or run scripts?
> We used to use the second approach in "the old NEMS" for many years and
> never had any problem.
>
> What are the advantages and are there any disadvantages?
>
> Thanks,
>
> Dusan
>
> On 05/12/2017 12:40 PM, Jun Wang - NOAA Affiliate wrote:
>
> Jim,
>
> Sam is working on that, we hope to make it to this release. Thanks.
>
> Jun
>
> On Fri, May 12, 2017 at 12:37 PM, James Rosinski - NOAA Affiliate <
> james.rosinski at noaa.gov> wrote:
>
>> Jun and others;
>>
>> I noticed there is a build capability for fv3gfs on jet, but not any run
>> instructions or runjob_jet.sh for jet. Is jet run capability intended, or
>> is that for later?
>>
>> Regards,
>> Jim Rosinski
>>
>>
>>
>> On Fri, May 12, 2017 at 10:16 AM, Rusty Benson - NOAA Federal <
>> rusty.benson at noaa.gov> wrote:
>>
>>> Eugene,
>>>
>>> Responding only to point 4, what options do you see that are Cray
>>> compilation flags?
>>>
>>> Rusty
>>> --
>>> Rusty Benson, PhD
>>> Modeling Systems Group
>>> NOAA Geophysical Fluid Dynamics Lab
>>> Princeton, NJ
>>>
>>> On Fri, May 12, 2017 at 11:58 AM, Eugene Mirvis <eugene.mirvis at noaa.gov>
>>> wrote:
>>>
>>>> Gerard,
>>>>
>>>> Just several points to clarify.
>>>>
>>>> 1. NEMS practice to call modulefiles what is actually the scripts
>>>> (calling module commands),
>>>> that require mostly bash env in order to source and keep environment
>>>> was always non standard use.
>>>>
>>>> Therefore, if the developers will take Sam's advice and make real
>>>> modulefiles (starting with #%Module) from the script, "export" and other
>>>> scripting works
>>>> wouldn't make sense, while Module util commands and Tcl/Tk will work.
>>>>
>>>> 2. There is another dilemma - to keep needed environment and change
>>>> within a workflow env. change.
>>>> btw,
>>>> unsetenv, append-path, prepend-path and remove-path module commands
>>>> are very useful for controlling that, but you have to keep
>>>> $LOADEDMODULES, and $MODULE PATH in consistent order.
>>>>
>>>> 3. Speaking of which,
>>>> module purge
>>>> and
>>>> module switch
>>>> are very useful to unload application driven modules.
>>>> a/ You just have to do that not any moment, but before apps module is
>>>> loaded, then, unload <appsModulefile> - not purge, but unload.
>>>> b/ On Crays, the are some internal dependencies inside of PrgEnv. So
>>>> you have to keep all "module use <knowns>" to recover, otherwise you might
>>>> find "module not found"
>>>>
>>>> 4.
>>>> Compiling on Theia, I'm just wondering why Cray's compilation flags are
>>>> utilized... allover:
>>>> See for instance
>>>> ...
>>>> *mpiifort -I/apps/netcdf/4.3.0-intel/include -fno-alias -auto
>>>> -safe-cray-ptr -ftz -assume byterecl -nowarn -sox -align array64byte -i4
>>>> -real-size 64 -no-prec-div -no-prec-sqrt -xCORE-AVX2**
>>>> -qno-opt-dynamic-align* -O2 -debug minimal -fp-model source
>>>> -qoverride-limits -qopt-prefetch=3 -qopenmp -I/apps/esmf/7.0.0/intel/intel
>>>> mpi/mod/modO/Linux.intel.64.intelmpi.default
>>>> -I/apps/esmf/7.0.0/intel/intelmpi/include
>>>> -I/apps/netcdf/4.3.0-intel/include -IENS_Cpl -I.
>>>> -I/scratch4/NCEPDEV/global/noscrub/Eugene.Mirvis/fv3gfs.v0beta/FV3/nems_dir
>>>> -c module_MEDIATOR_methods.f90
>>>> ...
>>>>
>>>> Thanks,
>>>> -Eugene
>>>> On 5/12/2017 3:27 AM, Samuel Trahan - NOAA Affiliate wrote:
>>>>
>>>> Gerard,
>>>>
>>>> The "module load module.free-nctools" is failing because that file contains
>>>> commands that are not valid in a modulefile. You can have as many "module"
>>>> commands as you want in a modulefile, but you cannot have bash code. Also,
>>>> it needs to start with #%Module
>>>>
>>>> export A=B # bad
>>>> setenv A B
>>>> prepend-path A (something)
>>>>
>>>> No "source /etc/profile"
>>>> No "source /.../init/bash"
>>>> No "module purge" because that causes infinite loops on some platforms.
>>>> (The module is trying to unload itself.)
>>>>
>>>> Obtaining the "module" command and purging modules has to be done before
>>>> loading the module. We have a pair of scripts (one csh, one sh) that do
>>>> that on all NOAA machines. They are aware of csh, tcsh, ksh, bash, and sh;
>>>> and run the correct initialization for each. The master copies are in
>>>> NEMS/src/conf/module-setup.*sh.inc (csh and sh versions). Those scripts
>>>> work on all NOAA machines and have a test suite inside NEMS/tests.
>>>>
>>>> man 4 modulefilehttp://modules.sourceforge.net/c/modulefile.html
>>>>
>>>>
>>>> Sincerely,
>>>> Sam Trahan
>>>>
>>>> On Thu, May 11, 2017 at 5:01 PM, Gerard Ketefian - NOAA Affiliate <gerard.ketefian at noaa.gov> wrote:
>>>>
>>>>
>>>> Hi Jun,
>>>>
>>>> Yes, after the missing "cp" fix, I (and Jim too) still had a problem with
>>>> this line in runjob_theia.sh:
>>>>
>>>> module load module.fre-nctools
>>>>
>>>> This syntax (where you specify a file name after "module load") is not
>>>> supported in our environment. So what we did was first change the above
>>>> line to:
>>>>
>>>> . ./module.fre-nctools
>>>>
>>>> to just source the module.fre-nctools file. Also, we changed the contents
>>>> of the file ${BASE_DIR}/fv3gfs.v0beta/release/v0/modulefiles/fv3gfs/fre-nctools.theia
>>>> to:
>>>>
>>>> module load impi/5.1.2.150
>>>> module load netcdf/4.3.0
>>>> module load hdf5/1.8.14
>>>> export HDF5_DIR=$HDF5
>>>> export NETCDF_DIR=$NETCDF
>>>> export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
>>>>
>>>> (Btw, should LIBRARY_PATH be changed to LD_LIBRARY_PATH in the last line?)
>>>> With this, the remapping worked without any error messages. When I
>>>> compared the output with the baseline run, all the files were the same
>>>> EXCEPT the three 1deg files from the remapping. I don't know why those
>>>> were differnt. They weren't exactly the same but had almost the same
>>>> sizes.
>>>>
>>>> It would be nice to be able to use the syntax
>>>>
>>>> module load module.fre-nctools
>>>>
>>>> in our environment as well. When I do a "module help", I don't see this
>>>> syntax as an option (where you're specifying a file name that contains
>>>> several commands). Is there a way to get this to work on theia?
>>>>
>>>> Thanks,
>>>> Gerard
>>>>
>>>>
>>>> On Thu, May 11, 2017 at 7:00 AM, Jun Wang - NOAA Affiliate <jun.wang at noaa.gov> wrote:
>>>>
>>>>
>>>> Gerard,
>>>>
>>>> With the fix last night, do you still have problem load the module
>>>> module.fre-nctools?
>>>>
>>>> Thanks for pointing out the forecast time specified in runjob_theia.sh, I
>>>> changed it to be consistent with the baseline.
>>>>
>>>> Another fix on the forecast executable is Jim Abeles are also committed
>>>> to the tag.
>>>>
>>>> I am planning to commit all the changes made to the temporary beta tag to
>>>> trunk later today for Sam to add jet extension, please send me any
>>>> fix/suggestion. Thanks to all who are doing testing.
>>>>
>>>>
>>>> Jun
>>>>
>>>> On Thu, May 11, 2017 at 5:03 AM, Gerard Ketefian - NOAA Affiliate <gerard.ketefian at noaa.gov> wrote:
>>>>
>>>>
>>>> Hi all,
>>>>
>>>> With Ligia's hints and Jun's last fix, I was able to complete the run
>>>> but not the remap. I think the remap fails because some modules don't get
>>>> loaded properly.
>>>>
>>>> To get the remap to also work, I replaced the following line in
>>>> runjob_theia.sh
>>>>
>>>> module load module.fre-nctools
>>>>
>>>> with the following block (copied and modified from the file
>>>> module.fre-nctools):
>>>>
>>>> module load impi/5.1.2.150
>>>> module load netcdf/4.3.0
>>>> module load hdf5/1.8.14
>>>> export HDF5_DIR=$HDF5
>>>> export NETCDF_DIR=$NETCDF
>>>> export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
>>>>
>>>> This change should allow the 1deg remapped netcdf files to be generated.
>>>>
>>>> When I do the comparison of the sample run's netcdf files with baseline,
>>>> there is a about factor of 5 difference (the sample run files being
>>>> larger). This is because there are only 8 output times in the baseline
>>>> files but 40 in the run output.
>>>>
>>>> Gerard
>>>>
>>>>
>>>> On Wed, May 10, 2017 at 9:56 PM, Jun Wang - NOAA Affiliate <jun.wang at noaa.gov> wrote:
>>>>
>>>>
>>>> Ligia,
>>>>
>>>> Thanks for the feedback. The suggestion on instruction is put in
>>>> readme.txt. It is found that "cp " is missing in line 124 in
>>>> runjob_theia.sh. I committed the changes to the beta test tag:https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t
>>>> ags/fv3gfs.v0beta
>>>>
>>>> Please check again. The results differences will need further
>>>> investigation. Thanks.
>>>>
>>>> Jun
>>>>
>>>> On Wed, May 10, 2017 at 10:28 PM, Ligia Bernardet - NOAA Affiliate <ligia.bernardet at noaa.gov> wrote:
>>>>
>>>>
>>>> Folks,
>>>>
>>>> Here is some feedback
>>>>
>>>>
>>>> *About the instructions*
>>>>
>>>> 1. Minor typo. The word "trunk" should be removed: Four executable
>>>> files will be created under fv3gfs.v0beta/*trunk*/NEMS/exe
>>>> 2. runjob_theia.sh: Non-NCEPDEV people need to change directories
>>>> DATA and ROTDIR to an area they can write to
>>>> 3. diff_baseline.sh:
>>>> 1. It would be helpful to tell users to add arguments
>>>> to diff_baseline.sh to set resolution and machine.
>>>> 2. Non-NCEPDEV people need to change directory dir1 to location
>>>> of their output
>>>>
>>>> *Outcome*
>>>> It seems I was able to get through the forecast but failed in remap.
>>>> Problem seems related to loading modules, I did not fully investigate yet.
>>>> Output is in /scratch4/BMC/gmtb/Ligia.Berna
>>>> rdet/fv3gfs.v0beta/release/v0/exp
>>>>
>>>> When running diff, NetCDF files differ from the baseline. I noticed
>>>> the file sizes are different (mine are larger than the baseline).
>>>>
>>>> Ligia
>>>>
>>>> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
>>>> exp/../modulefiles/fv3gfs/fre-nctools.theia module.fre-nctools
>>>>
>>>> /var/spool/torque/mom_priv/jobs/23566207.bqs3.SC: line 126:
>>>> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
>>>> exp/../modulefiles/fv3gfs/fre-nctools.theia: *Permission denied*
>>>>
>>>> + module load module.fre-nctools
>>>>
>>>> ++ /apps/lmod/lmod/libexec/lmod bash load module.fre-nctools
>>>>
>>>> Lmod has detected the following error: The following module(s) are
>>>> unknown:
>>>>
>>>> "module.fre-nctools"
>>>>
>>>> On Wed, May 10, 2017 at 5:47 PM, James Rosinski - NOAA Affiliate <james.rosinski at noaa.gov> wrote:
>>>>
>>>>
>>>> Hi Jun;
>>>>
>>>> I am about to head home for the day, but here are my comments so far,
>>>> after following the instructions for theia:
>>>>
>>>> o The builds of models and remap codes completed successfully. One
>>>> suggestion might be to have the user specify 32 or 64-bit, and nh vs. hydro
>>>> in order to cut down compilation time by a factor of 4.
>>>>
>>>> o The batch job attempting to run the model failed. Relevant lines in
>>>> err_theia are:
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> *+ cd /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900+
>>>> /bin/cp -p
>>>> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp/../../../NEMS/exe/fv3_gfs_nh.prod.32bit.x
>>>> /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900/.+ -prepend-rank
>>>> -np 288 ./fv3_gfs_nh.prod.32bit.x+ ERR=127+ export ERR+ err=127*
>>>> Looks like somehow "mpirun" was not found (note there is nothing in
>>>> front of "-prepend-rank"). FYI I use csh for my login shell--not sure if
>>>> this is behind the error. I had no modules loaded when submitting the job.
>>>>
>>>> If you'd like to examine the output you should have read access to
>>>> it here on theia:
>>>>
>>>> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp
>>>>
>>>> More info tomorrow...
>>>>
>>>> Regards,
>>>> Jim Rosinski
>>>>
>>>>
>>>> On Wed, May 10, 2017 at 2:55 PM, Jun Wang - NOAA Affiliate <jun.wang at noaa.gov> wrote:
>>>>
>>>>
>>>> Dear all,
>>>>
>>>> I was noticed that some directory names in the readme.txt are not
>>>> correct. I just updated the tag version, please let me know if you have any
>>>> further questions. Thanks.
>>>>
>>>> Jun
>>>>
>>>> On Wed, May 10, 2017 at 4:39 PM, Jun Wang - NOAA Affiliate <jun.wang at noaa.gov> wrote:
>>>>
>>>>
>>>> Rusty,
>>>>
>>>> Thanks for quick feedback. Today we just had a VLAB meeting on how
>>>> to provide information for the public release. Vijay mentioned that EMC
>>>> will be setting up an FV3GFS community web page through VLAB, some basic
>>>> document will be provided there. A formal instruction on how to get release
>>>> code, and to compile and run experiment will be on that web page too. For
>>>> questions/feedback, a forum will be set up for users to post questions and
>>>> to provide answers &feedback, the purpose is that all the developers will
>>>> see the questions/answers, it is suggested not to send questions/feedback
>>>> to any individual's personal email (If people receive questions from users,
>>>> we suggest that they post the questions along with their answers to the
>>>> forum). The readme.txt is a temporary solution to get the testing started,
>>>> it may be changed in the final release.
>>>>
>>>> Kate Howard (kate.howard at noaa.gov) is working on the VLAB fv3gfs
>>>> web page, she can add the gfdl fv3gfs support email on the web page too, if
>>>> you have any fv3 document for general developers, please send to her.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> Jun
>>>>
>>>> On Wed, May 10, 2017 at 4:02 PM, Rusty Benson - NOAA Federal <rusty.benson at noaa.gov> wrote:
>>>>
>>>>
>>>> Hi Jun and Vijay,
>>>>
>>>> In the readme.txt Q&A, you mention where to get help. Has there
>>>> been any thought to putting together a single email for tracking all
>>>> questions/requests that can be used as a basis for creating a knowledgebase
>>>> via a wiki or other forum? By segmenting FV3 and physics support, I think
>>>> we are missing an opportunity for personnel to get exposure to and learn
>>>> about system pieces for which they may not necessarily be responsible.
>>>>
>>>> If we do go the route of a single support email, we have an
>>>> existing email for FV3 support which could be used as an alias member.
>>>> Otherwise, we would want to publish the email inside of the readme.txt and
>>>> not have individual team members being contacted directly
>>>> <oar.gfdl.fvgfs_support at noaa.gov> <oar.gfdl.fvgfs_support at noaa.gov>
>>>>
>>>>
>>>>
>>>> Rusty
>>>> --
>>>> Rusty Benson, PhD
>>>> Modeling Systems Group
>>>> NOAA Geophysical Fluid Dynamics Lab
>>>> Princeton, NJ
>>>>
>>>> On Wed, May 10, 2017 at 2:08 PM, Jun Wang - NOAA Affiliate <jun.wang at noaa.gov> wrote:
>>>>
>>>>
>>>> Dear all,
>>>>
>>>> The following email is for people who are willing to do beta
>>>> testing for the fv3gfs May 15 release. Please ignore the email if you are
>>>> not going to run the test.
>>>>
>>>> The svn tag for beta testing is located at:
>>>>
>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/
>>>> tags/fv3gfs.v0beta
>>>>
>>>> The instruction file on how to get and compile the code and to
>>>> run an experiment is at:
>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t
>>>> ags/fv3gfs.v0beta/release/v0/readme.txt
>>>>
>>>> Please follow the instructions to see if you can run an
>>>> experiment.
>>>>
>>>> Thanks.
>>>>
>>>>
>>>> Jun
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Ncep.list.fv3-announce mailing listNcep.list.fv3-announce at lstsrv.ncep.noaa.govhttps://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>> fv3-announce
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Ncep.list.fv3-announce mailing listNcep.list.fv3-announce at lstsrv.ncep.noaa.govhttps://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>> fv3-announce
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Ncep.list.fv3-announce mailing listNcep.list.fv3-announce at lstsrv.ncep.noaa.govhttps://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>> fv3-announce
>>>>
>>>>
>>>>
>>>> --
>>>> Gerard Ketefian
>>>> Research Scientist
>>>> NOAA/OAR/ESRL/GSD/EMB, R/GSD1
>>>> 325 Broadway
>>>> Boulder, CO 80305
>>>> phone: 303-497-6209 <%28303%29%20497-6209> <(303)%20497-6209>
>>>>
>>>>
>>>>
>>>> --
>>>> Gerard Ketefian
>>>> Research Scientist
>>>> NOAA/OAR/ESRL/GSD/EMB, R/GSD1
>>>> 325 Broadway
>>>> Boulder, CO 80305
>>>> phone: 303-497-6209 <%28303%29%20497-6209>
>>>>
>>>>
>>>> _______________________________________________
>>>> Ncep.list.fv3-announce mailing listNcep.list.fv3-announce at lstsrv.ncep.noaa.govhttps://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
>>>>
>>>>
>>>> --
>>>> EUGENE MIRVIS, Tech Lead,
>>>> Senior Computational Scientist, IMSG @
>>>> Global Climate & Weather Modeling Branch of
>>>> NOAA/NCEP Environmental Modeling Center
>>>> NCWCP, rm 2183
>>>> 5830 University Research Ct.
>>>> College Park, MD 20740
>>>> Ph. 301.683.3809 <%28301%29%20683-3809>
>>>>
>>>>
>>>> _______________________________________________
>>>> Ncep.list.fv3-announce mailing list
>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>> fv3-announce
>>>>
>>>>
>>>
>>> _______________________________________________
>>> Ncep.list.fv3-announce mailing list
>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
>>>
>>>
>>
>
>
> _______________________________________________
> Ncep.list.fv3-announce mailing listNcep.list.fv3-announce at lstsrv.ncep.noaa.govhttps://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
>
>
>
> _______________________________________________
> Ncep.list.fv3-announce mailing list
> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.fv3-announce/attachments/20170512/c32091b1/attachment-0001.html
More information about the Ncep.list.fv3-announce
mailing list