[Ncep.list.fv3-announce] fv3gfs release beta test

Jun Wang - NOAA Affiliate jun.wang at noaa.gov
Fri May 12 10:50:59 UTC 2017


Sam, Thanks for explanation!

For the theia remap module file, I noticed for:

setenv A B

B has to have predefined value, otherwise an error message will be shown
before any module load commend is executed in the module file. Since the
setenv commands are not required to run remap, they are removed from the
module file.

The change was committed to beta tag last night, please let me know if you
still have problem loading remap modules.

Thanks.

Jun

On Fri, May 12, 2017 at 3:27 AM, Samuel Trahan - NOAA Affiliate <
samuel.trahan at noaa.gov> wrote:

> Gerard,
>
> The "module load module.free-nctools" is failing because that file
> contains commands that are not valid in a modulefile.  You can have as many
> "module" commands as you want in a modulefile, but you cannot have bash
> code.  Also, it needs to start with #%Module
>
> export A=B # bad
> setenv A B
> prepend-path A (something)
>
> No "source /etc/profile"
> No "source /.../init/bash"
> No "module purge" because that causes infinite loops on some platforms.
> (The module is trying to unload itself.)
>
> Obtaining the "module" command and purging modules has to be done before
> loading the module.  We have a pair of scripts (one csh, one sh) that do
> that on all NOAA machines.  They are aware of csh, tcsh, ksh, bash, and sh;
> and run the correct initialization for each.  The master copies are in
> NEMS/src/conf/module-setup.*sh.inc (csh and sh versions).  Those scripts
> work on all NOAA machines and have a test suite inside NEMS/tests.
>
> man 4 modulefile
> http://modules.sourceforge.net/c/modulefile.html
>
> Sincerely,
> Sam Trahan
>
> On Thu, May 11, 2017 at 5:01 PM, Gerard Ketefian - NOAA Affiliate <
> gerard.ketefian at noaa.gov> wrote:
>
>> Hi Jun,
>>
>> Yes, after the missing "cp" fix, I (and Jim too) still had a problem with
>> this line in runjob_theia.sh:
>>
>> module load module.fre-nctools
>>
>> This syntax (where you specify a file name after "module load") is not
>> supported in our environment.  So what we did was first change the above
>> line to:
>>
>> . ./module.fre-nctools
>>
>> to just source the module.fre-nctools file.  Also, we changed the
>> contents of the file ${BASE_DIR}/fv3gfs.v0beta/rele
>> ase/v0/modulefiles/fv3gfs/fre-nctools.theia to:
>>
>>    module load impi/5.1.2.150
>>    module load netcdf/4.3.0
>>    module load hdf5/1.8.14
>>    export HDF5_DIR=$HDF5
>>    export NETCDF_DIR=$NETCDF
>>    export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
>>
>> (Btw, should LIBRARY_PATH be changed to LD_LIBRARY_PATH in the last
>> line?)  With this, the remapping worked without any error messages.  When I
>> compared the output with the baseline run, all the files were the same
>> EXCEPT the three 1deg files from the remapping.  I don't know why those
>> were differnt.  They weren't exactly the same but had almost the same
>> sizes.
>>
>> It would be nice to be able to use the syntax
>>
>> module load module.fre-nctools
>>
>> in our environment as well.  When I do a "module help", I don't see this
>> syntax as an option (where you're specifying a file name that contains
>> several commands).  Is there a way to get this to work on theia?
>>
>> Thanks,
>> Gerard
>>
>>
>> On Thu, May 11, 2017 at 7:00 AM, Jun Wang - NOAA Affiliate <
>> jun.wang at noaa.gov> wrote:
>>
>>> Gerard,
>>>
>>> With the fix last night, do you still have problem load the module
>>> module.fre-nctools?
>>>
>>> Thanks for pointing out the forecast time specified in runjob_theia.sh,
>>> I changed it to be consistent with the baseline.
>>>
>>> Another fix on the forecast executable is Jim Abeles are also committed
>>> to the tag.
>>>
>>> I am planning to commit all the changes made to the temporary beta tag
>>> to trunk later today for Sam to add jet extension, please send me any
>>> fix/suggestion. Thanks to all who are doing testing.
>>>
>>>
>>> Jun
>>>
>>> On Thu, May 11, 2017 at 5:03 AM, Gerard Ketefian - NOAA Affiliate <
>>> gerard.ketefian at noaa.gov> wrote:
>>>
>>>> Hi all,
>>>>
>>>> With Ligia's hints and Jun's last fix, I was able to complete the run
>>>> but not the remap.  I think the remap fails because some modules don't get
>>>> loaded properly.
>>>>
>>>> To get the remap to also work, I replaced the following line in
>>>> runjob_theia.sh
>>>>
>>>> module load module.fre-nctools
>>>>
>>>> with the following block (copied and modified from the file
>>>> module.fre-nctools):
>>>>
>>>> module load impi/5.1.2.150
>>>> module load netcdf/4.3.0
>>>> module load hdf5/1.8.14
>>>> export HDF5_DIR=$HDF5
>>>> export NETCDF_DIR=$NETCDF
>>>> export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
>>>>
>>>> This change should allow the 1deg remapped netcdf files to be generated.
>>>>
>>>> When I do the comparison of the sample run's netcdf files with
>>>> baseline, there is a about factor of 5 difference (the sample run files
>>>> being larger).  This is because there are only 8 output times in the
>>>> baseline files but 40 in the run output.
>>>>
>>>> Gerard
>>>>
>>>>
>>>> On Wed, May 10, 2017 at 9:56 PM, Jun Wang - NOAA Affiliate <
>>>> jun.wang at noaa.gov> wrote:
>>>>
>>>>> Ligia,
>>>>>
>>>>> Thanks for the feedback.  The suggestion on instruction is put in
>>>>> readme.txt. It is found that "cp " is missing  in line 124 in
>>>>> runjob_theia.sh. I committed the changes to the beta test tag:
>>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t
>>>>> ags/fv3gfs.v0beta
>>>>>
>>>>> Please check again. The results differences will need further
>>>>> investigation. Thanks.
>>>>>
>>>>> Jun
>>>>>
>>>>> On Wed, May 10, 2017 at 10:28 PM, Ligia Bernardet - NOAA Affiliate <
>>>>> ligia.bernardet at noaa.gov> wrote:
>>>>>
>>>>>> Folks,
>>>>>>
>>>>>> Here is some feedback
>>>>>>
>>>>>>
>>>>>> *About the instructions*
>>>>>>
>>>>>>    1. Minor typo. The word "trunk" should be removed: Four
>>>>>>    executable files will be created under fv3gfs.v0beta/*trunk*/NEMS/exe
>>>>>>
>>>>>>    2. runjob_theia.sh: Non-NCEPDEV people need to change directories
>>>>>>    DATA and ROTDIR to an area they can write to
>>>>>>    3. diff_baseline.sh:
>>>>>>       1. It would be helpful to tell users to add arguments
>>>>>>       to diff_baseline.sh to set resolution and machine.
>>>>>>       2. Non-NCEPDEV people need to change directory dir1 to
>>>>>>       location of their output
>>>>>>
>>>>>> *Outcome*
>>>>>> It seems I was able to get through the forecast but failed in remap.
>>>>>> Problem seems related to loading modules, I did not fully investigate yet.
>>>>>> Output is in /scratch4/BMC/gmtb/Ligia.Berna
>>>>>> rdet/fv3gfs.v0beta/release/v0/exp
>>>>>>
>>>>>> When running diff, NetCDF files differ from the baseline. I noticed
>>>>>> the file sizes are different (mine are larger than the baseline).
>>>>>>
>>>>>> Ligia
>>>>>>
>>>>>> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
>>>>>> exp/../modulefiles/fv3gfs/fre-nctools.theia module.fre-nctools
>>>>>>
>>>>>> /var/spool/torque/mom_priv/jobs/23566207.bqs3.SC: line 126:
>>>>>> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
>>>>>> exp/../modulefiles/fv3gfs/fre-nctools.theia: *Permission denied*
>>>>>>
>>>>>> + module load module.fre-nctools
>>>>>>
>>>>>> ++ /apps/lmod/lmod/libexec/lmod bash load module.fre-nctools
>>>>>>
>>>>>> Lmod has detected the following error: The following module(s) are
>>>>>> unknown:
>>>>>>
>>>>>> "module.fre-nctools"
>>>>>>
>>>>>> On Wed, May 10, 2017 at 5:47 PM, James Rosinski - NOAA Affiliate <
>>>>>> james.rosinski at noaa.gov> wrote:
>>>>>>
>>>>>>> Hi Jun;
>>>>>>>
>>>>>>> I am about to head home for the day, but here are my comments so
>>>>>>> far, after following the instructions for theia:
>>>>>>>
>>>>>>> o The builds of models and remap codes completed successfully. One
>>>>>>> suggestion might be to have the user specify 32 or 64-bit, and nh vs. hydro
>>>>>>> in order to cut down compilation time by a factor of 4.
>>>>>>>
>>>>>>> o The batch job attempting to run the model failed. Relevant lines
>>>>>>> in err_theia are:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> *+ cd /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900+
>>>>>>> /bin/cp -p
>>>>>>> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp/../../../NEMS/exe/fv3_gfs_nh.prod.32bit.x
>>>>>>> /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900/.+ -prepend-rank
>>>>>>> -np 288 ./fv3_gfs_nh.prod.32bit.x+ ERR=127+ export ERR+ err=127*
>>>>>>> Looks like somehow "mpirun" was not found (note there is nothing in
>>>>>>> front of "-prepend-rank"). FYI I use csh for my login shell--not sure if
>>>>>>> this is behind the error. I had no modules loaded when submitting the job.
>>>>>>>
>>>>>>> If  you'd like to examine the output you should have read access to
>>>>>>> it here on theia:
>>>>>>>
>>>>>>> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp
>>>>>>>
>>>>>>> More info tomorrow...
>>>>>>>
>>>>>>> Regards,
>>>>>>> Jim Rosinski
>>>>>>>
>>>>>>>
>>>>>>> On Wed, May 10, 2017 at 2:55 PM, Jun Wang - NOAA Affiliate <
>>>>>>> jun.wang at noaa.gov> wrote:
>>>>>>>
>>>>>>>> Dear all,
>>>>>>>>
>>>>>>>> I was noticed that some directory names in the readme.txt are not
>>>>>>>> correct. I just updated the tag version, please let me know if you have any
>>>>>>>> further questions. Thanks.
>>>>>>>>
>>>>>>>> Jun
>>>>>>>>
>>>>>>>> On Wed, May 10, 2017 at 4:39 PM, Jun Wang - NOAA Affiliate <
>>>>>>>> jun.wang at noaa.gov> wrote:
>>>>>>>>
>>>>>>>>> Rusty,
>>>>>>>>>
>>>>>>>>> Thanks for quick feedback. Today we just had a VLAB meeting on how
>>>>>>>>> to provide information for the public release. Vijay mentioned that EMC
>>>>>>>>> will be setting up an FV3GFS community web page through VLAB, some basic
>>>>>>>>> document will be provided there. A formal instruction on how to get release
>>>>>>>>> code, and to compile and run experiment will be on that web page too. For
>>>>>>>>> questions/feedback,  a forum will be set up for users to post questions and
>>>>>>>>> to provide answers &feedback, the purpose is that all the developers will
>>>>>>>>> see the questions/answers, it is suggested not to send questions/feedback
>>>>>>>>> to any individual's personal email (If people receive questions from users,
>>>>>>>>> we suggest that they post the questions along with their answers to the
>>>>>>>>> forum). The readme.txt is a temporary solution to get the testing started,
>>>>>>>>> it may be changed in the final release.
>>>>>>>>>
>>>>>>>>> Kate Howard (kate.howard at noaa.gov) is working on the VLAB fv3gfs
>>>>>>>>> web page, she can add the gfdl fv3gfs support email on the web page too, if
>>>>>>>>> you have any fv3 document for general developers, please send to her.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Jun
>>>>>>>>>
>>>>>>>>> On Wed, May 10, 2017 at 4:02 PM, Rusty Benson - NOAA Federal <
>>>>>>>>> rusty.benson at noaa.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Jun and Vijay,
>>>>>>>>>>
>>>>>>>>>> In the readme.txt Q&A, you mention where to get help.  Has there
>>>>>>>>>> been any thought to putting together a single email for tracking all
>>>>>>>>>> questions/requests that can be used as a basis for creating a knowledgebase
>>>>>>>>>> via a wiki or other forum?  By segmenting FV3 and physics support, I think
>>>>>>>>>> we are missing an opportunity for personnel to get exposure to and learn
>>>>>>>>>> about system pieces for which they may not necessarily be responsible.
>>>>>>>>>>
>>>>>>>>>> If we do go the route of a single support email, we have an
>>>>>>>>>> existing email for FV3 support which could be used as an alias member.
>>>>>>>>>> Otherwise, we would want to publish the email inside of the readme.txt and
>>>>>>>>>> not have individual team members being contacted directly
>>>>>>>>>>
>>>>>>>>>> <oar.gfdl.fvgfs_support at noaa.gov>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Rusty
>>>>>>>>>> --
>>>>>>>>>> Rusty Benson, PhD
>>>>>>>>>> Modeling Systems Group
>>>>>>>>>> NOAA Geophysical Fluid Dynamics Lab
>>>>>>>>>> Princeton, NJ
>>>>>>>>>>
>>>>>>>>>> On Wed, May 10, 2017 at 2:08 PM, Jun Wang - NOAA Affiliate <
>>>>>>>>>> jun.wang at noaa.gov> wrote:
>>>>>>>>>>
>>>>>>>>>>> Dear all,
>>>>>>>>>>>
>>>>>>>>>>> The following email is for people who are willing to do beta
>>>>>>>>>>> testing for the fv3gfs May 15 release. Please ignore the email if you are
>>>>>>>>>>> not going to run the test.
>>>>>>>>>>>
>>>>>>>>>>> The svn tag for beta testing is located at:
>>>>>>>>>>>
>>>>>>>>>>>  https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/
>>>>>>>>>>> tags/fv3gfs.v0beta
>>>>>>>>>>>
>>>>>>>>>>> The instruction file on how to get and compile the code and to
>>>>>>>>>>> run an experiment is at:
>>>>>>>>>>>
>>>>>>>>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t
>>>>>>>>>>> ags/fv3gfs.v0beta/release/v0/readme.txt
>>>>>>>>>>>
>>>>>>>>>>> Please follow the instructions to see if you can run an
>>>>>>>>>>> experiment.
>>>>>>>>>>>
>>>>>>>>>>>  Thanks.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Jun
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Ncep.list.fv3-announce mailing list
>>>>>>>>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>>>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>>>>>>>>> fv3-announce
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Ncep.list.fv3-announce mailing list
>>>>>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>>>>>> fv3-announce
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Ncep.list.fv3-announce mailing list
>>>>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>>>>> fv3-announce
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Gerard Ketefian
>>>> Research Scientist
>>>> NOAA/OAR/ESRL/GSD/EMB, R/GSD1
>>>> 325 Broadway
>>>> Boulder, CO 80305
>>>> phone: 303-497-6209 <(303)%20497-6209>
>>>>
>>>>
>>>
>>
>>
>> --
>> Gerard Ketefian
>> Research Scientist
>> NOAA/OAR/ESRL/GSD/EMB, R/GSD1
>> 325 Broadway
>> Boulder, CO 80305
>> phone: 303-497-6209 <(303)%20497-6209>
>>
>>
>> _______________________________________________
>> Ncep.list.fv3-announce mailing list
>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.fv3-announce/attachments/20170512/0f1d1edc/attachment-0001.html 


More information about the Ncep.list.fv3-announce mailing list