[Ncep.list.fv3-announce] fv3gfs release beta test
Gerard Ketefian - NOAA Affiliate
gerard.ketefian at noaa.gov
Thu May 11 21:01:30 UTC 2017
Hi Jun,
Yes, after the missing "cp" fix, I (and Jim too) still had a problem with
this line in runjob_theia.sh:
module load module.fre-nctools
This syntax (where you specify a file name after "module load") is not
supported in our environment. So what we did was first change the above
line to:
. ./module.fre-nctools
to just source the module.fre-nctools file. Also, we changed the contents
of the file
${BASE_DIR}/fv3gfs.v0beta/release/v0/modulefiles/fv3gfs/fre-nctools.theia
to:
module load impi/5.1.2.150
module load netcdf/4.3.0
module load hdf5/1.8.14
export HDF5_DIR=$HDF5
export NETCDF_DIR=$NETCDF
export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
(Btw, should LIBRARY_PATH be changed to LD_LIBRARY_PATH in the last line?)
With this, the remapping worked without any error messages. When I
compared the output with the baseline run, all the files were the same
EXCEPT the three 1deg files from the remapping. I don't know why those
were differnt. They weren't exactly the same but had almost the same
sizes.
It would be nice to be able to use the syntax
module load module.fre-nctools
in our environment as well. When I do a "module help", I don't see this
syntax as an option (where you're specifying a file name that contains
several commands). Is there a way to get this to work on theia?
Thanks,
Gerard
On Thu, May 11, 2017 at 7:00 AM, Jun Wang - NOAA Affiliate <
jun.wang at noaa.gov> wrote:
> Gerard,
>
> With the fix last night, do you still have problem load the module
> module.fre-nctools?
>
> Thanks for pointing out the forecast time specified in runjob_theia.sh, I
> changed it to be consistent with the baseline.
>
> Another fix on the forecast executable is Jim Abeles are also committed to
> the tag.
>
> I am planning to commit all the changes made to the temporary beta tag to
> trunk later today for Sam to add jet extension, please send me any
> fix/suggestion. Thanks to all who are doing testing.
>
>
> Jun
>
> On Thu, May 11, 2017 at 5:03 AM, Gerard Ketefian - NOAA Affiliate <
> gerard.ketefian at noaa.gov> wrote:
>
>> Hi all,
>>
>> With Ligia's hints and Jun's last fix, I was able to complete the run but
>> not the remap. I think the remap fails because some modules don't get
>> loaded properly.
>>
>> To get the remap to also work, I replaced the following line in
>> runjob_theia.sh
>>
>> module load module.fre-nctools
>>
>> with the following block (copied and modified from the file
>> module.fre-nctools):
>>
>> module load impi/5.1.2.150
>> module load netcdf/4.3.0
>> module load hdf5/1.8.14
>> export HDF5_DIR=$HDF5
>> export NETCDF_DIR=$NETCDF
>> export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
>>
>> This change should allow the 1deg remapped netcdf files to be generated.
>>
>> When I do the comparison of the sample run's netcdf files with baseline,
>> there is a about factor of 5 difference (the sample run files being
>> larger). This is because there are only 8 output times in the baseline
>> files but 40 in the run output.
>>
>> Gerard
>>
>>
>> On Wed, May 10, 2017 at 9:56 PM, Jun Wang - NOAA Affiliate <
>> jun.wang at noaa.gov> wrote:
>>
>>> Ligia,
>>>
>>> Thanks for the feedback. The suggestion on instruction is put in
>>> readme.txt. It is found that "cp " is missing in line 124 in
>>> runjob_theia.sh. I committed the changes to the beta test tag:
>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t
>>> ags/fv3gfs.v0beta
>>>
>>> Please check again. The results differences will need further
>>> investigation. Thanks.
>>>
>>> Jun
>>>
>>> On Wed, May 10, 2017 at 10:28 PM, Ligia Bernardet - NOAA Affiliate <
>>> ligia.bernardet at noaa.gov> wrote:
>>>
>>>> Folks,
>>>>
>>>> Here is some feedback
>>>>
>>>>
>>>> *About the instructions*
>>>>
>>>> 1. Minor typo. The word "trunk" should be removed: Four executable
>>>> files will be created under fv3gfs.v0beta/*trunk*/NEMS/exe
>>>> 2. runjob_theia.sh: Non-NCEPDEV people need to change directories
>>>> DATA and ROTDIR to an area they can write to
>>>> 3. diff_baseline.sh:
>>>> 1. It would be helpful to tell users to add arguments
>>>> to diff_baseline.sh to set resolution and machine.
>>>> 2. Non-NCEPDEV people need to change directory dir1 to location
>>>> of their output
>>>>
>>>> *Outcome*
>>>> It seems I was able to get through the forecast but failed in remap.
>>>> Problem seems related to loading modules, I did not fully investigate yet.
>>>> Output is in /scratch4/BMC/gmtb/Ligia.Berna
>>>> rdet/fv3gfs.v0beta/release/v0/exp
>>>>
>>>> When running diff, NetCDF files differ from the baseline. I noticed the
>>>> file sizes are different (mine are larger than the baseline).
>>>>
>>>> Ligia
>>>>
>>>> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
>>>> exp/../modulefiles/fv3gfs/fre-nctools.theia module.fre-nctools
>>>>
>>>> /var/spool/torque/mom_priv/jobs/23566207.bqs3.SC: line 126:
>>>> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/
>>>> exp/../modulefiles/fv3gfs/fre-nctools.theia: *Permission denied*
>>>>
>>>> + module load module.fre-nctools
>>>>
>>>> ++ /apps/lmod/lmod/libexec/lmod bash load module.fre-nctools
>>>>
>>>> Lmod has detected the following error: The following module(s) are
>>>> unknown:
>>>>
>>>> "module.fre-nctools"
>>>>
>>>> On Wed, May 10, 2017 at 5:47 PM, James Rosinski - NOAA Affiliate <
>>>> james.rosinski at noaa.gov> wrote:
>>>>
>>>>> Hi Jun;
>>>>>
>>>>> I am about to head home for the day, but here are my comments so far,
>>>>> after following the instructions for theia:
>>>>>
>>>>> o The builds of models and remap codes completed successfully. One
>>>>> suggestion might be to have the user specify 32 or 64-bit, and nh vs. hydro
>>>>> in order to cut down compilation time by a factor of 4.
>>>>>
>>>>> o The batch job attempting to run the model failed. Relevant lines in
>>>>> err_theia are:
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> *+ cd /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900+
>>>>> /bin/cp -p
>>>>> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp/../../../NEMS/exe/fv3_gfs_nh.prod.32bit.x
>>>>> /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900/.+ -prepend-rank
>>>>> -np 288 ./fv3_gfs_nh.prod.32bit.x+ ERR=127+ export ERR+ err=127*
>>>>> Looks like somehow "mpirun" was not found (note there is nothing in
>>>>> front of "-prepend-rank"). FYI I use csh for my login shell--not sure if
>>>>> this is behind the error. I had no modules loaded when submitting the job.
>>>>>
>>>>> If you'd like to examine the output you should have read access to it
>>>>> here on theia:
>>>>>
>>>>> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp
>>>>>
>>>>> More info tomorrow...
>>>>>
>>>>> Regards,
>>>>> Jim Rosinski
>>>>>
>>>>>
>>>>> On Wed, May 10, 2017 at 2:55 PM, Jun Wang - NOAA Affiliate <
>>>>> jun.wang at noaa.gov> wrote:
>>>>>
>>>>>> Dear all,
>>>>>>
>>>>>> I was noticed that some directory names in the readme.txt are not
>>>>>> correct. I just updated the tag version, please let me know if you have any
>>>>>> further questions. Thanks.
>>>>>>
>>>>>> Jun
>>>>>>
>>>>>> On Wed, May 10, 2017 at 4:39 PM, Jun Wang - NOAA Affiliate <
>>>>>> jun.wang at noaa.gov> wrote:
>>>>>>
>>>>>>> Rusty,
>>>>>>>
>>>>>>> Thanks for quick feedback. Today we just had a VLAB meeting on how
>>>>>>> to provide information for the public release. Vijay mentioned that EMC
>>>>>>> will be setting up an FV3GFS community web page through VLAB, some basic
>>>>>>> document will be provided there. A formal instruction on how to get release
>>>>>>> code, and to compile and run experiment will be on that web page too. For
>>>>>>> questions/feedback, a forum will be set up for users to post questions and
>>>>>>> to provide answers &feedback, the purpose is that all the developers will
>>>>>>> see the questions/answers, it is suggested not to send questions/feedback
>>>>>>> to any individual's personal email (If people receive questions from users,
>>>>>>> we suggest that they post the questions along with their answers to the
>>>>>>> forum). The readme.txt is a temporary solution to get the testing started,
>>>>>>> it may be changed in the final release.
>>>>>>>
>>>>>>> Kate Howard (kate.howard at noaa.gov) is working on the VLAB fv3gfs
>>>>>>> web page, she can add the gfdl fv3gfs support email on the web page too, if
>>>>>>> you have any fv3 document for general developers, please send to her.
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>> Jun
>>>>>>>
>>>>>>> On Wed, May 10, 2017 at 4:02 PM, Rusty Benson - NOAA Federal <
>>>>>>> rusty.benson at noaa.gov> wrote:
>>>>>>>
>>>>>>>> Hi Jun and Vijay,
>>>>>>>>
>>>>>>>> In the readme.txt Q&A, you mention where to get help. Has there
>>>>>>>> been any thought to putting together a single email for tracking all
>>>>>>>> questions/requests that can be used as a basis for creating a knowledgebase
>>>>>>>> via a wiki or other forum? By segmenting FV3 and physics support, I think
>>>>>>>> we are missing an opportunity for personnel to get exposure to and learn
>>>>>>>> about system pieces for which they may not necessarily be responsible.
>>>>>>>>
>>>>>>>> If we do go the route of a single support email, we have an
>>>>>>>> existing email for FV3 support which could be used as an alias member.
>>>>>>>> Otherwise, we would want to publish the email inside of the readme.txt and
>>>>>>>> not have individual team members being contacted directly
>>>>>>>>
>>>>>>>> <oar.gfdl.fvgfs_support at noaa.gov>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Rusty
>>>>>>>> --
>>>>>>>> Rusty Benson, PhD
>>>>>>>> Modeling Systems Group
>>>>>>>> NOAA Geophysical Fluid Dynamics Lab
>>>>>>>> Princeton, NJ
>>>>>>>>
>>>>>>>> On Wed, May 10, 2017 at 2:08 PM, Jun Wang - NOAA Affiliate <
>>>>>>>> jun.wang at noaa.gov> wrote:
>>>>>>>>
>>>>>>>>> Dear all,
>>>>>>>>>
>>>>>>>>> The following email is for people who are willing to do beta
>>>>>>>>> testing for the fv3gfs May 15 release. Please ignore the email if you are
>>>>>>>>> not going to run the test.
>>>>>>>>>
>>>>>>>>> The svn tag for beta testing is located at:
>>>>>>>>>
>>>>>>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/
>>>>>>>>> tags/fv3gfs.v0beta
>>>>>>>>>
>>>>>>>>> The instruction file on how to get and compile the code and to run
>>>>>>>>> an experiment is at:
>>>>>>>>>
>>>>>>>>> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/t
>>>>>>>>> ags/fv3gfs.v0beta/release/v0/readme.txt
>>>>>>>>>
>>>>>>>>> Please follow the instructions to see if you can run an
>>>>>>>>> experiment.
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Jun
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Ncep.list.fv3-announce mailing list
>>>>>>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>>>>>>> fv3-announce
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Ncep.list.fv3-announce mailing list
>>>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>>>> fv3-announce
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Ncep.list.fv3-announce mailing list
>>>>> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.
>>>>> fv3-announce
>>>>>
>>>>>
>>>>
>>>
>>
>>
>> --
>> Gerard Ketefian
>> Research Scientist
>> NOAA/OAR/ESRL/GSD/EMB, R/GSD1
>> 325 Broadway
>> Boulder, CO 80305
>> phone: 303-497-6209 <(303)%20497-6209>
>>
>>
>
--
Gerard Ketefian
Research Scientist
NOAA/OAR/ESRL/GSD/EMB, R/GSD1
325 Broadway
Boulder, CO 80305
phone: 303-497-6209
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.fv3-announce/attachments/20170511/3be4f9c8/attachment-0001.html
More information about the Ncep.list.fv3-announce
mailing list