[Ncep.list.fv3-announce] fv3gfs release beta test
Dusan Jovic
dusan.jovic at noaa.gov
Thu May 11 15:35:53 UTC 2017
I started compilation on wcoss almost an hour ago and it's still not
done. In the meantime I am looking around this package. I have few
general comments. First of all I find the whole structure of the package
very confusing. It is not clear to me where each part of the system is
located. First example, there are 3 conf directories:
./fv3gfs.v0beta/*conf*
./fv3gfs.v0beta/FV3/*conf*
./fv3gfs.v0beta/NEMS/src/*conf*
If I want to add support for a new platform where should I go and create
new configuration file? All three? Does ./fv3gfs.v0beta/conf controls
the configuration of the whole system? Or does
./fv3gfs.v0beta/NEMS/src/conf controls just NEMS configuration?
Just look at these three files:
vi conf/configure.fv3.theia FV3/conf/configure.fv3.theia
NEMS/src/conf/configure.nems
and you'll find huge number of repeated makefile flags. Do they control
just fv3 compilation, of the whole NEMS app? What about preproc tools.
Then, the tests directories:
./fv3gfs.v0beta/*tests*
./fv3gfs.v0beta/FV3/*tests*
./fv3gfs.v0beta/NEMS/*tests*
Where should I go to find test runs? In the top level tests directory of
under NEMS? If this is primarily a NEMS application I would intuitively
go to ./fv3gfs.v0beta/NEMS/tests. There I find rt.sh which basically
just loops over four directories and guesses where the actual tests are.
Are users expected to run regression tests? If yes, then there should be
just one regression test directory for the whole system.
Finally, the release directory. It is basically just a directory with a
source code for the preprocessing/postprocessing tools (fre-nctools.fd)
which is fine, but why is then forecast execution script
(exglobal_fcst_nemsfv3gfs.sh) there? I suggest we rename this directory
to tools or preprocessing and move forecast execution script somewhere else.
The module files should also be in the single location so that users can
easily control which modules are loaded. Which part of the system
./fv3gfs.v0beta/modulefiles/theia/fv3 file controls? Only fv3? Maybe,
because under NEMS/src/conf there are few other module files which I
guess control the module for the NEMS part of the system. The
fre-nctools have their own module file located in
release/v0/modulefiles/fv3gfs/fre-nctools.theia.
I understand that this system is put together by just combining
standalone FV3 repo, a top layer of NEMS repo and small portion of
fv3gfs repo (mainly fre-nctools pre/postprocessing tools) but I still
think that we should try to build a system that is more coherent and is
simple to use and easy to change and maintain. Outside users who are not
familiar with our internal code structure will be very confused.
Of course, if you follow the readme file and execute 4 suggested steps
on wcoss or theia everything will just work, but I wonder how easy it's
going to be to modify this system to work on other machine. For example
questions 5 and 6 explain where to get external libraries implying that
some users might try to port the system on machines other then
wcoss_cray or theia.
Dusan
On 05/11/2017 09:00 AM, Jun Wang - NOAA Affiliate wrote:
> Gerard,
>
> With the fix last night, do you still have problem load the module
> module.fre-nctools?
>
> Thanks for pointing out the forecast time specified in
> runjob_theia.sh, I changed it to be consistent with the baseline.
>
> Another fix on the forecast executable is Jim Abeles are also
> committed to the tag.
>
> I am planning to commit all the changes made to the temporary beta tag
> to trunk later today for Sam to add jet extension, please send me any
> fix/suggestion. Thanks to all who are doing testing.
>
>
> Jun
>
> On Thu, May 11, 2017 at 5:03 AM, Gerard Ketefian - NOAA Affiliate
> <gerard.ketefian at noaa.gov <mailto:gerard.ketefian at noaa.gov>> wrote:
>
> Hi all,
>
> With Ligia's hints and Jun's last fix, I was able to complete the
> run but not the remap. I think the remap fails because some
> modules don't get loaded properly.
>
> To get the remap to also work, I replaced the following line in
> runjob_theia.sh
>
> module load module.fre-nctools
>
> with the following block (copied and modified from the file
> module.fre-nctools):
>
> module load impi/5.1.2.150 <http://5.1.2.150>
> module load netcdf/4.3.0
> module load hdf5/1.8.14
> export HDF5_DIR=$HDF5
> export NETCDF_DIR=$NETCDF
> export LIBRARY_PATH=${LIBRARY_PATH}:${NETCDF}/lib:${HDF5}/lib
>
> This change should allow the 1deg remapped netcdf files to be
> generated.
>
> When I do the comparison of the sample run's netcdf files with
> baseline, there is a about factor of 5 difference (the sample run
> files being larger). This is because there are only 8 output
> times in the baseline files but 40 in the run output.
>
> Gerard
>
>
> On Wed, May 10, 2017 at 9:56 PM, Jun Wang - NOAA Affiliate
> <jun.wang at noaa.gov <mailto:jun.wang at noaa.gov>> wrote:
>
> Ligia,
>
> Thanks for the feedback. The suggestion on instruction is put
> in readme.txt. It is found that "cp " is missing in line 124
> in runjob_theia.sh. I committed the changes to the beta test
> tag:
> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/tags/fv3gfs.v0beta
> <https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/tags/fv3gfs.v0beta>
>
> Please check again. The results differences will need further
> investigation. Thanks.
>
> Jun
>
> On Wed, May 10, 2017 at 10:28 PM, Ligia Bernardet - NOAA
> Affiliate <ligia.bernardet at noaa.gov
> <mailto:ligia.bernardet at noaa.gov>> wrote:
>
> Folks,
>
> Here is some feedback
>
>
> *About the instructions*
>
> 1. Minor typo. The word "trunk" should be removed: Four
> executable files will be created under
> fv3gfs.v0beta/*trunk*/NEMS/exe
> 2. runjob_theia.sh: Non-NCEPDEV people need to change
> directories DATA and ROTDIR to an area they can write to
> 3. diff_baseline.sh:
> 1. It would be helpful to tell users to add arguments
> to diff_baseline.sh to set resolution and machine.
> 2. Non-NCEPDEV people need to change directory dir1
> to location of their output
>
> *Outcome*
> It seems I was able to get through the forecast but failed
> in remap. Problem seems related to loading modules, I did
> not fully investigate yet. Output is in
> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/exp
>
> When running diff, NetCDF files differ from the baseline.
> I noticed the file sizes are different (mine are larger
> than the baseline).
>
> Ligia
>
> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/exp/../modulefiles/fv3gfs/fre-nctools.theia
> module.fre-nctools
>
> /var/spool/torque/mom_priv/jobs/23566207.bqs3.SC
> <http://23566207.bqs3.SC>: line 126:
> /scratch4/BMC/gmtb/Ligia.Bernardet/fv3gfs.v0beta/release/v0/exp/../modulefiles/fv3gfs/fre-nctools.theia:
> *Permission denied*
>
> + module load module.fre-nctools
>
> ++ /apps/lmod/lmod/libexec/lmod bash load module.fre-nctools
>
> Lmod has detected the following error: The following
> module(s) are unknown:
>
> "module.fre-nctools"
>
> On Wed, May 10, 2017 at 5:47 PM, James Rosinski - NOAA
> Affiliate <james.rosinski at noaa.gov
> <mailto:james.rosinski at noaa.gov>> wrote:
>
> Hi Jun;
>
> I am about to head home for the day, but here are my
> comments so far, after following the instructions for
> theia:
>
> o The builds of models and remap codes completed
> successfully. One suggestion might be to have the user
> specify 32 or 64-bit, and nh vs. hydro in order to cut
> down compilation time by a factor of 4.
>
> o The batch job attempting to run the model failed.
> Relevant lines in err_theia are:
>
> *+ cd
> /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900
> + /bin/cp -p
> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp/../../../NEMS/exe/fv3_gfs_nh.prod.32bit.x
> /scratch4/NCEPDEV/stmp3/James.Rosinski/C96fv3gfs2016092900/.
> + -prepend-rank -np 288 ./fv3_gfs_nh.prod.32bit.x
> + ERR=127
> + export ERR
> + err=127
>
> *
> Looks like somehow "mpirun" was not found (note there
> is nothing in front of "-prepend-rank"). FYI I use csh
> for my login shell--not sure if this is behind the
> error. I had no modules loaded when submitting the job.
>
> If you'd like to examine the output you should have
> read access to it here on theia:
>
> /scratch3/BMC/gsd-hpcs/rosinski/fv3gfs.v0beta/release/v0/exp
>
> More info tomorrow...
>
> Regards,
> Jim Rosinski
>
>
> On Wed, May 10, 2017 at 2:55 PM, Jun Wang - NOAA
> Affiliate <jun.wang at noaa.gov
> <mailto:jun.wang at noaa.gov>> wrote:
>
> Dear all,
>
> I was noticed that some directory names in the
> readme.txt are not correct. I just updated the tag
> version, please let me know if you have any
> further questions. Thanks.
>
> Jun
>
> On Wed, May 10, 2017 at 4:39 PM, Jun Wang - NOAA
> Affiliate <jun.wang at noaa.gov
> <mailto:jun.wang at noaa.gov>> wrote:
>
> Rusty,
>
> Thanks for quick feedback. Today we just had a
> VLAB meeting on how to provide information for
> the public release. Vijay mentioned that EMC
> will be setting up an FV3GFS community web
> page through VLAB, some basic document will be
> provided there. A formal instruction on how to
> get release code, and to compile and run
> experiment will be on that web page too. For
> questions/feedback, a forum will be set up for
> users to post questions and to provide answers
> &feedback, the purpose is that all the
> developers will see the questions/answers, it
> is suggested not to send questions/feedback to
> any individual's personal email (If people
> receive questions from users, we suggest that
> they post the questions along with their
> answers to the forum). The readme.txt is a
> temporary solution to get the testing started,
> it may be changed in the final release.
>
> Kate Howard (kate.howard at noaa.gov
> <mailto:kate.howard at noaa.gov>) is working on
> the VLAB fv3gfs web page, she can add the gfdl
> fv3gfs support email on the web page too, if
> you have any fv3 document for general
> developers, please send to her.
>
> Thanks.
>
>
> Jun
>
> On Wed, May 10, 2017 at 4:02 PM, Rusty Benson
> - NOAA Federal <rusty.benson at noaa.gov
> <mailto:rusty.benson at noaa.gov>> wrote:
>
> Hi Jun and Vijay,
>
> In the readme.txt Q&A, you mention where
> to get help. Has there been any thought to
> putting together a single email for
> tracking all questions/requests that can
> be used as a basis for creating a
> knowledgebase via a wiki or other forum?
> By segmenting FV3 and physics support, I
> think we are missing an opportunity for
> personnel to get exposure to and learn
> about system pieces for which they may not
> necessarily be responsible.
>
> If we do go the route of a single support
> email, we have an existing email for FV3
> support which could be used as an alias
> member. Otherwise, we would want to
> publish the email inside of the readme.txt
> and not have individual team members being
> contacted directly
>
> <oar.gfdl.fvgfs_support at noaa.gov
> <mailto:oar.gfdl.fvgfs_support at noaa.gov>>
>
>
>
> Rusty
> --
> Rusty Benson, PhD
> Modeling Systems Group
> NOAA Geophysical Fluid Dynamics Lab
> Princeton, NJ
>
> On Wed, May 10, 2017 at 2:08 PM, Jun Wang
> - NOAA Affiliate <jun.wang at noaa.gov
> <mailto:jun.wang at noaa.gov>> wrote:
>
> Dear all,
>
> The following email is for people who
> are willing to do beta testing for the
> fv3gfs May 15 release. Please ignore
> the email if you are not going to run
> the test.
>
> The svn tag for beta testing is
> located at:
>
> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/tags/fv3gfs.v0beta
> <https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/tags/fv3gfs.v0beta>
>
> The instruction file on how to get and
> compile the code and to run an
> experiment is at:
>
> https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/tags/fv3gfs.v0beta/release/v0/readme.txt
> <https://svnemc.ncep.noaa.gov/projects/nems/apps/NEMSfv3gfs/tags/fv3gfs.v0beta/release/v0/readme.txt>
>
> Please follow the instructions to see
> if you can run an experiment.
>
> Thanks.
>
>
> Jun
>
>
>
> _______________________________________________
> Ncep.list.fv3-announce mailing list
> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
> <mailto:Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov>
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce>
>
>
>
>
>
> _______________________________________________
> Ncep.list.fv3-announce mailing list
> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
> <mailto:Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov>
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce>
>
>
>
> _______________________________________________
> Ncep.list.fv3-announce mailing list
> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
> <mailto:Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov>
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
> <https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce>
>
>
>
>
>
>
> --
> Gerard Ketefian
> Research Scientist
> NOAA/OAR/ESRL/GSD/EMB, R/GSD1
> 325 Broadway
> Boulder, CO 80305
> phone: 303-497-6209 <tel:%28303%29%20497-6209>
>
>
>
>
> _______________________________________________
> Ncep.list.fv3-announce mailing list
> Ncep.list.fv3-announce at lstsrv.ncep.noaa.gov
> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/ncep.list.fv3-announce/attachments/20170511/a28bf4f4/attachment-0001.html
More information about the Ncep.list.fv3-announce
mailing list