[Ncep.list.nems.announce] Proposed NEMS commit for Performance Improvements to RRTM-G

Wed Jul 16 17:54:11 UTC 2014

Hi all,

I have an update on the effect of the changes on GFS performance.  My
earlier results were on Zeus, which has older Westmere processors.
Rerunning the test on WCOSS (Tide) this morning, there is a marked
improvement from the changes for GFS too.  On the GFS_16_32 regression case,
the time for the top of the trunk was 30.0 seconds; with the updates to
RRTMG I'm seeing 27.4 seconds.  This test also uses threading and I think
the Sandybridge processors on WCOSS do a better job of latency-hiding with
two threads per core than the Westmeres on Zeus.

So at this point I'm in the process of looking at differences in solution
output from the GFS, and I'll provide a report on that soon as part of my
commit proposal.  Aside from Moorthi's questions, I haven't heard back from
anyone else.  Please let me know if you have questions or concerns.  I'm
still shooting for having this committed by end of July, if possible.  

Thanks,

John

PS...   I've been doing my testing on WCOSS with the latest version of ESMF
(configure.nems.Wcoss.intel_ESMF_630r) and I've run into a couple of issues:

1.  I noticed that even though the code appears to run correctly, at the end
of the run each MPI tasks gives the following message:

GFS FINALIZE STEP SUCCEEDED
Abort(0) on node 15 (rank 15 in comm -2080374783): application called
MPI_Abort(comm=0x84000001, 0) - process 15

Is that normal and expected?

2.  When compiling for one of the tests with "make nmm_gsm GOCART_MODE=full"
I'm seeing a  compilation error:

mpiifort -g -openmp  -mkl=sequential -O2
-I/nems/save/Weiyu.Yang/ESMF_6_3_0r/esmf/mod/modO/Linux.intel.64.intelmpi.de
fault -I/nems/save/Weiyu.Yang/ESMF_6_3_0r/esmf/src/include
-I/nwprod/lib/incmod/nemsio -convert big_endian -assume byterecl -fp-model
precise -xAVX -fno-alias -free -O3 -r8 -free  -I../../share
-I../../../chem/gocart/Linux/include/Chem_Base
-I../../../chem/gocart/Linux/include/GMAO_mpeu
-I../../../chem/gocart/Linux/include/GEOSchem_GridComp -c
atmos_phy_chem_cpl_comp_mod.f90
ifort: command line warning #10120: overriding '-O2' with '-O3'
atmos_phy_chem_cpl_comp_mod.f90(284): error #6285: There is no matching
specific subroutine for this generic subroutine call.
[ESMF_FIELDBUNDLEADD]
         call ESMF_FieldBundleAdd(Bundle, Field, rc=RC )
--------------^
compilation aborted for atmos_phy_chem_cpl_comp_mod.f90 (code 1)

Is this a known issue?  

From: John Michalakes [ <mailto:john.michalakes at noaa.gov>
mailto:john.michalakes at noaa.gov] 
Sent: Monday, July 14, 2014 5:15 PM
To: 'ncep.list.nems.announce at lstsrv.ncep.noaa.gov'
Subject: Proposed NEMS commit for Performance Improvements to RRTM-G

Hi all,

I would like to start the process for obtaining approval to commit
performance related changes to the RRTMG in the NEMS trunk.

Performance improvement:

Performance improvement on Zeus for then NMM_CNTRL workload:

                Original (revision 42943)  :    93.5s      (0.794s)

                  With optimized RRTMG:     89.2s      (0.739s)

The first times is the time to run the atmospheric component from beginning
to end (that is, the time spent in the ATM_RUN subroutine).  In parentheses
is the time per radiation call averaged over the 1125 calls (25 calls on 45
MPI task).

Performance improvement is configuration and workload dependent and will be
the subject of ongoing work. With regard to the GFS cases, the RRTM changes
degrade performance a little bit, but assuming output is okay, I'm
requesting approval to commit the changes into the trunk now and then work
more on improving performance afterwards.  (I've been spending a lot of time
and energy keeping up with changes to the trunk).

Verification:  

The attached pdf file is a small sampling of "old-new" grads difference
plots at 03, 12, and 48 hours.  There are differences but generally
"snow-like" (more or less random) especially at the 03hr time period.

Since the new output does not agree bit-for-bit, I modified the regression
test script to report the difference and continue on through the next tests.
All 13 tests run to completion:

GFS_16_1_dfi
GFS_16_32
GFS_32
GFS_GOCART_NEMSIO
NMM_CNTRL
NMM_DECOMP
NMM_nest_rest
NMM_REG_gfsP
NMM_REG_NEMSIO
NMM_REG_RST
NMM_REST_NIO
NMM_THREAD
WAM_gh_l150

but at this point the only output I've looked at is the NMM_CNTRL test for
NMM.

I would like to also do difference plots for the GFS cases but I'm not sure
how to do that since the output isn't grads (as far as I can tell).  I'm
looking for suggestions, please.

Code location and summary of changes:  

The code is my home directory on Zeus:

/home/John.Michalakes/jm_proposal_20140715 

and is relative to Revision: 42438  (thus, a little bit behind the top of
the trunk right now, which is at Revision: 42943, but that's just Weiyu's
new changes to the regression scripts).  

A summary of the modified files is here:

M       src/atmos/nmm/module_DIAGNOSE.F90
M       src/atmos/nmm/module_RADIATION.F90
M       src/atmos/nmm/module_SOLVER_GRID_COMP.F90
M       src/atmos/phys/grrad.f
M       src/atmos/phys/grrad_nmmb.f
M       src/atmos/phys/machine.f
M       src/atmos/phys/makefile
M       src/atmos/phys/module_RA_RRTM.F90
M       src/atmos/phys/radiation_aerosols_nmmb.f
M       src/atmos/phys/radlw_main.f
M       src/atmos/phys/radsw_main.f
M       src/conf/configure.nems.Gaea.intel
M       src/conf/configure.nems.Gaea.pgi
M       src/conf/configure.nems.Jet.ifort
M       src/conf/configure.nems.Linux.g95
M       src/conf/configure.nems.Linux.gnu
M       src/conf/configure.nems.Linux.intel
M       src/conf/configure.nems.Linux.pgi
M       src/conf/configure.nems.Unicos.intel
M       src/conf/configure.nems.Wcoss.intel
M       src/conf/configure.nems.Wcoss.intel_ESMF_520rp1
M       src/conf/configure.nems.Wcoss.intel_ESMF_520rp2
M       src/conf/configure.nems.Wcoss.intel_ESMF_630r
M       src/conf/configure.nems.Yellowstone.intel
M       src/conf/configure.nems.Zeus.intel
M       src/esmf_version
M       src/makefile
M       src/module_NEMS_GRID_COMP.F90

Performing status on external item at 'src/atmos/gsm':
M       src/atmos/gsm/phys/gloopr.f

Requested action by group:

I would appreciate it if those interested could please review and get back
to me with questions, suggestions and concerns within the next week.   Then,
based on the resulting discussions, I would hope to have the modifications
committed to the trunk and passing the reg-tests (with a new set of
reference data sets) by the end of this month (July).

Thanks

John 
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://lstsrv.ncep.noaa.gov/pipermail/ncep.list.nems.announce/attachments/20140716/f6d1d81d/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: RRTMG plots.pdf
Type: application/pdf
Size: 470217 bytes
Desc: not available
Url : https://lstsrv.ncep.noaa.gov/pipermail/ncep.list.nems.announce/attachments/20140716/f6d1d81d/attachment-0001.pdf 