<div dir="ltr">Jim,<div><br></div><div>FV_DIAG counts the time for the fv_diagnstics.F90 and not any diagnostics related to GFS physics. I don't know if you have changed to namelist parameter to dump diagnostics when they are created instead of buffering them. If you are dumping data to disk as it is created, then I expect variability some variability. Another source of variability is load-imbalance from previous segments of the model as there is some communication being done in the fv3 diagnostics that will act as an implicit barrier.</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="font-size:12.8px">Rusty</div><div style="font-size:12.8px">--</div><div style="font-size:12.8px">Rusty Benson, PhD</div><div style="font-size:12.8px">Modeling Systems Group</div><div style="font-size:12.8px">NOAA Geophysical Fluid Dynamics Lab</div><div style="font-size:12.8px">Princeton, NJ</div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Fri, May 12, 2017 at 3:07 PM, James Abeles - NOAA Affiliate <span dir="ltr"><<a href="mailto:james.a.abeles@noaa.gov" target="_blank">james.a.abeles@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Moorthi,<div>I ran the tests with a simple C96 grid for 48 hours and the one that compile GFS_diagnostics with -O0 is actually faster:</div><div><div>wrk.C96/out.96:0: FV_DIAG 29.1041 0.0000 29.1041 0</div><div>wrk.C96.diag_O0/out.96:0: FV_DIAG 14.2025 0.0000 14.2025 0</div></div><div><br></div><div>Now I need to figure out if we can compile some routines with different flags.</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 12, 2017 at 3:03 PM, Shrinivas Moorthi <span dir="ltr"><<a href="mailto:shrinivas.moorthi@noaa.gov" target="_blank">shrinivas.moorthi@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
<div class="m_8312822555572549499m_-2345894083188714791moz-cite-prefix">Thanks Jim.<br>
Moorthi<br>
On 05/12/2017 02:55 PM, James Abeles - NOAA Affiliate wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr">Resending with correct -O0 flag.
<div><br>
</div>
<div><span style="font-size:12.8px">Moorthi,</span>
<div style="font-size:12.8px">The slow compile is
GFS_diagnostics.F90. I have tried many things to reduce the
compile time but the only one that worked was compiling at
-O0. There is really nothing being done in this routine
other than storing diagnostices so I am testing the
performance with -O0 vs -O2.</div>
<div style="font-size:12.8px"><br>
</div>
<div style="font-size:12.8px"><b><font size="4">-O2</font></b></div>
<div style="font-size:12.8px">
<div>/gpfs/hps/emc/global/noscrub/J<wbr>ames.A.Abeles/fv3gfs.v0beta/FV<wbr>3/gfsphysics>
time ftn -DNEW_TAUCTMAX -DSMALL_PE -DNEMS_GSM -fpp -Wp,-w
-I/opt/cray/netcdf/4.3.2/INTEL<wbr>/140/include
-I/opt/cray/netcdf/4.3.2/INTEL<wbr>/140/include -fno-alias
-auto -safe-cray-ptr -ftz -assume byterecl -nowarn -sox
-align array64byte -i4 -real-size 32 -O2 -debug minimal
-fp-model source -qoverride-limits -I../fms
-I../fms/include -c GFS_layer/GFS_diagnostics.F90 -o
GFS_layer/GFS_diagnostics.o</div>
<div><br>
</div>
<div>real 10m3.761s</div>
<div>user 10m1.842s</div>
<div>sys 0m0.396s</div>
<div><br>
</div>
<div><b><font size="4">-O0</font></b></div>
<div>/gpfs/hps/emc/global/noscrub/J<wbr>ames.A.Abeles/fv3gfs.v0beta/FV<wbr>3/gfsphysics>
time ftn -DNEW_TAUCTMAX -DSMALL_PE -DNEMS_GSM -fpp -Wp,-w
-I/opt/cray/netcdf/4.3.2/INTEL<wbr>/140/include
-I/opt/cray/netcdf/4.3.2/INTEL<wbr>/140/include -fno-alias
-auto -safe-cray-ptr -ftz -assume byterecl -nowarn -sox
-align array64byte -i4 -real-size 32 -O0 -debug minimal
-fp-model source -qoverride-limits -I../fms
-I../fms/include -c GFS_layer/GFS_diagnostics.F90 -o
GFS_layer/GFS_diagnostics.o</div>
<div>real 0m0.981s</div>
<div>user 0m0.728s</div>
<div>sys 0m0.072s</div>
</div>
<div style="font-size:12.8px"><br>
</div>
<div class="gmail_extra" style="font-size:12.8px"><br>
<div class="gmail_quote"><br><span class="m_8312822555572549499HOEnZb"><font color="#888888">
</font></span></div><span class="m_8312822555572549499HOEnZb"><font color="#888888">
</font></span></div><span class="m_8312822555572549499HOEnZb"><font color="#888888">
</font></span></div><span class="m_8312822555572549499HOEnZb"><font color="#888888">
</font></span></div><span class="m_8312822555572549499HOEnZb"><font color="#888888">
</font></span></blockquote><span class="m_8312822555572549499HOEnZb"><font color="#888888">
<br><span class="HOEnZb"><font color="#888888">
<p><br>
</p>
<pre class="m_8312822555572549499m_-2345894083188714791moz-signature" cols="72">--
Dr. Shrinivas Moorthi
Research Meteorologist
Modeling and Data Assimilation Branch
Environmental Modeling Center / National Centers for Environmental Prediction
5830 University Research Court - (W/NP23), College Park MD 20740 USA
Tel:<a href="tel:(301)%20683-3718" value="+13016833718" target="_blank">(301)683-3718</a></pre>
</font></span></font></span></div>
</blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
Ncep.list.fv3-announce mailing list<br>
<a href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.<wbr>ncep.noaa.gov</a><br>
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce" rel="noreferrer" target="_blank">https://www.lstsrv.ncep.noaa.<wbr>gov/mailman/listinfo/ncep.<wbr>list.fv3-announce</a><br>
<br></blockquote></div><br></div>