<div dir="ltr">Here are a few other helpful debugging tips...<div><br></div><div>For those who are seeing issues with reproducibility with the FV3GFS with IPD, there is a namelist option to help narrow down potential sources of error: atmos_model_nml::chksum_debug.<div>Setting this flag to .true. will print out the global checksum of most every variable in the IPD_Data sub-containers after each of the major steps: radiation; physics; stochastic physics; and the atmospheric state update. We use frequently use techniques such as this at GFDL to help narrow down reproducibility issues under various scenarios including code updates.<br></div><div><br></div><div>As has been alluded to, there are also ways to run the system in physics-only mode, which is analogous to a global domain "single-column model". Simply setting fv_core_nml::no_dycore=.true. will bypass the advection phase of the dynamic core.</div><div><br></div><div>To run the system in dynamics-only mode, set atmos_model_nml::dycore_only=.true.</div><div><br></div><div>Setting both atmos_model_nml::dycore_only and fv_core_nml::no_dycore to .true. will simply exercise the framework.</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="font-size:12.8px">Rusty</div><div style="font-size:12.8px">--</div><div style="font-size:12.8px">Rusty Benson, PhD</div><div style="font-size:12.8px">Modeling Systems Group</div><div style="font-size:12.8px">NOAA Geophysical Fluid Dynamics Lab</div><div style="font-size:12.8px">Princeton, NJ</div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Mon, Apr 10, 2017 at 10:33 AM, Shrinivas Moorthi <span dir="ltr"><<a href="mailto:shrinivas.moorthi@noaa.gov" target="_blank">shrinivas.moorthi@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div text="#000000" bgcolor="#FFFFFF">
<div class="m_2761746783453173330moz-cite-prefix">Thanks all,<br>
I am compiling with the DEBUG=Y option and it is using the
option suggested by Jim.<br>
However, I haven't had any success figuring out yet. I am trying
to debug with "no_dycore=.true." as suggested by SJ.<br>
Moorthi<br>
<br>
On 04/10/2017 10:13 AM, Dusan Jovic - NOAA Affiliate wrote:<br>
</div>
<blockquote type="cite">
<div dir="ltr"><span style="font-size:12.8px">Moorthi,</span>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px"> If you are using standalone
FV3, the configuration files already contain DEBUG flag
which turns on many debugging compiler flags. Simply run:</span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px"><font face="monospace,
monospace">make DEBUG=Y</font></span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">or if you are using compile
script (in the top-level directory of the standalone fv3
repository) </span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px"><font face="monospace,
monospace">./compile DEBUG=Y</font></span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">I'm not sure how this option
can be used from NEMS (ie. nems builder). </span></div>
<div><span style="font-size:12.8px"><br>
</span></div>
<div><span style="font-size:12.8px">Dusan</span></div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Apr 9, 2017 at 3:26 PM, James
Rosinski - NOAA Affiliate <span dir="ltr"><<a href="mailto:james.rosinski@noaa.gov" target="_blank">james.rosinski@noaa.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>FYI if you are using the Intel compiler, the flag
-fpe0 will cause the run to stop when a NaN is
generated. Adding "-traceback -g" will give a
human-readable traceback when the run aborts.<br>
<br>
</div>
Jim Rosinski<br>
<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Sun, Apr 9, 2017 at 10:34 AM,
Shian-Jiann Lin - NOAA Federal <span dir="ltr"><<a href="mailto:shian-jiann.lin@noaa.gov" target="_blank">shian-jiann.lin@noaa.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Not sure there is a general way to
diagnose NaN, as this can coe from any place.
<div>But a helpful option is to set</div>
<div><br>
</div>
<div>no_dycore = .T.</div>
<div><br>
</div>
<div>and run the whole model like multi-column
physics only (no FV3) model.</div>
<div><br>
</div>
<div>You may also set fv_debug = .T. to see where
the Nan started.</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943gmail_signature" data-smartmail="gmail_signature">
<div dir="ltr">
<div>
<div dir="ltr"><span><font color="#888888"><br>
Shian-Jiann Lin<br>
NOAA/Geophysical Fluid Dynamics
Laboratory<br>
</font></span></div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Sun, Apr 9, 2017 at
12:50 PM, Shrinivas Moorthi - NOAA Federal <span dir="ltr"><<a href="mailto:shrinivas.moorthi@noaa.gov" target="_blank">shrinivas.moorthi@noaa.gov</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="auto">
<div>SJ,</div>
<div id="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131AppleMailSignature">Any
suggestion on how to track a NAN in fv3?</div>
<div id="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131AppleMailSignature">Thanks</div>
<div id="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131AppleMailSignature">Moorthi<br>
<br>
Sent from my iPhone</div>
<div><br>
On Apr 9, 2017, at 1:53 AM, Shian-Jiann Lin
- NOAA Federal <<a href="mailto:shian-jiann.lin@noaa.gov" target="_blank">shian-jiann.lin@noaa.gov</a>>
wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div>If you only have the raw data with 6
tiles you must perform a cube2latlon
regrid.</div>
<div><br>
<div class="gmail_quote">
<div>On Sun, 9 Apr 2017 at 5:21 AM
Ruiyu Sun - NOAA Affiliate <<a href="mailto:ruiyu.sun@noaa.gov" target="_blank">ruiyu.sun@noaa.gov</a>>
wrote:<br>
</div>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">Hi
guys, <br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</div>
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg"><br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</div>
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">Can
anyone tell me how to calculate
the global zonal mean of cloud
condensate based on the 6
nggps3d tile nc files ? <br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</div>
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">Thanks!<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</div>
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">Ruiyu<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</div>
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg"><br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<a href="http://nggps3d.tile1.nc" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">nggps3d.tile1.nc</a>,
<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<a href="http://nggps3d.tile2.nc" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">nggps3d.tile2.nc</a>,<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<a href="http://nggps3d.tile3.nc" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">nggps3d.tile3.nc</a>,
<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<a href="http://nggps3d.tile4.nc" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">nggps3d.tile4.nc</a>,<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<a href="http://nggps3d.tile5.nc" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">nggps3d.tile5.nc</a>,
<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<a href="http://nggps3d.tile6.nc" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">nggps3d.tile6.nc</a>,
</div>
</div>
<div class="m_2761746783453173330gmail_extra
m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg"><br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<div class="m_2761746783453173330gmail_quote
m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">On
Fri, Apr 7, 2017 at 9:19 AM, Jun
Wang - NOAA Affiliate <span class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg"><<a href="mailto:jun.wang@noaa.gov" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">jun.wang@noaa.gov</a>></span>
wrote:<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<blockquote class="m_2761746783453173330gmail_quote
m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<div class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">Dear
all,<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</div>
This is a test email,
please ignore. Thanks.<span class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131m_7870984784518225018HOEnZb
m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg"><font class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" color="#888888"><br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</font></span></div>
<span class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131m_7870984784518225018HOEnZb
m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg"><font class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" color="#888888">Jun<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</font></span></div>
<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
______________________________<wbr>_________________<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
Ncep.list.fv3-announce mailing
list<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<a href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">Ncep.list.fv3-announce@lstsrv.<wbr>ncep.noaa.gov</a><br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce" rel="noreferrer" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">https://www.lstsrv.ncep.noaa.g<wbr>ov/mailman/listinfo/ncep.list.<wbr>fv3-announce</a><br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</blockquote>
</div>
<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</div>
______________________________<wbr>_________________<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
Ncep.list.fv3-announce mailing list<br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<a href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">Ncep.list.fv3-announce@lstsrv.<wbr>ncep.noaa.gov</a><br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce" rel="noreferrer" class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg" target="_blank">https://www.lstsrv.ncep.noaa.g<wbr>ov/mailman/listinfo/ncep.list.<wbr>fv3-announce</a><span class="m_2761746783453173330m_-1685408001050068473HOEnZb"><font color="#888888"><span class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943HOEnZb"><font color="#888888"><br class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943m_-6337630025002008131gmail_msg">
</font></span></font></span></blockquote>
</div>
</div>
<span class="m_2761746783453173330m_-1685408001050068473HOEnZb"><font color="#888888"><span class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943HOEnZb"><font color="#888888">
<div dir="ltr">-- <br>
</div>
<div data-smartmail="gmail_signature">Sent
from Gmail Mobile</div>
</font></span></font></span></div>
</blockquote>
<span class="m_2761746783453173330m_-1685408001050068473HOEnZb"><font color="#888888"><span class="m_2761746783453173330m_-1685408001050068473m_-2269832990582603943HOEnZb"><font color="#888888">
<blockquote type="cite">
<div><span>______________________________<wbr>_________________</span><br>
<span>Ncep.list.fv3-announce
mailing list</span><br>
<span><a href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov" target="_blank">Ncep.list.fv3-announce@lstsrv.<wbr>ncep.noaa.gov</a></span><br>
<span><a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce" target="_blank">https://www.lstsrv.ncep.noaa.g<wbr>ov/mailman/listinfo/ncep.list.<wbr>fv3-announce</a></span><br>
</div>
</blockquote>
</font></span></font></span></div>
</blockquote>
</div>
<br>
</div>
<br>
______________________________<wbr>_________________<br>
Ncep.list.fv3-announce mailing list<br>
<a href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov" target="_blank">Ncep.list.fv3-announce@lstsrv.<wbr>ncep.noaa.gov</a><br>
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce" rel="noreferrer" target="_blank">https://www.lstsrv.ncep.noaa.g<wbr>ov/mailman/listinfo/ncep.list.<wbr>fv3-announce</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
______________________________<wbr>_________________<br>
Ncep.list.fv3-announce mailing list<br>
<a href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov" target="_blank">Ncep.list.fv3-announce@lstsrv.<wbr>ncep.noaa.gov</a><br>
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce" rel="noreferrer" target="_blank">https://www.lstsrv.ncep.noaa.g<wbr>ov/mailman/listinfo/ncep.list.<wbr>fv3-announce</a><br>
<br>
</blockquote>
</div>
<br>
</div>
<br>
<fieldset class="m_2761746783453173330mimeAttachmentHeader"></fieldset>
<br>
<pre>______________________________<wbr>_________________
Ncep.list.fv3-announce mailing list
<a class="m_2761746783453173330moz-txt-link-abbreviated" href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov" target="_blank">Ncep.list.fv3-announce@lstsrv.<wbr>ncep.noaa.gov</a>
<a class="m_2761746783453173330moz-txt-link-freetext" href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce" target="_blank">https://www.lstsrv.ncep.noaa.<wbr>gov/mailman/listinfo/ncep.<wbr>list.fv3-announce</a><span class="HOEnZb"><font color="#888888">
</font></span></pre><span class="HOEnZb"><font color="#888888">
</font></span></blockquote><span class="HOEnZb"><font color="#888888">
<br>
<p><br>
</p>
<pre class="m_2761746783453173330moz-signature" cols="72">--
Dr. Shrinivas Moorthi
Research Meteorologist
Global Climate and Weather Modeling Branch
Environmental Modeling Center / National Centers for Environmental Prediction
5830 University Research Court - (W/NP23), College Park MD 20740 USA
Tel:<a href="tel:(301)%20683-3718" value="+13016833718" target="_blank">(301)683-3718</a></pre>
</font></span></div>
<br>______________________________<wbr>_________________<br>
Ncep.list.fv3-announce mailing list<br>
<a href="mailto:Ncep.list.fv3-announce@lstsrv.ncep.noaa.gov">Ncep.list.fv3-announce@lstsrv.<wbr>ncep.noaa.gov</a><br>
<a href="https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.fv3-announce" rel="noreferrer" target="_blank">https://www.lstsrv.ncep.noaa.<wbr>gov/mailman/listinfo/ncep.<wbr>list.fv3-announce</a><br>
<br></blockquote></div><br></div></div>