<div dir="ltr">Adam,<div><br></div><div>Yes, your run is completed successfully. Congratulations, Adam!</div><div><br></div><div>What Adam did was to change "pe_node_f=6" (which was hardwired in the original config file, which happens to be the same as on wcoss) to "pe_node_f=$((pe_node/nth_f1))" (which becomes pe_node_f=24 in this case on theia): </div><div><br></div><div><div>#pe_node_f=6</div><div># Adam's suggestion</div><div> pe_node_f=$((pe_node/nth_f1))</div></div><div><br></div><div>Now we can move on to start to do coupled wam-ipe test runs with wdas-iau cycling on theia.</div><div><br></div><div class="gmail_extra">
<div class="gmail_extra">The reason that Adam's run was slow is because we had turned on <br></div><div class="gmail_extra"><br></div><div class="gmail_extra">ESMF_RUNTIME_COMPLIANCECHECK=ON:depth=4</div><div class="gmail_extra"><br></div><div class="gmail_extra">when Gerhard and I were testing, now it is about 6 min for 1 hr fcst:<br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><div class="gmail_extra">-rw-r--r-- 1 swpc.spacepara swpc 19402536 Jun 9 17:46 sigf02.gdas.2016090100</div><div class="gmail_extra">-rw-r--r-- 1 swpc.spacepara swpc 2274170 Jun 9 17:45 flxf02.gdas.2016090100</div><div class="gmail_extra">-rw-r--r-- 1 swpc.spacepara swpc 3177360 Jun 9 17:45 sfcf02.gdas.2016090100</div><div class="gmail_extra">-rw-r--r-- 1 swpc.spacepara swpc 19402536 Jun 9 17:39 sigf01.gdas.2016090100</div><div class="gmail_extra">-rw-r--r-- 1 swpc.spacepara swpc 2279606 Jun 9 17:39 flxf01.gdas.2016090100</div><div class="gmail_extra">-rw-r--r-- 1 swpc.spacepara swpc 3177360 Jun 9 17:39 sfcf01.gdas.2016090100</div><div class="gmail_extra">-rw-r--r-- 1 swpc.spacepara swpc 19402536 Jun 9 17:34 sigf00.gdas.2016090100</div><div class="gmail_extra">-rw-r--r-- 1 swpc.spacepara swpc 2237128 Jun 9 17:34 flxf00.gdas.2016090100</div><div class="gmail_extra">-rw-r--r-- 1 swpc.spacepara swpc 3177360 Jun 9 17:34 sfcf00.gdas.2016090100</div><div><br></div><div>See also the dayfile:</div><div><br></div></div><div class="gmail_extra"><div class="gmail_extra">/scratch4/NCEPDEV/stmp4/swpc.spacepara/prwam.2017/wam2016090100gdasfcst1_34.dayfile</div><div><br></div><div>Houjun</div></div><div class="gmail_extra"><br></div><br><div class="gmail_quote">On Fri, Jun 9, 2017 at 11:03 AM, Adam Kubaryk - NOAA Affiliate <span dir="ltr"><<a href="mailto:adam.kubaryk@noaa.gov" target="_blank">adam.kubaryk@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">I'm not sure why my runs completed, then. I wonder if it has something to do with the processor or node distribution... the only things I changed from your para_config was the same processors-per-node variables that I altered from the IAU para_config. The MEDIATOR start initializeP5 sits for a couple minutes, but clears with MEDIATOR done initializeP5 and then throws "Time to do RegridStore WAM->IPE is 36872.3511695862 msec" before continuing to actually run the model ... see /scratch4/NCEPDEV/stmp4/Adam.<wbr>Kubaryk/pripe/<wbr>ipe2009011500gdasfcst1_3.<wbr>dayfile or /scratch4/NCEPDEV/stmp4/Adam.<wbr>Kubaryk/pripe/<wbr>ipe2017022600gdasfcst1_3.<wbr>dayfile<br><br><br></div><div class="gmail-HOEnZb"><div class="gmail-h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 9, 2017 at 12:53 PM, Houjun Wang - NOAA Affiliate <span dir="ltr"><<a href="mailto:houjun.wang@noaa.gov" target="_blank">houjun.wang@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Adam,<div><br></div><div>The issue with <span style="font-size:12.8px">para_config_wam_ipe and exglobal_fcst_nems.sh.ecf-wam-</span><span style="font-size:12.8px"><wbr>ipe2 is that it will simply hang at </span></div><div><br></div><div><div><span style="font-size:12.8px"> MEDIATOR done initializeP4</span></div><div><span style="font-size:12.8px"> MEDIATOR done initializeP4</span></div><div><span style="font-size:12.8px"> MEDIATOR start initializeP5</span></div><div><span style="font-size:12.8px"> MEDIATOR start initializeP5</span></div><div><span style="font-size:12.8px"> MEDIATOR start initializeP5</span></div><div><span style="font-size:12.8px"> MEDIATOR start initializeP5</span></div><div><span style="font-size:12.8px"> MEDIATOR start initializeP5</span></div><div><span style="font-size:12.8px"> MEDIATOR start initializeP5</span></div><div><span style="font-size:12.8px"> MEDIATOR start initializeP5</span></div><div style="font-size:12.8px"><br></div></div><div style="font-size:12.8px"><div style="font-size:small"><span style="font-size:12.8px">as you can see from many *dayfile in /scratch4/NCEPDEV/stmp4/swp<wbr>c.spacepara/prwam.2017</span></div><div><span style="font-size:12.8px"><br></span></div><div>This is a test for single fcst run. If you'd like, you can give it a try with that config file. Suspect it's some run time env problem. I can do a single coupled run on WCOSS using the corresponding 'same' config.</div><div><br></div><div>Thanks.</div><span class="gmail-m_-4593389730587668260HOEnZb"><font color="#888888"><div><span style="font-size:12.8px"><br></span></div></font></span></div></div><div class="gmail_extra"><span class="gmail-m_-4593389730587668260HOEnZb"><font color="#888888"><br clear="all"><div><div class="gmail-m_-4593389730587668260m_-5284608106652822937gmail_signature"><div dir="ltr"><div><div dir="ltr">Houjun Wang</div></div></div></div></div>
<br></font></span><div class="gmail_quote"><span>On Fri, Jun 9, 2017 at 10:07 AM, Adam Kubaryk - NOAA Affiliate <span dir="ltr"><<a href="mailto:adam.kubaryk@noaa.gov" target="_blank">adam.kubaryk@noaa.gov</a>></span> wrote:<br></span><div><div class="gmail-m_-4593389730587668260h5"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div><div>Hi Houjun,<br><br></div>Can you share the current issues with the para_config_wam_ipe and exglobal_fcst_nems.sh.ecf-wam-<wbr>ipe2? Yesterday I ran an old 1hr compset just to get some baseline expectations of how the model runs when coupled, and I've now tested the para_config file using mostly default settings. It runs coupled on Theia, albeit fairly slowly. A 9hr forecast takes about 65 minutes on 64 processors (although it's not clear that the 64 processors are being fully utilized). There is a substantial wait at the mediator ("MEDIATOR starts InitializeP4" etc...), but that clears after a couple minutes.<br><br>I'm not sure yet if this is the fault of the compliance checker, or if this is simply a property of the model. The mediator does not appear in the compset out or err files, so I'm still looking into this.<br><br></div>Are there IPE output files we should be saving in the ROTDIR from run to run when looking to cycle? I don't yet understand the intracacies of the coupled system, what output files will turn into input files for the next cycle, etc. If you have any details, please let me know.<br><br></div>Thanks,<br></div>Adam<br></div>
</blockquote></div></div></div><br></div>
</blockquote></div><br></div>
</div></div></blockquote></div><br></div></div>