[WAM-IPE] WAM-IPE para/exglobal scripts

[Houjun Wang - NOAA Affiliate]

Adam,

Yes, your run is completed successfully. Congratulations, Adam!

What Adam did was to change "pe_node_f=6" (which was hardwired in the
original config file, which happens to be the same as on wcoss) to
"pe_node_f=$((pe_node/nth_f1))" (which becomes pe_node_f=24 in this case on
theia):

#pe_node_f=6
# Adam's suggestion
 pe_node_f=$((pe_node/nth_f1))

Now we can move on to start to do coupled wam-ipe test runs with wdas-iau
cycling on theia.

The reason that Adam's run was slow is because we had turned on

ESMF_RUNTIME_COMPLIANCECHECK=ON:depth=4

when Gerhard and I were testing, now it is about 6 min for 1 hr fcst:

-rw-r--r-- 1 swpc.spacepara swpc   19402536 Jun  9 17:46
sigf02.gdas.2016090100
-rw-r--r-- 1 swpc.spacepara swpc    2274170 Jun  9 17:45
flxf02.gdas.2016090100
-rw-r--r-- 1 swpc.spacepara swpc    3177360 Jun  9 17:45
sfcf02.gdas.2016090100
-rw-r--r-- 1 swpc.spacepara swpc   19402536 Jun  9 17:39
sigf01.gdas.2016090100
-rw-r--r-- 1 swpc.spacepara swpc    2279606 Jun  9 17:39
flxf01.gdas.2016090100
-rw-r--r-- 1 swpc.spacepara swpc    3177360 Jun  9 17:39
sfcf01.gdas.2016090100
-rw-r--r-- 1 swpc.spacepara swpc   19402536 Jun  9 17:34
sigf00.gdas.2016090100
-rw-r--r-- 1 swpc.spacepara swpc    2237128 Jun  9 17:34
flxf00.gdas.2016090100
-rw-r--r-- 1 swpc.spacepara swpc    3177360 Jun  9 17:34
sfcf00.gdas.2016090100

See also the dayfile:

/scratch4/NCEPDEV/stmp4/swpc.spacepara/prwam.2017/wam2016090100gdasfcst1_34.dayfile

Houjun


On Fri, Jun 9, 2017 at 11:03 AM, Adam Kubaryk - NOAA Affiliate <
adam.kubaryk at noaa.gov> wrote:

> I'm not sure why my runs completed, then. I wonder if it has something to
> do with the processor or node distribution... the only things I changed
> from your para_config was the same processors-per-node variables that I
> altered from the IAU para_config. The MEDIATOR start initializeP5 sits for
> a couple minutes, but clears with MEDIATOR done initializeP5 and then
> throws "Time to do RegridStore WAM->IPE is    36872.3511695862      msec"
> before continuing to actually run the model ... see
> /scratch4/NCEPDEV/stmp4/Adam.Kubaryk/pripe/ipe2009011500gdasfcst1_3.dayfile
> or /scratch4/NCEPDEV/stmp4/Adam.Kubaryk/pripe/ipe2017022600gdasfcst1_3.
> dayfile
>
>
>
> On Fri, Jun 9, 2017 at 12:53 PM, Houjun Wang - NOAA Affiliate <
> houjun.wang at noaa.gov> wrote:
>
>> Hi Adam,
>>
>> The issue with para_config_wam_ipe and exglobal_fcst_nems.sh.ecf-wam-ipe2
>> is that it will simply hang at
>>
>>   MEDIATOR done initializeP4
>>   MEDIATOR done initializeP4
>>   MEDIATOR start initializeP5
>>   MEDIATOR start initializeP5
>>   MEDIATOR start initializeP5
>>   MEDIATOR start initializeP5
>>   MEDIATOR start initializeP5
>>   MEDIATOR start initializeP5
>>   MEDIATOR start initializeP5
>>
>> as you can see from many *dayfile in /scratch4/NCEPDEV/stmp4/swp
>> c.spacepara/prwam.2017
>>
>> This is a test for single fcst run. If you'd like, you can give it a try
>> with that config file. Suspect it's some run time env problem. I can do a
>> single coupled run on WCOSS using the corresponding 'same' config.
>>
>> Thanks.
>>
>>
>> Houjun Wang
>>
>> On Fri, Jun 9, 2017 at 10:07 AM, Adam Kubaryk - NOAA Affiliate <
>> adam.kubaryk at noaa.gov> wrote:
>>
>>> Hi Houjun,
>>>
>>> Can you share the current issues with the para_config_wam_ipe and
>>> exglobal_fcst_nems.sh.ecf-wam-ipe2? Yesterday I ran an old 1hr compset
>>> just to get some baseline expectations of how the model runs when coupled,
>>> and I've now tested the para_config file using mostly default settings. It
>>> runs coupled on Theia, albeit fairly slowly. A 9hr forecast takes about 65
>>> minutes on 64 processors (although it's not clear that the 64 processors
>>> are being fully utilized). There is a substantial wait at the mediator
>>> ("MEDIATOR starts InitializeP4" etc...), but that clears after a couple
>>> minutes.
>>>
>>> I'm not sure yet if this is the fault of the compliance checker, or if
>>> this is simply a property of the model. The mediator does not appear in the
>>> compset out or err files, so I'm still looking into this.
>>>
>>> Are there IPE output files we should be saving in the ROTDIR from run to
>>> run when looking to cycle? I don't yet understand the intracacies of the
>>> coupled system, what output files will turn into input files for the next
>>> cycle, etc. If you have any details, please let me know.
>>>
>>> Thanks,
>>> Adam
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/wam-ipe/attachments/20170609/ce37ca07/attachment.html