[WAM-IPE] Realtime wam-ipe runs on theia: archive

Thu Jul 27 17:19:49 UTC 2017

Rashid and Houjun,

yes, I can suggest how to change the IPE code in order to stop the run
immediately when a flux tube fails first.

Houjun:
could you please commit your IPE branch to our IPE so that I can easily
search for the differences between your IPE version vs. the IPE master in
the IPE repository on github (please also the instruction on how to work
with WAM-IPE branch:
https://docs.google.com/a/noaa.gov/document/d/1iGSijW5c-y4YW29bXhZhO0SzXUz3Ywjz0e1r-xHI7hk/edit?usp=sharing).
I am going to send you an invitation as a collaborator from github.

Thanks.
naomi

On Tue, Jul 25, 2017 at 5:32 PM, Rashid Akmaev - NOAA Federal <
rashid.akmaev at noaa.gov> wrote:

> Is it possible to catch when a tube first fails? Does the restart from
> MSIS Houjun uses make them fail? Plotting composition after the fact
> probably may not help much as failed tubes will most likely stay that way
> no matter what the composition, right?
>
> Rashid Akmaev
>
> On Jul 25, 2017, at 4:25 PM, Naomi Maruyama - NOAA Affiliate <
> naomi.maruyama at noaa.gov> wrote:
>
> Hi Houjun,
>
> Thanks for your showing some updates at our meeting. Your results look
> very encouraging. But could you please also plot the WAM fields from your
> runs in order to confirm that the WAM fields that are used in IPE are
> reasonable?
>
> In your plotting script, set the VarType to 4 (instead of 7 for NmF2) if
> you want to plot WAM neutral atomic Oxygen, for example. Also you could
> change the altitude of your plot by changing the line, "setenv alt" inside
> the visualization script.
>
> Thanks.
> naomi
>
>
>
>
>
>
> On Tue, Jul 25, 2017 at 2:29 PM, Naomi Maruyama - NOAA Affiliate <
> naomi.maruyama at noaa.gov> wrote:
>
>> Thanks.
>>
>> I see that failed flux tubes in your plots and FLIP_ERR in your runs. But
>> I cannot suggest anything to solve the problem until I can make some test
>> runs.
>>
>> In order to solve this issue, I would need to reproduce your run by
>> myself, and try to debug the run. Could you please suggest how I can do
>> that?
>>
>> Thanks.
>> naomi
>>
>> On Tue, Jul 25, 2017 at 10:30 AM, Houjun Wang - NOAA Affiliate <
>> houjun.wang at noaa.gov> wrote:
>>
>>> yes
>>>
>>> Houjun Wang
>>>
>>> On Tue, Jul 25, 2017 at 10:25 AM, Naomi Maruyama - NOAA Affiliate <
>>> naomi.maruyama at noaa.gov> wrote:
>>>
>>>> Houjun
>>>>
>>>> Thank you very much for archiving the files. I am going to study them.
>>>>
>>>>
>>>>
>>>> >>I have saved the entire rundir for two runs in:
>>>>
>>>> /scratch3/NCEPDEV/swpc/noscrub/swpc.spacepara/archive/wam201
>>>> 7071606gdasfcst1
>>>> /scratch3/NCEPDEV/swpc/noscrub/swpc.spacepara/archive/wam201
>>>> 7071612gdasfcst1
>>>>
>>>> you mentioned at the previous meeting that IPE is slowing down with
>>>> failed flux tubes. Are these two runs have the problem?
>>>>
>>>> Thanks.
>>>> naomi
>>>>
>>>> On Mon, Jul 24, 2017 at 4:45 PM, Houjun Wang - NOAA Affiliate <
>>>> houjun.wang at noaa.gov> wrote:
>>>>
>>>>> In /scratch3/NCEPDEV/swpc/noscrub/swpc.spacepara/archive/ipe
>>>>>
>>>>> 1. The following are already saved:
>>>>>
>>>>>    - plasmaXX:
>>>>>    - ut_rec
>>>>>    - IPE.inp
>>>>>    - stupXX
>>>>>
>>>>> Note: Since stup* files are from previous run, they are already saved
>>>>> in the previous (6 hours earlier) run dir. The soft link in the current run
>>>>> dir are for the NEXT run. I will remove these soft link (and change
>>>>> my run scripts accordingly), because this can be confusing.
>>>>>
>>>>> 2. The following will be saved in the future:
>>>>>
>>>>>    - FLIP_ERR
>>>>>    - logfile_input_params.log
>>>>>
>>>>>
>>>>> 3. The following are saved to mass storage (HPSS): they are called
>>>>> 'dayfile' which contain both output:
>>>>>
>>>>>    - standard output
>>>>>    - standard error output
>>>>>
>>>>>
>>>>> I have saved the entire rundir for two runs in:
>>>>>
>>>>> /scratch3/NCEPDEV/swpc/noscrub/swpc.spacepara/archive/wam201
>>>>> 7071606gdasfcst1
>>>>> /scratch3/NCEPDEV/swpc/noscrub/swpc.spacepara/archive/wam201
>>>>> 7071612gdasfcst1
>>>>>
>>>>> and the dayfile:
>>>>>
>>>>> /scratch3/NCEPDEV/swpc/noscrub/swpc.spacepara/archive/wam201
>>>>> 7071606gdasfcst1_0.dayfile
>>>>> /scratch3/NCEPDEV/swpc/noscrub/swpc.spacepara/archive/wam201
>>>>> 7071612gdasfcst1_0.dayfile
>>>>>
>>>>> and the figs of NmF2 in
>>>>>
>>>>> /scratch3/NCEPDEV/swpc/noscrub/swpc.spacepara/archive/fig/
>>>>>
>>>>> Let me know if I missed anything.
>>>>>
>>>>> (I will compare NmF2 in the forecast and in the initial conditions
>>>>> used for restart.)
>>>>>
>>>>>
>>>>> Houjun Wang
>>>>>
>>>>> On Thu, Jul 20, 2017 at 1:23 PM, Naomi Maruyama - NOAA Affiliate <
>>>>> naomi.maruyama at noaa.gov> wrote:
>>>>>
>>>>>> Houjun and Adam,
>>>>>>
>>>>>> here is a list of files I would like to suggest to be saved under
>>>>>> your archive(/scratch3/NCEPDEV/swpc/noscrub/swpc.spacepara/archiv
>>>>>> e/ipe)
>>>>>>
>>>>>>
>>>>>>    - plasmaXX
>>>>>>    - ut_rec
>>>>>>    - IPE.inp
>>>>>>    - stupXX
>>>>>>    - FLIP_ERR
>>>>>>    - logfile_input_params.log
>>>>>>    - standard output
>>>>>>    - standard error output
>>>>>>
>>>>>>
>>>>>>
>>>>>> Theses files would be helpful if you want me to look at your runs and
>>>>>> verify the results.
>>>>>>
>>>>>> Thanks.
>>>>>> naomi
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 12, 2017 at 2:30 PM, Adam Kubaryk - NOAA Affiliate <
>>>>>> adam.kubaryk at noaa.gov> wrote:
>>>>>>
>>>>>>> Hi Naomi and Houjun,
>>>>>>>
>>>>>>> I'll chime in quickly on the parallel script workflow.
>>>>>>>
>>>>>>> That para_config file is used to submit a job that:
>>>>>>>
>>>>>>> runs /scratch3/NCEPDEV/swpc/svn/gfs/branches/gfs_workflow.v2.0.0/
>>>>>>> para/jobs/fcst.sh
>>>>>>> fcst.sh immediately calls the para_config before running anything
>>>>>>> else
>>>>>>> para_config then calls /scratch3/NCEPDEV/swpc/svn/gfs
>>>>>>> /branches/gfs_workflow.v2.0.0/para/ush/reconcile.sh
>>>>>>> after reconcile.sh, para_config finishes as well, and then fcst.sh
>>>>>>> runs
>>>>>>> fcst.sh calls /scratch3/NCEPDEV/swpc/svn/gfs
>>>>>>> /branches/global_shared.v14.0.0/scripts/exglobal_fcst_nems.s
>>>>>>> h.ecf-wam-ipe_wdas_iau2
>>>>>>>
>>>>>>> A lot of what's relevant to IPE itself (copying in or linking files
>>>>>>> to the RUNDIR, setting up the namelists, etc.) is going to occur in that
>>>>>>> final exglobal script.
>>>>>>>
>>>>>>> Let me know if there's anything else I can clarify.
>>>>>>>
>>>>>>> Best,
>>>>>>> Adam
>>>>>>>
>>>>>>> On Wed, Jul 12, 2017 at 4:22 PM, Houjun Wang - NOAA Affiliate <
>>>>>>> houjun.wang at noaa.gov> wrote:
>>>>>>>
>>>>>>>> 1. Yes, we have hourly output from ipe and since this WDAS-IAU
>>>>>>>> cycled runs, each run only have 12 hour duration (and since ipe doesn't
>>>>>>>> output the last time step - George know this well and may have a fix for it
>>>>>>>> now - it does't have 15 UT for example).
>>>>>>>>
>>>>>>>> So this is not an issue.
>>>>>>>>
>>>>>>>> 2. The source code is in
>>>>>>>>
>>>>>>>> /scratch3/NCEPDEV/swpc/svn/nems/WAM-IPE_NEMS201606_IAU
>>>>>>>>
>>>>>>>> see especially the mods in
>>>>>>>>
>>>>>>>> /scratch3/NCEPDEV/swpc/svn/nems/WAM-IPE_NEMS201606_IAU/IPELI
>>>>>>>> B/src/plasma/io_plasma_bin.f90
>>>>>>>>
>>>>>>>> for initializing ipe from ipe output from previous cycle in
>>>>>>>> WDAS-IAU.
>>>>>>>>
>>>>>>>> I added another option "sw_record_number=2" for this, see IPE.inp
>>>>>>>> for more details and other changes. I also reset record number to zero for
>>>>>>>> each run in WDAS-IAU cycles to make it more practical for this purpose.
>>>>>>>>
>>>>>>>> 3. Now, since I am using EMC para workflow, there is not just one
>>>>>>>> script.
>>>>>>>>
>>>>>>>> The config file I am using is:
>>>>>>>>
>>>>>>>> /scratch3/NCEPDEV/swpc/save/swpc.spacepara/para/para_config_
>>>>>>>> wam_ipe_wdas_iau
>>>>>>>>
>>>>>>>> which defines all the runs.
>>>>>>>>
>>>>>>>> Kate wrote some notes on using EMC para scripts/workflow for
>>>>>>>> wam/wdas:
>>>>>>>>
>>>>>>>> https://docs.google.com/a/noaa.gov/document/d/1OmlGZc0WQy8kd
>>>>>>>> jdPWqHLQZUVn47fXqTn41rbgyrCguc/edit?usp=sharing
>>>>>>>>
>>>>>>>> l'll see if I can answer if you have questions.
>>>>>>>>
>>>>>>>> Houjun Wang
>>>>>>>>
>>>>>>>> On Wed, Jul 12, 2017 at 1:24 PM, Naomi Maruyama - NOAA Affiliate <
>>>>>>>> naomi.maruyama at noaa.gov> wrote:
>>>>>>>>
>>>>>>>>> Houjun,
>>>>>>>>>
>>>>>>>>> Thank you very much for your kindly sharing your output files.
>>>>>>>>>
>>>>>>>>> I have a feeling that your IPE output files were generated from
>>>>>>>>> only 3UT to 14UT only for each day, after looking at some of your IPE
>>>>>>>>> ARCHIVE directory.
>>>>>>>>>
>>>>>>>>> In order to understand this potential issue in further detail, I
>>>>>>>>> would need to understand how your restart of IPE has been set up in your
>>>>>>>>> script and source code. Would you mind sharing your script and source code
>>>>>>>>> with which you have generated these output files?
>>>>>>>>>
>>>>>>>>> Thanks.
>>>>>>>>> naomi
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Tue, Jul 11, 2017 at 3:49 PM, Houjun Wang - NOAA Affiliate <
>>>>>>>>> houjun.wang at noaa.gov> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> Some info on realtime wam-ipe runs on theia:
>>>>>>>>>>
>>>>>>>>>> 1) RUNDIR (this is temporary dir and it could be gone anytime)
>>>>>>>>>>
>>>>>>>>>> /scratch4/NCEPDEV/stmp4/swpc.spacepara/wam2017070506gdasfcst1
>>>>>>>>>>
>>>>>>>>>> 2) ARCHIVE (not all files are saved for ipe; what else should be
>>>>>>>>>> saved?)
>>>>>>>>>>
>>>>>>>>>> /scratch3/NCEPDEV/swpc/noscrub/swpc.spacepara/archive/ipe
>>>>>>>>>>
>>>>>>>>>> 3) HPSS ARCHIVE (George can get ICs for WAM here):
>>>>>>>>>>
>>>>>>>>>> /5year/NCEPDEV/swpc-para/swpc.spacepara/THEIA/prwam_ipe
>>>>>>>>>>
>>>>>>>>>> Let me know if you have any questions or suggestions.
>>>>>>>>>>
>>>>>>>>>> Best,
>>>>>>>>>>
>>>>>>>>>> Houjun Wang
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Naomi Maruyama
>>>>>>>>> CIRES, Univ. of Colorado Boulder
>>>>>>>>> NOAA Space Weather Prediction Center
>>>>>>>>> phone: +1 303.497.4857 <%28303%29%20497-4857>
>>>>>>>>> email: naomi.maruyama at noaa.gov
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> WAM-IPE mailing list
>>>>>>>> WAM-IPE at lstsrv.ncep.noaa.gov
>>>>>>>> https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/wam-ipe
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Naomi Maruyama
>>>>>> CIRES, Univ. of Colorado Boulder
>>>>>> NOAA Space Weather Prediction Center
>>>>>> phone: +1 303.497.4857 <%28303%29%20497-4857>
>>>>>> email: naomi.maruyama at noaa.gov
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>>
>>>> Naomi Maruyama
>>>> CIRES, Univ. of Colorado Boulder
>>>> NOAA Space Weather Prediction Center
>>>> phone: +1 303.497.4857 <%28303%29%20497-4857>
>>>> email: naomi.maruyama at noaa.gov
>>>>
>>>>
>>>
>>
>>
>> --
>>
>> Naomi Maruyama
>> CIRES, Univ. of Colorado Boulder
>> NOAA Space Weather Prediction Center
>> phone: +1 303.497.4857 <%28303%29%20497-4857>
>> email: naomi.maruyama at noaa.gov
>>
>>
>
>
> --
>
> Naomi Maruyama
> CIRES, Univ. of Colorado Boulder
> NOAA Space Weather Prediction Center
> phone: +1 303.497.4857 <%28303%29%20497-4857>
> email: naomi.maruyama at noaa.gov
>
>

-- 

Naomi Maruyama
CIRES, Univ. of Colorado Boulder
NOAA Space Weather Prediction Center
phone: +1 303.497.4857 <%28303%29%20497-4857>
email: naomi.maruyama at noaa.gov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: https://www.lstsrv.ncep.noaa.gov/pipermail/wam-ipe/attachments/20170727/fadd90f6/attachment-0001.html 