<div dir="ltr">Additional updates:<div><br></div><div>- Free-forecast mode not working after SLURM update, patch for that is in the works and will come back to master ASAP!</div><div>- The rocoto_viewer no longer works on Theia after SLURM update, there is no support for this script currently so until we can either replace or fix please use rocoto commands to view the status of your experiments on Theia. See slide 43 of the FV3GFS how-to for the most used commands or other online rocoto command documentation for the full set:</div><div><br></div><div><a href="https://docs.google.com/presentation/d/1DegO_WU8i_3BcyaxrbXaeQ2w3s-yQgLNRNNyaZMMEJk/edit#slide=id.g2c24c0b1fb_0_1">https://docs.google.com/presentation/d/1DegO_WU8i_3BcyaxrbXaeQ2w3s-yQgLNRNNyaZMMEJk/edit#slide=id.g2c24c0b1fb_0_1</a> <br clear="all"><div><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><font color="#000000"><br></font></div><div><font color="#000000">Thanks!</font></div><div dir="ltr"><font color="#000000"><br></font></div><div dir="ltr"><font color="#000000">Kate Friedman (formerly Howard)<br>NOAA/NWS/NCEP/EMC Engineering and Implementation Branch</font><br></div></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 10, 2019 at 4:59 PM Kate Friedman - NOAA Federal <<a href="mailto:Kate.Friedman@noaa.gov">Kate.Friedman@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Additionally the following locally installed copies of the global-workflow master have been updated:<div><br></div><div>Theia: /scratch4/NCEPDEV/global/save/glopara/git/global-workflow/master <br></div><div>WCOSS-Dells: /gpfs/dell2/emc/modeling/noscrub/emc.glopara/git/global-workflow/master</div><div><div><div dir="ltr" class="gmail-m_-4737741006486590379gmail-m_-3615904627028201120gmail_signature"><div dir="ltr"><div><div dir="ltr"><font color="#000000"><br></font></div><div dir="ltr"><font color="#000000">Kate Friedman (formerly Howard)<br>NOAA/NWS/NCEP/EMC Engineering and Implementation Branch</font><br></div></div></div></div></div><br></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, May 10, 2019 at 4:11 PM Kate Friedman - NOAA Federal <<a href="mailto:Kate.Friedman@noaa.gov" target="_blank">Kate.Friedman@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><h1 style="margin:0px;line-height:1.4em"><font face="arial, helvetica, sans-serif" size="2" style="font-weight:normal">All,</font></h1><div>Thank you for your patience! The SLURM updates for the global-workflow are now in its master! Thank you to everyone who worked hard to test and resolve any final issues this week! Some additional action items were produced during the final testing phase so look for that mentioned below and in upcoming commits. Please let me know if you have any issues with the workflow after this commit.</div><div><br></div><div>One very important note is that some developers (our workflow team included) noticed differences in SLURM runs done before and after this week's Theia maintenance. These differences are NOT seen between repeated runs done after the maintenance with the exact same inputs and settings. These differences are very small (butterflies) but will be discussed with the machine admins. Any needed fixes will be implemented ASAP.</div><h1 style="font-family:"Trebuchet MS",Verdana,sans-serif;margin:0px;font-size:1.3em;line-height:1.4em"><br></h1><h1 style="font-family:"Trebuchet MS",Verdana,sans-serif;margin:0px;font-size:1.3em;line-height:1.4em"><a href="https://vlab.ncep.noaa.gov/redmine/news/767" style="color:rgb(17,102,153)" target="_blank">SLURM workflow updates for R&D machines</a></h1><p style="font-family:Verdana,sans-serif;font-size:14px">SLURM workflow updates for R&D machines (Theia - no impact on WCOSS)</p><p style="font-family:Verdana,sans-serif;font-size:14px">Redmine Issue: <a class="gmail-m_-4737741006486590379gmail-m_-3615904627028201120gmail-m_-5616525901425166409gmail-m_8086629938378542421external" href="https://vlab.ncep.noaa.gov/redmine/issues/58894" style="color:rgb(17,102,153)" target="_blank">https://vlab.ncep.noaa.gov/redmine/issues/58894</a></p><p style="font-family:Verdana,sans-serif;font-size:14px">Summary of changes:</p><ul style="font-family:Verdana,sans-serif;font-size:14px"><li style="margin-left:15px">env/THEIA.env - change launcher format for SLURM</li><li style="margin-left:15px">modulefiles/module_base.theia - updates to add slurm prod_util module</li><li style="margin-left:15px">ush/rocoto/rocoto.py - added partition statements</li><li style="margin-left:15px">ush/rocoto/setup_expt.py - set icsdir option requirement to false</li><li style="margin-left:15px">ush/rocoto/setup_workflow.py - add SLURM checks for Theia parts</li><li style="margin-left:15px">ush/rocoto/workflow_utils.py - add SLURM checks for Theia parts</li><li style="margin-left:15px">util/sorc/compile_gfs_util_wcoss.sh - small fixes</li><li style="margin-left:15px">updated fit2obs jobcard for SLURM</li></ul><p style="font-family:Verdana,sans-serif;font-size:14px">Things still to address in later commits:</p><ul style="font-family:Verdana,sans-serif;font-size:14px"><li style="margin-left:15px">Downstream jobs (e.g. gempak) are not yet supported/tested on R&D machines or under SLURM</li></ul><p style="font-family:Verdana,sans-serif;font-size:14px">Caveats:</p><ul style="font-family:Verdana,sans-serif;font-size:14px"><li style="margin-left:15px">Differences have been observed between SLURM tests done before and after May 7th Theia maintenance. This will be investigated and any necessary fixes will be tested and committed to the global-workflow master ASAP. Please report any similar behavior observed!</li></ul><p style="font-family:Verdana,sans-serif;font-size:14px">How to incorporate changes into your own copy. Your options:</p><ol style="font-family:Verdana,sans-serif;font-size:14px"><li style="margin-left:15px">Sync merge the global-workflow master into your branch (preferred/advised option)</li><li style="margin-left:15px">Apply changes via patch:<p>See changes in following file on Theia:</p><p>/scratch4/NCEPDEV/global/save/glopara/utilities/global-workflow_slurm_master.diff</p><p>Apply changes to your branch while within clone:</p><p>git apply --reject --whitespace=fix /scratch4/NCEPDEV/global/save/glopara/utilities/global-workflow_slurm_master.diff</p></li></ol><div><div dir="ltr" class="gmail-m_-4737741006486590379gmail-m_-3615904627028201120gmail-m_-5616525901425166409gmail_signature"><div dir="ltr"><div><div dir="ltr"><font color="#000000"><br></font></div><div dir="ltr"><font color="#000000">Kate Friedman (formerly Howard)<br>NOAA/NWS/NCEP/EMC Engineering and Implementation Branch</font><br></div></div></div></div></div></div>
</blockquote></div>
</blockquote></div>