<div dir="ltr"><div dir="ltr">Hello all,<div><br></div><div>The commit is in. Please let me know if you find any problems.</div><div><br></div><div>The compset logs will not reflect that fv3_lheatstrg has been run because I ran it in a separate test, just before my prior email. </div><div><br></div><div>Sincerely,</div><div>Sam Trahan</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 11 Feb 2019 at 12:54, Samuel Trahan - NOAA Affiliate <<a href="mailto:samuel.trahan@noaa.gov">samuel.trahan@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">Hello all,<div><br></div><div>My commit is taking slightly longer than anticipated. I'm predicting I'll finish by 2 PM. All repositories except NEMSfv3gfs have been updated; I neglected to test the fv3_lheatstrg compset for that app with SLURM. In the unlikely event that it fails, I'll disable that compset for SLURM and commit.</div><div><br></div><div>Sincerely,</div><div>Sam Trahan</div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, 11 Feb 2019 at 10:43, Samuel Trahan - NOAA Affiliate <<a href="mailto:samuel.trahan@noaa.gov" target="_blank">samuel.trahan@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello all,<div><br></div><div>The aforementioned commit will happen between about noon and 1 PM Eastern time today.</div><div><br></div><div>As a reminder, this commit will only affect you if:</div><div><br></div><div>1. You are running the multi-machine, multi-app, regression test system that we use for huge NEMS commit and nightly tests, or</div><div><br></div><div>2. You are annoyed by the spurious error messages from the "make clean" of FMS in the NEMSfv3gfs's compile.sh</div><div><br></div><div>Sincerely,</div><div>Sam Trahan</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 8 Feb 2019 at 15:11, Samuel Trahan - NOAA Affiliate <<a href="mailto:samuel.trahan@noaa.gov" target="_blank">samuel.trahan@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div>Hello all,</div><div><br></div><div>Update based on feedback and testing:</div><div><br></div><div>1. I have updated the SLURM vs. Torque logic so you will not have to specify your desired target on Theia. The code now assumes you are using SLURM if "sbatch" is in your environment, and will assume you want Moab/Torque otherwise. As long as the default environment on Jet and Theia doesn't change, this should be sufficient.<br></div><div><br></div><div>2. On Theia, all Moab/Torque compsets match baselines when run in SLURM.</div><div><br></div><div>3. On Jet, all Moab/Torque compsets match baselines when run in SLURM -- except -- the fv3_wrtGauss_nemsio_c768 which hangs. With that one, the FV3 prints nothing, hangs forever, and an error message can be seen in the system logs suggesting a Mellanox firmware bug. I submitted a ticket a few months ago and never heard back from admins. Until this is fixed, that compset is disabled on uJet SLURM. It is possible this problem is specific to uJet, not to SLURM. The Moab/Torque tests are run on tJet right now, because all of uJet is reserved for SLURM. While the t and u Jets are supposed to be identical, that isn't necessarily the case. Soon, parts of xJet will be available to SLURM, and we may find the answer. Note that this compset is the closest one to the operational configuration; it differs just in the physics selection.</div><div><br></div><div>Sincerely,</div><div>Sam Trahan<br></div><div><br></div><div><br></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, 8 Feb 2019 at 12:23, Samuel Trahan - NOAA Affiliate <<a href="mailto:samuel.trahan@noaa.gov" target="_blank">samuel.trahan@noaa.gov</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi all,</div><div><br></div><div>A NEMS master commit is coming in soon; this is a purely technical one. The NCEPLIBS-pyprodutil master and Supported apps' masters will be updated as well. The relevant branch is called "slurm" in NEMS and NEMSfv3gfs; and "slurm-v2" in NCEPLIBS-pyprodutil.<br></div><div><br></div><div>1. SLURM support for NEMSfv3gfs app's NEMSCompsetRun on uJet and Theia (see notes below). Results match the Moab/Torque baselines.<br></div><div><br></div><div>2. Bug fix from Dusan Jovic to eliminate error messages when cleaning FMS, and remove one temporary file created during the cleaning process.</div><div><br></div><div>3. Major bug fix to the multi-app test system to allow multiple, multi-app, tests, to happen at the same time. This bug was causing the nightly test website to incorrectly report some branch-specific tests people were doing as the nightly test results. The change adds a "test id" that is passed around; the nightly test uses "ngt".</div><div><br></div><div><br></div><div>SLURM porting details:<br></div><div><br></div><div><br></div><div>1. From now on, when running NEMSfv3gfs NEMSCompsetRun on Theia, you will have to specify whether you want a MOAB or SLURM test. The NEMSCompsetRun will complain if you don't.<br></div><div><br></div><div>To run with Moab/Torque: NEMSCompsetRun --platform theia.intel ...</div><div>To run with SLURM: NEMSCompsetRun --platform theia.slurm.intel ...</div><div><br></div><div>Once Moab/Torque are gone, the theia.slurm.intel will be removed, and theia.intel will use SLURM.<br></div><div><br></div><div>2. On Jet, only uJet has SLURM. We're expecting parts of xJet to be SLURMified soon, at which point we can add that target.<br></div><div><br></div><div>3. On Theia, the SLURM is misconfigured to think there are only 12 cores per node instead of 24 when task geometries are requested. I've compensated by telling the nightly tests that there are only 12 cores per node, which doubles the number of nodes we use. To avoid pounding the machine TOO hard, the Theia SLURM "nightly" tests will only run once a week. This can be changed once the admins fix the SLURM misconfiguration.</div><div><br></div><div>4. For now, we're putting the GAEA SLURM port on hold. This is because GAEA's SLURM configuration may be undergoing a major change in the near future. Presently it has a very non-standard configuration which would require extra effort to support. The new configuration may require very different extra effort, and we don't want to do that twice.</div><div><br></div><div>Sincerely,</div><div>Sam Trahan<br></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>