<div dir="ltr">Sam,<div><br></div><div>Here is the excerpt from the c-shell script we use for bypassing the mpirun wrapper on Jet.</div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote>## when using Intel MPI<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote>## setenv I_MPI_PIN_PROCESSOR_LIST allcores:grain=1,shift=$THREADS<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote>## mpiexec.hydra -n XX --envall ./cubed.x<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote><br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote>## when using MVAPICH2<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> # determine which partition you are on<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> set partitionID = `cat ${PBS_NODEFILE} | cut -c1 | sort -u`<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> switch ( ${partitionID} )<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> case 'n':<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> set cores_per_node = 8<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> breaksw<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> case 't':<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> set cores_per_node = 12<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> breaksw<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> case 'u':<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> set cores_per_node = 12<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> breaksw<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> case 's':<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> set cores_per_node = 16<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> breaksw<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> case 'v':<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> set cores_per_node = 16<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> breaksw<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> case 'x':<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> set cores_per_node = 24<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> breaksw<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> endsw<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> echo "cores_per_node=${cores_per_node}"<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote><br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote># when running with threads, need to use the following command<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> @ npes = ${layout_x} * ${layout_y} * 6<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> setenv HYDRA_ENV all<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> setenv HYDRA_HOST_FILE ${PBS_NODEFILE}<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> @ maxprocnum = ${cores_per_node} - 1<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> setenv MV2_CPU_MAPPING `seq -s: 0 ${nthreads} ${maxprocnum}`<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> set run_cmd = "mpiexec.hydra -np ${npes} ./$executable:t"<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote><br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> setenv MALLOC_MMAP_MAX_ 0<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> setenv MALLOC_TRIM_THRESHOLD_ 536870912<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote> setenv NC_BLKSZ 1M<br><blockquote style="margin:0 0 0 40px;border:none;padding:0px"></blockquote># necessary for OpenMP when using Intel<br> setenv KMP_STACKSIZE 256m<div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="font-size:12.8px">Rusty</div><div style="font-size:12.8px">--</div><div style="font-size:12.8px">Rusty Benson, PhD</div><div style="font-size:12.8px">Modeling Systems Group</div><div style="font-size:12.8px">NOAA Geophysical Fluid Dynamics Lab</div><div style="font-size:12.8px">Princeton, NJ</div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Fri, May 12, 2017 at 1:39 PM, Samuel Trahan - NOAA Affiliate <span dir="ltr"><<a href="mailto:samuel.trahan@noaa.gov" target="_blank">samuel.trahan@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div>Rusty,<br><br></div>Yes, I have also noticed Jet MVAPICH2 to be problematic when placing ranks. I have not worked on optimizing the FV3 on Jet yet; I only made it function. Once we get closer to the point of running FV3 workflows on Jet, we can focus more on optimization.<br><br></div>Sincerely,<br></div>Sam Trahan<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 12, 2017 at 1:31 PM, Rusty Benson - NOAA Federal <span dir="ltr"><<a href="mailto:rusty.benson@noaa.gov" target="_blank">rusty.benson@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div dir="ltr" style="font-size:12.8px">I don't know how much experience people have with Jet, but please be aware there is a mpirun wrapper that supersedes the package version (impi, mvapich, etc). I point this out because the behavior is very different than the experience on Theia, where one uses the package version mpirun directly.<div><br></div><div>We at GFDL have found Jet's wrapper to be particularly bothersome when trying to run with OpenMP as we place/space MPI-ranks explicitly as part of the mpirun request and internally manage threading via namelist options.<br></div><div><br></div></div><div class="gmail_extra" style="font-size:12.8px"><span><div><div class="m_2800578371468643854m_-2742086870988853159gmail-m_-2636068731429181455gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div style="font-size:12.8px">Rusty</div><div style="font-size:12.8px">--</div><div style="font-size:12.8px">Rusty Benson, PhD</div><div style="font-size:12.8px">Modeling Systems Group</div><div style="font-size:12.8px">NOAA Geophysical Fluid Dynamics Lab</div><div style="font-size:12.8px">Princeton, NJ</div></div></div></div></div></div><br></span><div><div class="m_2800578371468643854h5"><div class="gmail_quote">On Fri, May 12, 2017 at 12:40 PM, Jun Wang - NOAA Affiliate <span dir="ltr"><<a href="mailto:jun.wang@noaa.gov" target="_blank">jun.wang@noaa.gov</a>></span> <wbr>wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div>Jim,<br><br></div>Sam is working on that, we hope to make it to this release. Thanks.<br><br></div>Jun<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, May 12, 2017 at 12:37 PM, James Rosinski - NOAA Affiliate <span dir="ltr"><<a href="mailto:james.rosinski@noaa.gov" target="_blank">james.rosinski@noaa<wbr>.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div><div>Jun and others;<br><br></div>I noticed there is a build capability for fv3gfs on jet, but not any run instructions or runjob_jet.sh for jet. Is jet run capability intended, or is that for later?<br><br></div>Regards,<br></div>Jim Rosinski</div></blockquote></div></div></blockquote></div></div></div></div></div></div></blockquote></div></div></blockquote></div></div></div>