<div dir="ltr">Eugene,<div><br></div><div>There is code inside of FMS that relies on Cray pointers because Fortran pointers don't have tne needed functionality for the way they are being used.</div><div><br></div><div>Cray recommended it because you are in a cross-compiling environment and you need to tell the compiler to target a specific instruction set for best performance. </div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="font-size:12.8px">Rusty</div><div style="font-size:12.8px">--</div><div style="font-size:12.8px">Rusty Benson, PhD</div><div style="font-size:12.8px">Modeling Systems Group</div><div style="font-size:12.8px">NOAA Geophysical Fluid Dynamics Lab</div><div style="font-size:12.8px">Princeton, NJ</div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Fri, May 12, 2017 at 1:50 PM, Eugene Mirvis <span dir="ltr"><<a href="mailto:eugene.mirvis@noaa.gov" target="_blank">eugene.mirvis@noaa.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Rusty,<br>
<br>
It may be standard.<br>
However, we implemented "-xcore-AVX2" by the vendor recommendation to make compatible build for both the sandybridge and haswell.<br>
As far as I know we never implemented that for NEMS on any other machines but on WCOSS-cray for this purpose. with ftn wrapper. Not mpiifort.<br>
<br>
-safe-cray-ptr yes, it tells the compiler that Cray pointers do not alias (that is, do not specify sharing memory with) other variables.<br>
Do we have any?<br>
<br>
I just think that it happened because of the conf structure hierarchy.<br>
Now, if it has been done on Theia on purpuse - please let me know.<br>
<br>
Thanks,<br>
-Eugene<br>
<br>
<br>
<br>
<br>
<br>
<br>
On 5/12/2017 1:30 PM, Rusty Benson - NOAA Federal wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Eugene,<br>
<br>
These are standard Intel compiler options and have no relationship to a<br>
particular vendor's system. "-xcore-AVX2" describes the instruction set<br>
support you are requesting from the compiler and will generate instructions<br>
suitable for the latest *well family of processors from Intel.<br>
<br>
"-safe-cray-ptr" deals with a Fortran extension known as "Cray pointers"<br>
and states that your use of them is unique and it is safe to optimize code<br>
containing these objects.<br>
<br>
Rusty<br>
--<br>
Rusty Benson, PhD<br>
Modeling Systems Group<br>
NOAA Geophysical Fluid Dynamics Lab<br>
Princeton, NJ<br>
<br>
On Fri, May 12, 2017 at 1:23 PM, Eugene Mirvis <<a href="mailto:eugene.mirvis@noaa.gov" target="_blank">eugene.mirvis@noaa.gov</a>><br>
wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Rusty,<br>
<br>
For instance the flags "-xCORE-AVX2" and "-safe-cray-ptr" we use only on<br>
Crays on the certain purposes.<br>
<br>
-Eugene<br>
<br>
On 5/12/2017 12:16 PM, Rusty Benson - NOAA Federal wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Eugene,<br>
<br>
Responding only to point 4, what options do you see that are Cray<br>
compilation flags?<br>
<br>
Rusty<br>
--<br>
Rusty Benson, PhD<br>
Modeling Systems Group<br>
NOAA Geophysical Fluid Dynamics Lab<br>
Princeton, NJ<br>
<br>
On Fri, May 12, 2017 at 11:58 AM, Eugene Mirvis <<a href="mailto:eugene.mirvis@noaa.gov" target="_blank">eugene.mirvis@noaa.gov</a>><br>
wrote:<br>
<br>
Gerard,<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Just several points to clarify.<br>
<br>
1. NEMS practice to call modulefiles what is actually the scripts<br>
(calling module commands),<br>
that require mostly bash env in order to source and keep environment was<br>
always non standard use.<br>
<br>
Therefore, if the developers will take Sam's advice and make real<br>
modulefiles (starting with #%Module) from the script, "export" and other<br>
scripting works<br>
wouldn't make sense, while Module util commands and Tcl/Tk will work.<br>
<br>
2. There is another dilemma - to keep needed environment and change<br>
within<br>
a workflow env. change.<br>
btw,<br>
unsetenv, append-path, prepend-path and remove-path module commands are<br>
very useful for controlling that, but you have to keep<br>
$LOADEDMODULES, and $MODULE PATH in consistent order.<br>
<br>
3. Speaking of which,<br>
module purge<br>
and<br>
module switch<br>
are very useful to unload application driven modules.<br>
a/ You just have to do that not any moment, but before apps module is<br>
loaded, then, unload <appsModulefile> - not purge, but unload.<br>
b/ On Crays, the are some internal dependencies inside of PrgEnv. So you<br>
have to keep all "module use <knowns>" to recover, otherwise you might<br>
find "module not found"<br>
<br>
4.<br>
Compiling on Theia, I'm just wondering why Cray's compilation flags are<br>
utilized... allover:<br>
See for instance<br>
...<br>
*mpiifort -I/apps/netcdf/4.3.0-intel/inc<wbr>lude -fno-alias -auto<br>
-safe-cray-ptr -ftz -assume byterecl -nowarn -sox -align array64byte -i4<br>
-real-size 64 -no-prec-div -no-prec-sqrt -xCORE-AVX2**<br>
-qno-opt-dynamic-align* -O2 -debug minimal -fp-model source<br>
-qoverride-limits -qopt-prefetch=3 -qopenmp -I/apps/esmf/7.0.0/intel/<br>
intelmpi/mod/modO/Linux.intel.<wbr>64.intelmpi.default<br>
-I/apps/esmf/7.0.0/intel/intel<wbr>mpi/include -I/apps/netcdf/4.3.0-intel/inc<br>
lude<br>
-IENS_Cpl -I. -I/scratch4/NCEPDEV/global/nos<br>
crub/Eugene.Mirvis/fv3gfs.v0be<wbr>ta/FV3/nems_dir<br>
-c module_MEDIATOR_methods.f90<br>
...<br>
<br>
Thanks,<br>
-Eugene<br>
<br><span class="HOEnZb"><font color="#888888">
</font></span></blockquote></blockquote></blockquote></blockquote><span class="HOEnZb"><font color="#888888">
<br>
-- <br>
EUGENE MIRVIS, Tech Lead,<br>
Senior Computational Scientist, IMSG @<br>
Global Climate & Weather Modeling Branch of<br>
NOAA/NCEP Environmental Modeling Center<br>
NCWCP, rm 2183<br>
5830 University Research Ct.<br>
College Park, MD 20740<br>
Ph. <a href="tel:301.683.3809" value="+13016833809" target="_blank">301.683.3809</a><br>
<br>
</font></span></blockquote></div><br></div>