[Ncep.list.nems.announce] nems r98008: Bug fixes and netcdf variable data comparison support
Samuel.Trahan at noaa.gov
Samuel.Trahan at noaa.gov
Fri Sep 29 20:13:04 UTC 2017
...
Message-ID: <59cea950.ewR46jB5j3DOL0hu%Samuel.Trahan at noaa.gov>
User-Agent: Heirloom mailx 12.5 7/5/10
MIME-Version: 1.0
Content-Type: multipart/mixed;
boundary="=_59cea950.3dVHTLnJpnzrFUXkoTxVzlT6zUUXwxVcFXAqzr0aj74zwKjV"
This is a multi-part message in MIME format.
--=_59cea950.3dVHTLnJpnzrFUXkoTxVzlT6zUUXwxVcFXAqzr0aj74zwKjV
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Friendly NEMS developers,
This is an automated email about a NEMS commit.
Project: nems
URL: https://svnemc.ncep.noaa.gov/projects/nems/trunk
Revision: 98008
Author: samuel.trahan at noaa.gov
Date: 2017-09-29T20:01:17.830678Z
Message:
Bug fixes and netcdf variable data comparison support
1. Bug fix from Jun Wang to .bitcmp. comparison. If an output file is compared to a baseline by specifying the baseline directory (instead of filename within the directory) then the baseline creation fails. It does not realize that the target is a directory because the directory does not exist before the baseline is created:
"RESTART/my_output.rst" .bitcmp. "@[BASELINE_DIR]/RESTART/"
The fix is to assume the target is a directory if:
- it ends with a /
- the last path component is .
- the last path component is ..
Hence, after the commit, "@[BASELINE_DIR]/RESTART/" will work correctly if it ends in a /
2. Add the capability to NEMSCompsetRun to compare NetCDF files variable-by-variable instead of bit-for-bit. This is needed for the CICE5 files, which contain a global metadata value containing the date at which the file was created. The actual contents of the variables should match.
3. produtil: When generating a baseline, the NEMSCompsetRun --resume feature forgot it was generating a baseline and reset to execution (verification) mode. This is corrected.
4. NEMS + apps: All /gpfs/hps/emc paths I found are now /gpfs/hps3/emc. (produtil had none)
5. NEMS: Removed an erroneous fingerprint file check from baseline generation during a time at which the target directory does not yet exist.
Differences:
Index: checkout/tests/rtgen
===================================================================
--- checkout/tests/rtgen (revision 97132)
+++ checkout/tests/rtgen (revision 98008)
@@ -860,6 +860,7 @@
TEMP_AREA={temp_area} ## temporary area, auto-detected or specified at command line
RUNDIR={run_dir} ## top directory of generated workflow
SETS='{setarith}' ## set arithmetic specification of which sets to run
+RUN_MODE='{run_mode}' ## BASELINE = generate baseline, otherwise verify
"""
contents=contents.format(
platform_name=self.scope.resolve('plat%PLATFORM_NAME'),
@@ -868,7 +869,8 @@
unique_id=self.unique_id,
temp_area=os.path.dirname(os.path.realpath(self.outloc)),
run_dir=self.outloc,
- setarith=self.setarith
+ setarith=self.setarith,
+ run_mode='BASELINE' if self.run_mode==BASELINE else 'EXECUTION'
)
self.make_rtscript(self.outloc,"info.sh.inc",contents)
def make_rtreport(self):
@@ -1076,7 +1078,7 @@
info_sh_inc=os.path.join(resume,'info.sh.inc')
with open(info_sh_inc,'rt') as fd:
for line in fd:
- m=re.match('([A-Za-z][A-Za-z0-9_]*)=(.*?) ##',line)
+ m=re.match(r'''([A-Za-z][A-Za-z0-9_]*)=["']?(.*?)['"]? ##''',line)
if not m: continue
(var,val)=m.groups()
if var.lower()=='baseline_dir':
@@ -1096,8 +1098,13 @@
jlogger.info('Platform name: %s'%(repr(platform_name),))
assert('/' not in platform_name)
elif var.lower()=='sets':
- resume_sets=val[1:-1] # 'set,set,set' => set,set,set
+ resume_sets=val # 'set,set,set' => set,set,set
jlogger.info('Set specification: %s'%(repr(resume_sets),))
+ elif var.lower()=='run_mode':
+ if val.lower()=='baseline':
+ baseline=True
+ elif val.lower()=='execution':
+ baseline=False
if baseline_dir is None or unique_id is None or temp is None \
or run_dir is None or platform_name is None or resume_sets is None:
@@ -1261,7 +1268,8 @@
testgen.parse()
jlogger.info('Verifying repo fingerprint against data fingerprint.')
- verify_fingerprint(baseline,testgen,logger)
+ if not baseline:
+ verify_fingerprint(baseline,testgen,logger)
try:
jlogger.info('Generating workflow with id %s.'%(repr(unique_id),))
Index: checkout/tests
===================================================================
--- checkout/tests (revision 97132)
+++ checkout/tests (revision 98008)
Property changes on: checkout/tests
___________________________________________________________________
Modified: svn:externals
## -1 +1 ##
-produtil -r96302 https://svnemc.ncep.noaa.gov/projects/nceplibs/produtil/branches/regtests-run
+produtil -r98007 https://svnemc.ncep.noaa.gov/projects/nceplibs/produtil/branches/regtests-run
Modified: svn:mergeinfo
## -0,0 +0,1 ##
Merged /nems/branches/nccmp-deliver/tests:r97663-98007
Index: checkout
===================================================================
--- checkout (revision 97132)
+++ checkout (revision 98008)
Property changes on: checkout
___________________________________________________________________
Modified: svn:mergeinfo
## -0,0 +0,1 ##
Merged /nems/branches/nccmp-deliver:r97663-98007
--=_59cea950.3dVHTLnJpnzrFUXkoTxVzlT6zUUXwxVcFXAqzr0aj74zwKjV
Content-Type: text/plain;
charset=us-ascii
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment;
filename="98008.diff"
Index: checkout/tests/rtgen
===================================================================
--- checkout/tests/rtgen (revision 97132)
+++ checkout/tests/rtgen (revision 98008)
@@ -860,6 +860,7 @@
TEMP_AREA={temp_area} ## temporary area, auto-detected or specified at command line
RUNDIR={run_dir} ## top directory of generated workflow
SETS='{setarith}' ## set arithmetic specification of which sets to run
+RUN_MODE='{run_mode}' ## BASELINE = generate baseline, otherwise verify
"""
contents=contents.format(
platform_name=self.scope.resolve('plat%PLATFORM_NAME'),
@@ -868,7 +869,8 @@
unique_id=self.unique_id,
temp_area=os.path.dirname(os.path.realpath(self.outloc)),
run_dir=self.outloc,
- setarith=self.setarith
+ setarith=self.setarith,
+ run_mode='BASELINE' if self.run_mode==BASELINE else 'EXECUTION'
)
self.make_rtscript(self.outloc,"info.sh.inc",contents)
def make_rtreport(self):
@@ -1076,7 +1078,7 @@
info_sh_inc=os.path.join(resume,'info.sh.inc')
with open(info_sh_inc,'rt') as fd:
for line in fd:
- m=re.match('([A-Za-z][A-Za-z0-9_]*)=(.*?) ##',line)
+ m=re.match(r'''([A-Za-z][A-Za-z0-9_]*)=["']?(.*?)['"]? ##''',line)
if not m: continue
(var,val)=m.groups()
if var.lower()=='baseline_dir':
@@ -1096,8 +1098,13 @@
jlogger.info('Platform name: %s'%(repr(platform_name),))
assert('/' not in platform_name)
elif var.lower()=='sets':
- resume_sets=val[1:-1] # 'set,set,set' => set,set,set
+ resume_sets=val # 'set,set,set' => set,set,set
jlogger.info('Set specification: %s'%(repr(resume_sets),))
+ elif var.lower()=='run_mode':
+ if val.lower()=='baseline':
+ baseline=True
+ elif val.lower()=='execution':
+ baseline=False
if baseline_dir is None or unique_id is None or temp is None \
or run_dir is None or platform_name is None or resume_sets is None:
@@ -1261,7 +1268,8 @@
testgen.parse()
jlogger.info('Verifying repo fingerprint against data fingerprint.')
- verify_fingerprint(baseline,testgen,logger)
+ if not baseline:
+ verify_fingerprint(baseline,testgen,logger)
try:
jlogger.info('Generating workflow with id %s.'%(repr(unique_id),))
Index: checkout/tests
===================================================================
--- checkout/tests (revision 97132)
+++ checkout/tests (revision 98008)
Property changes on: checkout/tests
___________________________________________________________________
Modified: svn:externals
## -1 +1 ##
-produtil -r96302 https://svnemc.ncep.noaa.gov/projects/nceplibs/produtil/branches/regtests-run
+produtil -r98007 https://svnemc.ncep.noaa.gov/projects/nceplibs/produtil/branches/regtests-run
Modified: svn:mergeinfo
## -0,0 +0,1 ##
Merged /nems/branches/nccmp-deliver/tests:r97663-98007
Index: checkout
===================================================================
--- checkout (revision 97132)
+++ checkout (revision 98008)
Property changes on: checkout
___________________________________________________________________
Modified: svn:mergeinfo
## -0,0 +0,1 ##
Merged /nems/branches/nccmp-deliver:r97663-98007
--=_59cea950.3dVHTLnJpnzrFUXkoTxVzlT6zUUXwxVcFXAqzr0aj74zwKjV--
More information about the Ncep.list.nems.announce
mailing list