Add NetCDF data file write/read #1007

drroe · 2022-12-05T16:47:15Z

Version 6.16.3.

Adds a new data file type based on NetCDF. This is something I've wanted to do for a while to take advantage of all the features of NetCDF. The main benefits of using the NetCDF data format are:

Data is stored more or less exactly how it is allocated in CPPTRAJ itself. This means no precision is lost for anything as it is with ASCII data files. Matrices and eigenmodes in particular should benefit from this. It also means that data set metadata (things like data set name and aspect) are not lost.
All data sets can be written to a single file, regardless of set type - no other file format in CPPTRAJ supports this. So e.g. you can store a distance, a matrix, a grid, and a cluster pairwise matrix all in the same file.

Note that there was already a NetCDF format type specific to cluster pairwise matrices (.nccmatrix). This is still supported, but probably won't be developed further.

Adds tests for most data set types and updates the manual.

Below shows an example of a NetCDF file header after several matrices/eigenmodes are written to a single NetCDF file with the following CPPTRAJ input:

parm ../tz2.parm7
trajin ../tz2.nc
matrix name CA covar @CA out matrix.nc
matrix name N.CA mwcovar @N @CA out matrix.nc
matrix name heavyAtom !@H= mwcovar out matrix.nc
2drms RMS2D @CA out matrix.nc
diagmatrix CA out matrix.nc name MyEvecs
diagmatrix heavyAtom out matrix.nc name heavyEvecs
run
writedata ca.matrix.dat.save CA nosquare2d prec 12.4
writedata n.ca.matrix.dat.save N.CA nosquare2d prec 12.4
writedata heavyAtom.matrix.dat.save heavyAtom nosquare2d prec 12.4
writedata ca.rms2d.dat.save RMS2D nosquare2d prec 12.4
writedata MyEvecs.dat.save MyEvecs
writedata heavyEvecs.dat.save heavyEvecs

Disk usage of NetCDF vs disk usage of individual ASCII matrix/modes files:

$ du matrix.nc 
1508	matrix.nc
$ du ca.matrix.dat.save n.ca.matrix.dat.save heavyAtom.matrix.dat.save ca.rms2d.dat.save MyEvecs.dat.save heavyEvecs.dat.save 
44	ca.matrix.dat.save
44	n.ca.matrix.dat.save
3852	heavyAtom.matrix.dat.save
320	ca.rms2d.dat.save
16	MyEvecs.dat.save
1356	heavyEvecs.dat.save

netcdf matrix {
dimensions:
	size.0 = 666 ;
	vectsize.1 = 36 ;
	size.2 = 1404 ;
	vectsize.3 = 75 ;
	nmass.4 = 13 ;
	size.5 = 61776 ;
	vectsize.6 = 351 ;
	nmass.7 = 117 ;
	size.8 = 5050 ;
	nmodes.9 = 36 ;
	nevecelts.10 = 1296 ;
	ncoords.11 = 36 ;
	nmodes.12 = 351 ;
	nevecelts.13 = 123201 ;
	ncoords.14 = 351 ;
	nmass.15 = 117 ;
variables:
	double CA.matrix(size.0) ;
		CA.matrix:name = "CA" ;
		CA.matrix:legend = "CA" ;
		CA.matrix:scalartype = "covariance" ;
		CA.matrix:scalarmode = "matrix" ;
		CA.matrix:nindexvar = 2 ;
		CA.matrix:min0 = 1. ;
		CA.matrix:step0 = 1. ;
		CA.matrix:label0 = "Atom" ;
		CA.matrix:min1 = 1. ;
		CA.matrix:step1 = 1. ;
		CA.matrix:description = "double matrix" ;
		CA.matrix:ncols = 36 ;
		CA.matrix:nrows = 36 ;
		CA.matrix:matrixkind = "half" ;
		CA.matrix:nsnapshots = 101 ;
		CA.matrix:vectid = 1 ;
	double CA.vect(vectsize.1) ;
	double N.CA.matrix(size.2) ;
		N.CA.matrix:name = "N.CA" ;
		N.CA.matrix:legend = "N.CA" ;
		N.CA.matrix:scalartype = "mass-weighted covariance" ;
		N.CA.matrix:scalarmode = "matrix" ;
		N.CA.matrix:nindexvar = 2 ;
		N.CA.matrix:min0 = 1. ;
		N.CA.matrix:step0 = 1. ;
		N.CA.matrix:label0 = "Atom" ;
		N.CA.matrix:min1 = 1. ;
		N.CA.matrix:step1 = 1. ;
		N.CA.matrix:description = "double matrix" ;
		N.CA.matrix:nco
8000
ls = 39 ;
		N.CA.matrix:nrows = 36 ;
		N.CA.matrix:matrixkind = "full" ;
		N.CA.matrix:nsnapshots = 101 ;
		N.CA.matrix:vectid = 3 ;
		N.CA.matrix:massid = 4 ;
	double N.CA.vect(vectsize.3) ;
	double N.CA.mass(nmass.4) ;
	double heavyAtom.matrix(size.5) ;
		heavyAtom.matrix:name = "heavyAtom" ;
		heavyAtom.matrix:legend = "heavyAtom" ;
		heavyAtom.matrix:scalartype = "mass-weighted covariance" ;
		heavyAtom.matrix:scalarmode = "matrix" ;
		heavyAtom.matrix:nindexvar = 2 ;
		heavyAtom.matrix:min0 = 1. ;
		heavyAtom.matrix:step0 = 1. ;
		heavyAtom.matrix:label0 = "Atom" ;
		heavyAtom.matrix:min1 = 1. ;
		heavyAtom.matrix:step1 = 1. ;
		heavyAtom.matrix:description = "double matrix" ;
		heavyAtom.matrix:ncols = 351 ;
		heavyAtom.matrix:nrows = 351 ;
		heavyAtom.matrix:matrixkind = "half" ;
		heavyAtom.matrix:nsnapshots = 101 ;
		heavyAtom.matrix:vectid = 6 ;
		heavyAtom.matrix:massid = 7 ;
	double heavyAtom.vect(vectsize.6) ;
	double heavyAtom.mass(nmass.7) ;
	float RMS2D.matrix(size.8) ;
		RMS2D.matrix:name = "RMS2D" ;
		RMS2D.matrix:legend = "RMS2D" ;
		RMS2D.matrix:nindexvar = 2 ;
		RMS2D.matrix:min0 = 1. ;
		RMS2D.matrix:step0 = 1. ;
		RMS2D.matrix:min1 = 1. ;
		RMS2D.matrix:step1 = 1. ;
		RMS2D.matrix:description = "float matrix" ;
		RMS2D.matrix:ncols = 101 ;
		RMS2D.matrix:nrows = 101 ;
		RMS2D.matrix:matrixkind = "tri" ;
	double MyEvecs.eigenvalues(nmodes.9) ;
		MyEvecs.eigenvalues:name = "MyEvecs" ;
		MyEvecs.eigenvalues:legend = "MyEvecs" ;
		MyEvecs.eigenvalues:scalartype = "covariance" ;
		MyEvecs.eigenvalues:nindexvar = 0 ;
		MyEvecs.eigenvalues:description = "eigenmodes" ;
		MyEvecs.eigenvalues:eigenvectorsid = 10 ;
		MyEvecs.eigenvalues:avgcoordsid = 11 ;
	double MyEvecs.eigenvectors(nevecelts.10) ;
	double MyEvecs.avgcoords(ncoords.11) ;
	double heavyEvecs.eigenvalues(nmodes.12) ;
		heavyEvecs.eigenvalues:name = "heavyEvecs" ;
		heavyEvecs.eigenvalues:legend = "heavyEvecs" ;
		heavyEvecs.eigenvalues:scalartype = "mass-weighted covariance" ;
		heavyEvecs.eigenvalues:nindexvar = 0 ;
		heavyEvecs.eigenvalues:description = "eigenmodes" ;
		heavyEvecs.eigenvalues:eigenvectorsid = 13 ;
		heavyEvecs.eigenvalues:avgcoordsid = 14 ;
		heavyEvecs.eigenvalues:massid = 15 ;
	double heavyEvecs.eigenvectors(nevecelts.13) ;
	double heavyEvecs.avgcoords(ncoords.14) ;
	double heavyEvecs.mass(nmass.15) ;

// global attributes:
		:program = "cpptraj" ;
		:programVersion = "V6.16.3" ;
		:Conventions = "CPPTRAJ_DATA" ;
		:ConventionVersion = "1.0" ;
}

matrix is sparse), use indexid to point to index variable. Otherwise use min and step.

dimensions.

matrix

unsigned int type

read/write.

src/DataIO_NetCDF.cpp

lgtm-com · 2022-12-05T17:28:59Z

This pull request introduces 4 alerts when merging b4eac12 into b87be3c - view on LGTM.com

new alerts:

2 for 'new[]' array freed with 'delete'
1 for Multiplication result converted to larger type
1 for Too few arguments to formatting function

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

src/Action_Bounds.cpp

lgtm-com · 2022-12-05T19:50:17Z

This pull request introduces 2 alerts when merging 204634f into b87be3c - view on LGTM.com

new alerts:

1 for Wrong type of arguments to formatting function
1 for Too few arguments to formatting function

Heads-up: LGTM.com's PR analysis will be disabled on the 5th of December, and LGTM.com will be shut down ⏻ completely on the 16th of December 2022. It looks like GitHub code scanning with CodeQL is already set up for this repo, so no further action is needed 🚀. For more information, please check out our post on the GitHub blog.

drroe added 30 commits September 1, 2022 15:40

Start generic netcdf data io class

993be9f

Add id and initial creation of file

26d461d

Start getting dimensions

a9be17e

Update files and dependencies

db31546

Add to DataFile

ad02dac

Start test

8a8e8ff

Indicate netcdf will be valid for 1d, 2d, 3d

4a4c9db

Start adding sets

98f8a9d

Save dimension indices for each set

8f5fca4

Define dimensions

5c807d9

New grouping strategy

5fcd289

Start defining variables

4575126

Choose type

61e3403

Add legends

dc8d1e5

Do variable write

2301c06

Do index write

a772073

Put 1D set write into a separate routine

a4d3507

Ensure all metadata is added to each variable

81f61ad

Add nctest

e12d530

Merge branch 'master' into netcdf.data

b04cf41

Merge branch 'master' into netcdf.data

f71b396

Improve code docs

beb9419

Add ifdef

f3c82ed

Start read. Ensure netcdf file is closed after read/write.

b5a134e

Start inquiring about variables stored in the netcdf file

395df94

Add monotonic attribute to distinguish between double and xymesh

0835452

Start testing the mesh set

ada6e62

Add DvalPtr function to DataSet_1D so writing XY mesh works correctly

e9cf5d4

Added an xy mesh set

8f5d366

Fix typo

8d740a8

drroe added 15 commits December 5, 2022 08:59

If frames were sieved out of the pairwise cluster matrix (i.e. the

a327829

matrix is sparse), use indexid to point to index variable. Otherwise use min and step.

Change nindexdim to nindexvar. Really it refers to index variables, not

b9d7600

dimensions.

Update with nindexdim -> nindexvar

bc97d12

Update warning

fa70066

Hide debug info

2e4f19c

Remove unused code

8e01e53

Remove more unused code

fa96346

Test user specified name for netcdf dataset read

b036ba8

Actually use user specfied name

a79a764

Test storing cluster num vs time in netcdf data along with pairwise

1730035

matrix

Change write for unsigned int set; netcdf classic does not have an

83bc8a9

unsigned int type

Add routine for reading unsigned ints

017558d

Add test for read/write unsigned ints

13d1074

Add netcdf data file format to table

ebbc25a

Version 6.16.3. Revision bump for addition of netcdf datafile

b4eac12

read/write.

drroe added the enhancement label Dec 5, 2022

drroe self-assigned this Dec 5, 2022

github-advanced-security bot found potential problems Dec 5, 2022

View reviewed changes

src/DataIO_NetCDF.cpp Fixed Show fixed Hide fixed

src/DataIO_NetCDF.cpp Fixed Show fixed Hide fixed

drroe added 4 commits December 5, 2022 12:57

Use delete[] instead of incorrect delete

dc6bc80

Add missing arg for mprinterr

6a13af7

Convert to size_t before multiply to prevent potential overflows.

933a798

Ensure that if a grid is to be allocated, all processes allocate it

204634f

github-advanced-security bot found potential problems Dec 5, 2022

View reviewed changes

src/Action_Bounds.cpp Fixed Show fixed Hide fixed

src/Action_Bounds.cpp Fixed Show fixed Hide fixed

drroe added 3 commits December 5, 2022 15:01

Fix error message formatting

8942e2e

Try to fix TOCTOU issue

c3e4d38

Try to avoid multiplication overflows

1694217

drroe merged commit cac269c into Amber-MD:master Dec 6, 2022

drroe deleted the netcdf.data branch December 6, 2022 13:12

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add NetCDF data file write/read #1007

Add NetCDF data file write/read #1007

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Add NetCDF data file write/read #1007

Add NetCDF data file write/read #1007

Uh oh!

Conversation

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!