This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
>To: address@hidden >From: Robert Fischer <address@hidden> >Subject: comments on netCDF v3 >Keywords: 199701221547.IAA02331 Hi Bob, > i use netCDF version 2.4, and am interested in the continued development > of netCDF. i have some comments about potential future versions: Thanks for your thoughtful suggestions. We will carefully consider your comments along with others we have received. > 1) you've eliminated the use of generic void pointers in the ncdf v3. > although this is good from a "user's" point of view, it renders impossible > some things i'm doing now. i have a generic multi-dimensional array class > in C++, which i have interfaced with netCDF. it's very convenient, since > variables can be simply and easily "copied" (general array section copy) > from file to memory, and back again. along with some other routines which > allow me to "initialize" memory arrays the same size as netCDF arrays and > vice versa, as well as to define netCDF arrays without having to tediously > define every dimension, etc, i have quite a convenient interface. > > needless to say, eliminating void pointers would put quite a damper on my > ability to implement my array class. please retain a void* interface. Although the generic void* pointers are eliminated from the public documented interface, they are still available in the library. The following declarations excerpted from src/libsrc/nc.h are the generic interfaces for nc_{put,get}_att, and nc_{put,get}_var{1,a,s,m,}: /* * These functions are used to support * interface version 2 backward compatiblity. * N.B. these are tested in ../nc_test even though they are * not public. So, be careful to change the declarations in * ../nc_test/tests.h if you change these. */ extern int nc_put_att(int ncid, int varid, const char *name, nc_type datatype, size_t len, const void *value); extern int nc_get_att(int ncid, int varid, const char *name, void *value); extern int nc_put_var1(int ncid, int varid, const size_t *index, const void *value); extern int nc_get_var1(int ncid, int varid, const size_t *index, void *value); extern int nc_put_vara(int ncid, int varid, const size_t *start, const size_t *count, const void *value); extern int nc_get_vara(int ncid, int varid, const size_t *start, const size_t *count, void *value); extern int nc_put_vars(int ncid, int varid, const size_t *start, const size_t *count, const ptrdiff_t *stride, const void * value); extern int nc_get_vars(int ncid, int varid, const size_t *start, const size_t *count, const ptrdiff_t *stride, void * value); extern int nc_put_varm(int ncid, int varid, const size_t *start, const size_t *count, const ptrdiff_t *stride, const ptrdiff_t * map, const void *value); extern int nc_get_varm(int ncid, int varid, const size_t *start, const size_t *count, const ptrdiff_t *stride, const ptrdiff_t * map, void *value); While this may solve the immediate problem of permitting you to implement your array classes over netCDF-3, the problem of what to do in netCDF-4 and subsequent versions remains. In netCDF-4, we intend to support "packed" external data types such as arrays of 11-bit values. Such an external type has no natural internal type associated with it. It is always converted to or from a specified unpacked internal type when it is accessed. A user who accesses the values as doubles need not know or care that the values are actually represented externally as packed 11-bit values, but can determine this is the type of the variable using an nc_inq_var function. The use of the void* generic interfaces won't make much sense for such packed data types. > 2) i would find an ability to remove variables and dimensions from the > file to be very useful. i realize that removing dimensions is not always > possible, if the dimensions are being used; an error in that case would be > fine. removing variables should always be possible (i realize it would > require rewriting the entire file; that's OK). Permitting removal of variables and dimensions causes a problem in the C and FORTRAN interfaces: after a variable with variable ID 1 is removed, are the variable IDs of subsequent variables decremented? If so, some programs that assume variable IDs never change break; if not, programs that assume all variables have consecutive variable IDs (e.g. ncdump) break. In C++ and Java, this is not a problem, since variable IDs are not needed as object identifier surrogates, but I don't see a way around this in C and FORTRAN ... > 3) an ability to read & write arrays of arbitrary data types (you just > give it the size of the type). i realize that the XDR stuff could not be > used for such an operation, since netCDF would not know how to interpret > the bits involved. however, i think decisions about the non-portability > inherent in such a use should be left to the application programmer. as a > fix for this problem, you could let the programmer supply a subroutine, > presumeably based on XDR, which convert the data type correctly between You can currently read and write such arrays by just treating them as a bland array of bytes, in a variable of type NC_BYTE. Any application-specific conversion for such types may be implemented in a layer above netCDF. I'm not sure what advantage would be gained by putting this in the netCDF library, since the library would have to indirectly invoke a user-supplied function or method on every value on each access, and the user would have to define the type by registering a function or method pointer as well as a size. Maybe it would be more convenient? As far as I know, yours is the first request we've had for such a capability. It might be a way to implement the packed data types in an extensible way ... > 4) allow hierarchical variables. i have some data structures which are > stored as a few netCDF arrays. for example, a compressed row-format (cmr) > matrix consists of three arrays, called pntre, indx, and val. if i want > to write a cmr matrix called "damp" to a netCDF file, I have to do it > using three seemingly separate variables, as follows: > > netcdf t6-mingrad { > dimensions: > damp-nrow = 362 ; > damp-ncol = 362 ; > damp-nel = 72522 ; > variables: > long damp-pntre(damp-nrow) ; > long damp-indx(damp-nel) ; > double damp-val(damp-nel) ; > > // global attributes: > :damp-type = "mingrad" ; > :tessa-file = "t6.cdf" ; > :remark = "2-D Damping matrix for spherical spline > tesselation, degree 6" ; > } > > i would like to have netCDF take care of this hierarchical stuff for me, > and be able to write my damping matrix as one sub-file, like: > > netcdf t6-mingrad { > dimensions: > nrow = 362 ; > ncol = 362 ; > nel = 72522 ; > variables: > netcdf damp { > variables: > long pntre(nrow) ; > long indx(nel) ; > :remark = "attribute on damp.indx" > double val(nel) ; > > // global attributes for damp > :type = "mingrad" ; > :remark = "2-D Damping matrix for spherical spline > }; > int funny(nrow); > > // global attributes: > :tessa-file = "t6.cdf" ; > } > > standard lexical scoping naming rules could be used to determine what a > name means at a certain position in the CDL file. This is a very interesting suggestion, and will require more study ... Thanks again for your comments. --Russ _____________________________________________________________________ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu