Re: 19990428: Questions about the UNLIMITED dimension (records)

>To: address@hidden
>From: "Roy Dennington" <address@hidden>
>Subject: Questions about the UNLIMITED dimension (records)
>Organization: .
>Keywords: 199904282011.OAA25479

Hi Roy,

> I reviewed your FAQ and searched the archives for information on 
> moving/copying records along the "growable" UNLIMITED dimension, but
> did not find any discussion on this topic.
> 
> Since all the variables that include the UNLIMITED dimension are stored
> in a block with a regular offset, I was hoping to move the blocks
> around in a sorting operation. Is it possible to act on the RECORD
> collectively without having to move each variable?

No, unfortunately the netCDF data model and interfaces don't support
this kind of operation, although with the current implementation and
format it could be done relatively efficiently.  I'm afraid you'll
have to sort (record_number, key) tuples by key, then copy the records
to a new netCDF data set in the new order.  It would not be too hard
to write a nc_copy_record(ncid1, ncid2, numrec) function to copy a
record from one open netCDF dataset to another in terms of the
supported interface.

The netCDF 2 interface had some functions that dealt with records as a
whole: ncrecput(), ncrecget(), and ncrecinq().  These functions are
still supported in the netCDF 2 compatibility code for netCDF-3, and
in fact you can see an implementation of them in the netCDF-2 test
code netcdf/src/nctest/rec.c, where the recput(), recget(), and
recinq() are essentially the same as the documented netCDF-2 functions
(see <http://www.unidata.ucar.edu/packages/netcdf/guide_9.html#SEC63>,
for the ncrecput() documentation, for example).  These might be useful
in implementing an nc_copy_record() function.

> I would also like to compress the UNLIMITED dimension. For example,
> I have the UNLIMITED dimension set to 10 records. I want to throw
> away the last five. Can I delete the last 5 by updating the record count?

No, unfortunately the netCDF data model and interfaces only support
adding data or overwriting data to a netCDF dataset, not deleting
data.  To delete data, you have to open a new netCDF dataset and copy
all but the data you want to delete from the original dataset to the
new dataset.

You could go below the netCDF interface and use knowledge of the
netCDF format to efficiently truncate a netCDF dataset and change the
number of records, but this violates the abstraction and would break
if the format changes (which it so far hasn't since 1989 :-).  We
thought about supporting deletion of variables and records, but this
wouldn't be very useful unless you could delete records in the middle
of a dataset instead of just at the end, and the resulting garbage
collection issues were not worth solving for the perceived benefits.
NetCDF is not intended to be a database management system, but rather
a data model for array-oriented scientific data, and the scientists we
dealt with never seemed to want to delete data from existing datasets
...

--Russ