[netCDF #TLB-677315]: FW: Net CDF-4 Issues

Coy,

> I have attached three very simple NetCDF-3 files that I hope will
> better illustrate the issue.
> 
> OneMode.txt - This file shows the output after data has been
> acquired using a single mode.
> 
> TwoModes.txt - This file shows what I would like the file to look
> like after data has been acquired from the second mode. Note that
> there is a different number of heights.
> 
> ThreeModes.txt - This file shows what I would like the file to look
> like after data has been acquired from the third mode. Note that
> there is a different number of heights along with a new variable.
> 
> I would prefer to use NetCDF-3 if possible for compatibility
> reasons. I also need this code to be robust as it will be running 24
> hours a day.

If you must use netCDF-3, only one unlimited dimension is permitted
and no other dimensions can be increased.  That constraint precludes
any solution other than copying to a new file.  Adding new variables,
dimensions, and attributes is not a problem, because space can be
reserved in the header to permit that without copying data.  But
changing a dimension size other than the unlimited dimension might
require changing the location of data values in the file, hence it is
not a supported operation for netCDF-3.

The netCDF-4 format allows multiple unlimited dimensions, but arrays
that use multiple unlimited dimensions are still rectangular, not
ragged.

> Since I do not know exactly how many modes might appear in a file, I
> would prefer to add modes when needed instead of defining a bunch of
> extra modes and filling them in later.
> 
> So, let's assume my instrument acquires data from the first mode for
> a period of time. Then it acquires data from the second mode for a
> period of time. Then it acquires data from the third mode for a
> while. In this case, the file would look like OneMode.txt for the
> first period. When I come to the second mode, I would like to be
> able to expand file to look like TwoModes.txt. Thus I would like to
> expand the 'Mode' dimension from 1 to 2 and fill in the value of
> variable 'Gate' with the value of 75. I would also need to expand
> the 'Heights' dimension from 60 to 75 to make space for data from
> the extra 15 heights. When I come to the third mode, I would like to
> be able to expand the file to look like ThreeMode.txt.  Thus I would
> like to expand the 'Mode' dimension from 2 to 3, fill in the
> variable 'Gate' with a value of 30. And if possible add the new
> variable 'RassStep' which would hold blanks for the first two modes.

You can do all this with netCDF-4 without copying data, but not using
the simpler netCDF-3 data model.  It permits adding variables, but not
increasing the size of more than one dimension.

> I believe I can do the above by copying the entire data file into a
> new file. While this may work, I am concerned my file may be
> extremely large at times. Is there a more graceful way to handle
> expanding a file as I have described above?

Sorry, not within the constraints of netCDF-3.

> If not, could you point me to an example of creating a new file and
> copying the old data into the new file?

We plan to have an nccopy utility included in version 4.1 of netCDF-4
that will work with netCDF-3 files.  In the meantime, you can look at
the benchmark program nc_test4/bm_file.c, from the netCDF-4 source
distribution, built if you configure with --enable-benchmarks. To use
it to copy a large classic format file named "afile.nc" to another
classic format file "bfile.nc", use

  bm_file -o afile.nc -f 1 bfile.nc
  
If the target file has to be in the 64-bit offset variant instead of
the classic format, you can use

  bm_file -o afile.nc -f 2 bfile.nc

Alternatively, you can install the NCO utilities

  http://www.unidata.ucar.edu/netcdf/docs/software.html#NCO

and look at the source of the "ncks" utility, which can be used to
copy netCDF files as follows:

  ncks -a --fl_fmt=classic afile.nc bfile.nc
or
  ncks -a --fl_fmt=64bit afile.nc bfile.nc

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: TLB-677315
Department: Support netCDF
Priority: Normal
Status: Closed