Re: 970320: Var. dim. sizes in NetCDF

>From: "U.Poppe" <address@hidden>
>Subject: Var. dim. sizes in NetCDF
>Organization:  ?
>Keywords: 199703201322.GAA06015 netCDF variable dimension sizes

Hi Ulrich,

> I am using your NetCDF library 2.4 and it works fine! Compilation an
> installation on HP-UX without any problems - congratulations!
> My problem addresses the use of variable dimension sizes in NetCDF
> files. As I understand (refering to your (very good) reference guide),
> it is impossible to delete existing dimensions or to change dimension
> sizes in a NetCDF file. Varying a dimension size results in a copy of
> the complete file. I expect a size of 1.5 Gigabytes of my NetCDF file
> - - making a copy would be an expensive job. On the other hand, I can
> not manage my data without changing dimension sizes because I store
> CFD point data or point data of wing geometries for example, which
> often come with different dimension sizes. Using an unlinited
> dimension is not a solution either, because the variables I use can
> not grow together along that unlinited dimension. I probably could use
> new dimensions for new variables (if they have different sizes), but
> how could I manage the old stuff in the file? And as I know, the
> number of dimensions in a NetCDF file is limited to 100.
> Could you give me some help or some suggestions?

First, let me apologize for not responding to your question more
quickly.  I somehow lost track of it and then went on vacation for a
week, but just now discovered the question.

Using new dimensions for new variables is not a practical solution,
because adding a new dimension, which is a change to the "schema" for a
netCDF file, results in copying the file too, just as adding a new
variable or attribute or even renaming a netCDF component with a longer
name requires copying the file.

Since netCDF currently supports only one unlimited dimension per file
and you need multiple unlimited dimensions that change independently,
there are a few "work-arounds" I can suggest to this netCDF
limitation, all of which have their own disadvantages:

1.  Accept wasting some space by dimensioning variables to a maximum
    size.  This is unacceptably wasteful if the sizes of the changing
    dimensions vary widely.

2.  Use multiple netCDF files for your data, one for each unlimited
    dimension.  This is inconvenient because separate files would have
    to store their own copies of common netCDF components, and it would
    not work if you had a single variable that used more than one of the
    changing dimensions.

3.  Use indirection in your netCDF file to implement something like a
    "heap" for storing values from multiple variables, using separate
    index variables that point into the heap.  The heap would be a
    single variable that uses the unlimited dimension, and appending a
    new value to a variable would involve appending the value to the
    heap variable as well as updating an index variable to point into
    the heap.  You might still need to use a maximum size for each
    associated index variable, so this also wastes some space.  This is
    also fairly low-level, so you would need to write an access layer
    over netCDF that hides this implementation.  It would also not work
    well if you have data of multiple types that need to grow
    independently.

You can also see the example of storing a "ragged array" that's given in
the User's Guide section on "Data Structures", but you may have already
noticed that that solution doesn't work very well if row sizes grow
independently.

The netCDF limitation of only one unlimited dimension per file is
fundamental to the current format and implementation.  We have
considered ways to eliminate this limitation in netCDF 4, which will use
a new format, but our plans currently call for implementing packed data
first, so I can't promise that will be available soon.

The limitation of 100 dimensions per netCDF file is not fundamental, and
can be changed by modifying one statement in an include file and
recompiling.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu