Re: C++ interface and efficient multi-dimensional arrays (compression?)

> To: address@hidden, "Glenn P. Davis" <address@hidden>
> Subject: C++ interface and efficient multi-dimensional arrays (compression?)
> Date: Thu, 24 Dec 1998 15:08:59 -0700
> From: Joe Van Andel <address@hidden>

Hi Joe,

> March 1998, Russ said:
> > However, we may provide a new standards-based C++ interface written
> > "from the ground up" rather than as a veneer over the C interface.
> > This would probably be more like the new Java interface than the old
> > C++ interface.  Glenn has written one of these (all C++ netCDF), but
> > the available compilers did not handle exceptions or templates
> > according to the standard, so it was never completed or released. 
> 
> Now that EGCS 1.1 is released, with good support for templates (supports
> handles STL) and exceptions, is there any thought to completing and 
> releasing a better C++ interface?

Improvements to the C++ interface don't currently have a high priority
on our list of netCDF enhancements.  Your's is one of the few requests
we've gotten for this.  It seems there are far more users of the
Fortran and C interfaces than the C++ interface, judging by the
support questions we get.

We didn't get approval for all the resources we requested in our last
NSF proposal
<http://www.unidata.ucar.edu/proposals/NSF2003/2003.index.html>, and
as a result we've had to put most of our netCDF development plans on
the back burner.  Lately we've been concentrating our development
resources on netCDF for Java.

> Also, we have an ongoing need to efficiently store data with 2 or more 
> variable sized dimensions.  E.g., radar data has a variable number of 
> range gates and a variable number of rays (beams).  Currently, we have 
> to fix one of these dimensions at its maximum size, and simply waste 
> the space that isn't used.  E.g. we're only using 500 gates for a 
> particular data set, but the arrays are dimensioned for 1500 gates.
>
> I've seen some discussion on the netCDF mail list about a patch that 
> uses compression to avoid the wasted space on disk.  I saw a posting 
> that you aren't interested in adding compression as a standard part of 
> the netCDF library, but had considered simple run length encoding.

The zlib patch implemented by Bill Noon
<http://snow.cit.cornell.edu/noon/z_netcdf.html> is getting
significant use at the Regional Climate Centers, and perhaps by others
as well.

Our proposed solution to making netCDF files smaller is packing, not
run length encoding.  This would add packed data types for arrays of
n-bit data, for example.  This is described in more detail in the
paper "Unidata's NetCDF Interface for Data Access: Status and Plans"
<http://www.unidata.ucar.edu/packages/netcdf/ams97.html>.

> Any idea on when you might release a standard solution to aid the use
> of netCDF for datasets that require 2 or more variable size dimensions?
> 
> We could use netCDF for many more applications if there was a standard, 
> supported solution to this problem.

This requires a format change from the version 1 format used for all
netCDF files for the last ten years.  We had decided to bite the
bullet and implement the next version of a netCDF format in terms of
the HDF 5 "big file" format, thus leveraging the support available for
various improvements in HDF 5.  These netCDF improvements are
currently scheduled to start in Q1 of 2000 and continue through Q4 of
2001, according to our Revised Milestones at
<http://www.unidata.ucar.edu/proposals/NSF2003/RespondReviewPanel/attachmentb.html>,
a new schedule developed in response to discussions about our expected
budget from NSF.

So, in summary, I'm sorry I can't give you any assurance that these
planned improvements will be developed soon, but at least the
knowledge that no solution is imminent might help with your planning.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu