Re: incompatible changes in NetCDF?

Hi Tom,

> I have a scientist who is considering moving lots of processing output 
> to netCDF (which I, of course, have been encouraging).  He wrote me, 
> however:
> 
> "The FAQ mentions that a version 4 is planned that will include a number 
> of improvements.  My concern is that like HDF 4 & 5, netCDF 3.5 & 4 will 
> be mutually incompatible.  I was just wondering if you knew anything 
> about these plans and what kind of timeline we might see for this 
> upgrade?  (or maybe you know whom I could ask?) "
> 
> The community is so disenfranchised with HDF because of this 4->5 
> incompatibility (among other things) that I thought it might be a good 
> idea to change the FAQ to talk about backward compatibility for netCDF 
> versions.  I would also like to pass your words ("stright from the 
> developer") to him...

Thanks for passing this on.

We consider backward compatibility a very important issue, and I will
try to make this clear in an updated FAQ entry about our netCDF-4
plans.

As you may know, netCDF 3.5 still uses the same format as netCDF 1.0
first devised in 1988, so we haven't changed the format yet, but we
have changed the interface.  NetCDF 3 supported a completely different
interface from netCDF 2 for both C and Fortran, but we provided
backward compatibility by layering the netCDF 2 interfaces on top of
the netCDF 3 interfaces, so old programs still worked with a
recompilation.

With netCDF 4, there will two major compatibility issues:
 
 - a different format, based on HDF5
 - an extended interface, netCDF-4

We have to change format, because the original netCDF format has run
out of steam after 15 years, since its 31-bit offsets can't support
very large files.

Our plans are to continue to support reading and writing netCDF format
files by detecting on open whether a file is netCDF or HDF5 and using
the appropriate functions.  So archives of netCDF data will still be
accessible through the new interface, and it's our aim to support the
ability to modify and extend netCDF files with code that currently
works, by just recompiling.  When you create a *new* file through the
netCDF-4 interface, the default will be to use the new HDF5 format,
but there will be a way to specify that netCDF-format files are
created instead.

The extended interface will be, as much as practical, a superset of
the current netCDF-3 interface, so current programs should work by
recompiling and relinking them to the new library.  

As an interim step and proof of concepts, we're currently implementing
the netCDF 3 interface on top of HDF5, without many of the
enhancements that will be in netCDF-4.  Much of the work is to achieve
the two kinds of backwards compatibility described above.  So far, we
haven't run into any "show stoppers" ...

--Russ