Re: NCO, SDO, netCDF-HDF, libnco_c++

> From: Charlie Zender <address@hidden>
> To: Russ Rew <address@hidden>
> Subject: NCO, SDO, netCDF-HDF, libnco_c++
> CC: Henry Butowsky <address@hidden>, Rorik Peterson <address@hidden>
> Date: Mon, 18 Aug 2003 11:22:13 -0700

Hi Charlie,

> I need some info. on what the plans are for merging netCDF with HDF, 
> implementing an HDF-native backend to the netCDF API, etc.
> The reason is that I plan to submit a proposal to NASA this year.
> I missed last year's SEEDS proposals, so I'm not sure what NRA yet.
> The basic idea is an HDF-native version of NCO called SDO. 
> SDO would extend the "file is a unit of data" concept to HDF files.
> Last year I thought SDO would be a re-write of NCO where we abstract
> the library to branch into either HDF or netCDF native calls depending
> on the input/output file format. 
> Since your netCDF-HDF proposal was funded, it's possible that NCO 
> will simply "work" on HDF files when you are finished, without any
> HDF-specific code in NCO. 
> That would be great, although the real situation is probably more
> complex that that. 
> In any case, I want to refine my ideas based on what you expect to
> produce so I can draft a proposal that does not reinvent the wheel.
> So I need a peek at your plans. Please keep me posted as to any
> milestones you reach or major changes in plan you take. 

Our proposed plans are here:

  http://www.unidata.ucar.edu/proposals/NASA-AIST-2002/Description.pdf

and we're currently implementing netCDF-3 on top of HDF5 as a warm up
and to better understand the problems we will encounter in adding
enhancements to produce netCDF-4 on top of an enhanced HDF5.  This new
implementation of the netCDF-3 interface using the HDF5 file format
may be ready for testing fairly soon.  I'll certainly let you know
about it when it's ready.

Some draft information on the features that we hope will comprise
netCDF-4, which is not ready yet for widespread release or comment, is
here:

  http://my.unidata.ucar.edu/content/software/netcdf/netcdf-4/index.html

This list is ambitious enough that we may eventually have to drop some
of the items that have been added since the original proposal, hence I
don't want to commit to them at this time.

For your purposes, it looks like you may have to treat arbitrary HDF5
files separately, since we don't intend that you will be able to
create or access arbitrary HDF5 data through the netCDF-4 interface.
Here's an excerpt from some email I wrote about this:

  Initially, we think there should be one library but two sets of
  interfaces:
    - HDF5 interfaces, to support all current HDF5 programs;
    - netCDF interfaces, to support current netCDF-3 programs and
      the extended netCDF-4 interfaces.

  Off the top of our heads, we don't think it would be practical to try
  to support all HDF5 functionality through netCDF-4 interfaces.  Files
  written through a netCDF-3 or netCDF-4 interface should be readable
  through a netCDF-4 interface, but some information in some HDF5 files
  would not be readable using just netCDF interfaces.  On the other
  hand, all HDF5 files created through the netCDF-4 interface should be
  readable using HDF5 interfaces.  So we look at netCDF-4 as a smaller,
  simpler, and less general interface than HDF5.

  ... Since netCDF-4 has to be able to read/modify current netCDF
  files for backward compatibility, [a modified] netCDF-3 will need to
  be part of the library.  On opening a file, if it is determined to
  be the old netCDF format, the old netCDF-3 library handles it.  We
  think this is a relatively simple way to provide backward
  compatibility, without requiring changes to the current HDF5 library
  to handle old netCDF files.

--Russ