[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: NetCDF/HDF5 Proposal Approved

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: NetCDF/HDF5 Proposal Approved
  • Date: Mon, 17 Mar 2003 09:33:34 -0700
Hi Roy,

> Congrats on getting the grant. Two comments or concerns that I am 
> sure you will get from everyone is first good conversion tools (we 
> have had a lot of trouble with the hdf4->hdf5 and hdf-eos->hdf-eos5 
> not working entirely correctly).  We have many gigabytes of data in 
> netcdf at this point, so easy and correct conversion is a must.

Our plan is to support reading current netCDF data through the new
interface, so no conversion of old archives should not be needed for
reading data through the netCDF-4 interface.  We may even be able to
support writing current netCDF data in the current format for
applications that don't need any of the new features.

Another goal for upward compatibility is to provide some of the
benefits of netCDF-4 by merely recompiling existing netCDF
applications, although it will be necessary to use new open or create
calls to specify non-default parameters of new features, such as
chunking.

> Second, some of the timings made by the HDF folks suggests that HDF5 
> might be slow compared to what you are using now (though there are 
> other benefits that may  offset the speed).

We have complaints from modelers that netCDF is a bottleneck on
parallel platforms, forcing them to switch to HDF5 with its parallel
MPI-IO.  HDF5's support for chunking can also yield significant
speedups for some patterns of access.  One of the objectives of the
project is to realize demonstrable performance gains in modeling
contexts on advanced architectures.  I'm hopeful that we can provide
some speedup, but that's why they call it "research" ...

--Russ