Re: off the wall question

Hi Dennis,

> What is the status of netCDF packing? 

I'm presenting a paper on "UNIDATA'S NETCDF INTERFACE FOR DATA ACCESS: 
STATUS AND PLANS" at the AMS meeting in Anaheim next month that will
discuss this.  Here's an excerpt from the paper:

    The ability to store packed arrays of n-bit values without wasting
    space will be the first of the above limitations addressed in
    netCDF-4.  Interfaces have already been designed to add new external
    data types for packed data, and to permit transparent packing and
    scaling of limited-precision floating-point values as n-bit arrays.

    Both predefined and adaptive scaling will be supported. Parameters
    (scales and offsets) for packing will be permitted to vary along one
    or more variable dimensions. One or more exact and extreme values
    may be specified that will be preserved in packing and
    unpacking. Whether data is packed or not will be transparent to data
    readers, since the unpacking will be handled by the library. It will
    be possible to suppress unpacking and read the raw packed data, if
    desired.

    The current 2 Gbyte file size limitation will also be eliminated
    with the use of 64-bit offsets in netCDF-4.

    To support packed data and larger file sizes, netCDF-4 will require
    a new format, the first format change for netCDF. For backward
    compatibility with programs and data archives that use the current
    netCDF format, the netCDF-4 software must support access to data in
    both the old and new formats. Fortunately, netCDF already includes a
    format version number in the file format, so users and programs need
    not know whether they are accessing data in the old or new
    format. It will not be possible to add packed data to old format
    files, but otherwise the change should be relatively transparent.

You can see it's in our plans, but isn't implemented yet.  We are
currently working on getting netCDF version 3 released.  It contains the
biggest change to the C and FORTRAN interfaces required to support
packing (clean separation of internal and external types), so it's a big
step in that direction.

We're also engaged in discussions with the HDF folks at NCSA on sharing
a common file format for the next generation of netCDF and HDF.  If we
succeed in specifying a common format and low-level interface, this
would be the format used for netCDF-4.

> The off the wall question is:
> 
> When it is ready ... has Unidata considered approaching 
> the WMO about netCDF replacing BUFR/GRIB? This means
> NCEP/ECMWF switch from BUFR/GRIB to netCDF? This is not
> an option until the packing is available due to the need
> for "data packing efficiency" at the centers.  It might be that 
> Unidata may have to educate the NCEP staff ...

No, we haven't seriously considered this.  The WMO might not be very
receptive to such a suggestion, considering their investment in
developing BUFR and GRIB and how disruptive such a format change would
be.  It's a complex question, and it's not clear to me that GRIB isn't
still better at some of the things it's designed for (e.g. very compact
representation of horizontal 2-D fields) than netCDF will be, even with
packing.

--Russ