Re: information about NetCDF file structure

> Organization: CIRES
> Keywords: 199405111718.AA11117

Hi Jim,

> I picked up the latest NetCDF User's Guide via ftp.
> But I would like more detailed information about
> the physical structure of a NetCDF file.
> The data access and analysis system I am working on
> divides the description of data into a symbolic or logical component
> and a file system component.
> The symbolic component is essentially a programming language
> that includes the kind of stuff in your CDL.
> Developing this language is the problem at hand.
> But I am also looking forward to developing another language
> specifically designed to describe physical file formats.
> I am convinced that I can so describe any kind of file that can
> be written by Fortran writes on any kind of system,
> and I have thought through certain useful structures
> for handling random access to variable-length data.
> But NetCDF represents a very practical real example
> of such random access, and it would be useful as a target
> for my development.
> So, if you can point to a document on your ftp server,
> or somewhere else, that indicates the detailed structure of
> a NetCDF file, I'll pick it up.

We don't have such a document for several reasons.  First, there is a
chapter in the netCDF User's Guide on "NetCDF File Structure and
Performance" that explains the physical structure of netCDF data at a high
enough level to make clear the performance implications of different data
organizations.

Second, we don't want netCDF users to write programs that depend on the
physical representation of netCDF data.  If they did that, we would not be
free in the future to change the physical representation.  If users only go
through the documented interfaces to access the data, any changes we make in
the future physical representation will be transparent to current users.

Finally, the file structure is completely specified by the source code, and
by the description that it is the XDR-encoding of the NC structures defined
in netcdf/libsrc/local_nc.h.  Since XDR is specified elsewhere in a separate
document, we didn't want to copy that specification but instead just refer
to it.  That specification is available via a WWW browser such as Mosaic or
via gopher at

        gopher://ds.internic.net/00/rfc/rfc1014.txt

I realize this is not a very satisfactory answer for your purposes in trying
to test a language for representing physical file formats.  But I think any
such language must be able to handle multiple levels of representation.

For example, data might be physically represented in a compressed "tar" file
of XDR structures.  In this case, describing the physical representation of
the data would require describing the way tar stores directory hierarchies
in a single file, the way the compression algorithm works to compress strings
of bytes dynamically, and the way XDR encodes arrays of primitive types.
I'm not sure how useful it would be to have a language that could capture
everything that's going on at these different levels of representation.

Anyway, I hope this helps explain why I can't point at a single specific
document.

--Russ

__________________________________________________________________________
                      
Russ Rew                                              UCAR Unidata Program
address@hidden                                        P.O. Box 3000
(303)497-8645                                 Boulder, Colorado 80307-3000