Re: 980113: netCDF application questions

> >cc: "Tim Crawford" <address@hidden>
> >From: "Edward Dumas" <address@hidden>
> >Subject: netCDF application questions
> >Organization: NOAA/ATDD
> >Keywords: 199801132234.PAA29322 netCDF

Hi Ed,

> We are considering using netCDF for archiving of atmospheric turbulence
> data collected from an aircraft.  Is there any way to store raw voltage
> data as collected from an airborne data acquisition system and apply
> polynomial scaling coefficients (more than just mx+b) for some
> channels.  How is the best way to do this in netCDF and still maintain
> general compatibility with existing file readers (e.g. MATLAB 5.1)?  We
> would also like to apply calibrations which involve combining other
> data channels before applying calibration coefficients.  For example,
> the LiCor 6262 CO2/H2O analyzer requires pressure corrections before
> application of a 5th order polynomial for final calibration.  Is this
> possible and, if so, how would be the best way to do it?

NetCDF has no facilities for applying a user-defined function to values
before they are stored or after they are read.  It converts
floating-point values between their native representation and IEEE
representation on disk but cannot be made to apply a polynomial function
to the data at the same time.  One problem with trying to do something
this general is the non-invertability of some polynomial functions.  For
example, if x^2 were applied in one direction and x^(1/2) in the other
direction, then -2 would be stored as 4 and be read as 2, so the value
read would be very different from the value stored.

If you want to apply a function to variable values, I think the best way
to represent this in netCDF is with attributes.  For example, if some
parameter v has a huge range of positive values so that it would make
more sense to store the logarithm of the variable rather than the
variable, you might do something like this:

  float logv(dim);
        logv:long_name = "natural logarithm of v";
        logv:definition = "ln(v)";
        logv:units = "1";   // dimensionless

The use of the "definition" attribute here is just a suggestion; you may
have a better name for a way to represent such derived parameters.  In a
case like this, you might want to define a "dummy parameter" v as a
scalar, just to hang attributes such as units off of it, since the
derived parameter may be unitless.  But in any case, the interpretation
of string-valued attributes such as "definition" above is left up to the
applications; the netCDF library has no expression parsers and will not
attempt to perform any evaluation of such expressions.  There is a
separate udunits library that parses units expressions; perhaps
something similar for derived expressions would be useful in cases like
yours.

> I have also read the information on the conventions of NCAR and others.
>  This looks good, but suppose we want to develop our own convention,
> and then share the data with others...  Is it possible for others to
> use our files without having to re-link their reader software to
> include the notations of our convention (for example, using MATLAB to
> read a file with our specific convention definition).  If this is
> possible, how can we go about defining the convention? 

About the best you can do currently is to include a global attribute
"Conventions" with a value that is the URL of a document defining your
conventions.  There is currently no syntax defined for machine-readable
conventions that can be used by programs.  If you find a general
notation for such a syntax is useful, we would be interested in what you
come up with.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu