Re: 19990423: ** NetCDF - simple question **

>To: address@hidden
>cc: address@hidden
>From: James T Brown <address@hidden>
>Subject: ** NetCDF - simple question **
>Organization: Geography/Fisheries & Wildlife, Michigan State University
>Keywords: 199904231301.HAA16897

Hi Jim,

> I am pretty familiar with your NetCDF package and
> data format.  I used it quite often in the three
> years that I worked at NOAA's Climate Diagnostics
> Center in Boulder.  
> 
> I am now partly working on some projects for the
> Ag Meteorologist at Michigan State University.  We
> will be setting up a network of automated weather
> stations throughout the state and I am looking for
> a means to store, archive, and access the data.
> A previous programming has setup some Perl scripts
> to store the data using a SQL database - mSQL.

The Oklahoma Mesonet had a similar requirement, and I'm not sure what
they ended up using, although it wasn't netCDF.  I found this
description at http://www.nssl.noaa.gov/~schuur/meaprs/meaprs.html

    The Oklahoma Mesoscale surface Network (Oklahoma Mesonet) is
    operated by the Oklahoma Climatological Survey (OCS) and
    consists of 111 automated sites. 5-minute data are received at
    OCS, in Norman, Oklahoma, where they are quality controlled and
    archived. All mesonet sites measure the standard surface
    meteorological parameters, with some sites taking additional
    measurements from specialized instruments.

> I have never used NetCDF for station data.  It was
> always gridded model data.  I believe NetCDF's
> strength may lie with gridded model data.  I am 
> looking for different methods to possibly work 
> with our station data.  I am sure that a number
> of sites are working with station data.  Do you
> have any idea what the majority of them are 
> using?  Do they make use of NetCDF or are most 
> of the sites using other means (simple ASCII
> files or possible expensive database schemes).

I think there is quite a variety of data formats used, but I don't
know what the majority use.  Although we have netCDF representations
for collections of decoded station data (see, for example, some of the
CDL files in the directory ftp://ftp.unidata.ucar.edu/pub/netcdf/cdl/,
such as fsl-mesonet.cdl fsl-sao.cdl, sao.cdl, or the files produced by
our perl decoders in the freely available decoders package from
http://www.unidata.ucar.edu/packages/decoders/), we don't get many
questions about these, so I doubt they are being used much.  

Also, storing decoded data rather than the raw data usually takes
more space.  And netCDF is not ideal when there is a large variance in
the amount of data from each station.  Typically, you have to choose
some maximum (for example the maximum number of significant levels for
upper air data), and use that for all the stations, which can waste
space.

> We will be looking to store both hourly and 
> daily station data and we are looking for a 
> format that will be not only easy to work with,
> but allow for quick data access and retrieval.
> It is our hope to make our dataset or database
> of automated weather station data available
> via the Web.
> 
> Any help or suggestions would be most appreciated.

Quick data access and retrieval may dictate a database solution,
especially if you need to access the data by several different keys
and you have a large number of stations.  However if the number of
stations is small enough, using netCDF might work OK.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu