[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: NetCDF structures and Profile/in situ data

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: NetCDF structures and Profile/in situ data
  • Date: Tue, 02 Aug 2005 12:18:42 -0600
Roy Mendelssohn wrote:


Hi John:
On your web pages, you have some very interesting suggestions on how to use structures and linked lists within netcdf to store profile or in situ data. I am wondering if this is being used by anyone on real data sets, and there experience with access times etc. etc. 

Any info appreciated.

Regards,

-Roy M.


Hi Roy:

The latest draft of that is at

http://my.unidata.ucar.edu/content/software/netcdf-java/formats/UnidataObsConvention.html
we still havent finalized it, however we are using it in the IDD metar decoder, a sample file (50 Mb) can be downloaded at 

http://motherlode.ucar.edu:8088/thredds/fileServer/station/metar/20050802_metar.nc

in this catalog:

http://motherlode.ucar.edu:8088/thredds/catalog.html
we have not yet done any timing on this, but we know that for our current purposes (implementing StationObsDataset in CDM) it has good response time when reading from local files (not so good for remote files dure to latency). The queries we optimize for are "get all obs at a station", and "get all stations in a bounding box". The metar file above has 25 - 50 observations for each of 4000 stations. Each obs probably requires a disk access / network round trip. If you are writing these files with complete control over the layout, id use the contiguous list layout. (That assumes you want to optimise the "get all obs at a station".)
we havent yet written or tested files for trajectory or profile data yet, but we will.
let me know what plans you have, perhaps we can work together to finalize this and get some tests run. 


John