Re: in memory netcdf

>To: address@hidden
>From: ahoward <address@hidden>
>Subject: Re: 20020327: in memory netcdf
>Organization: NOAA/FSL

Hi Ara,

> don't know if you recall my name, but i was one of the early users of your
> netcdf java library a few years back.  in any case, i've switched
> departments here at fsl and am now in the ITS group responsible for the
> realtime data ingest.  we're currently in the process of revamping our
> systems design and implementation language (c -->> c++).  i'm working on
> generic handling of point data and need a good general data structure in
> which to store all point data for eventual munging into output files of
> either netcdf or grib format.  netcdf itself would be an ideal data
> structure as the metadata contained in the file is complete enough for my
> purposes (with some creative uses of attributes) and the typed accessor
> methods (c++ interface) are exactly the type of thing needed.
> 
> my problem is that any use of netcdf requires a disk hit to read in an
> existing cdf (output of ncgen) and each subsequent put/get also hits disk.
> i'm looking into replacing the IO layer (posixio.c) with a layer that
> operates entirely in memory.  that would leave only the hit to read in an
> existing template (i'm also planning on looking at ncgen to see if it
> could be modified to place a netcdf file structure in memory instead of on
> disk).  eg. i want to eliminate most, if not all, of the disk accesses in
> the c/c++ netcdf api.
> 
> alternatively, applying the same set of requirements, and availibility of
> a constructor to the NcVar class might prove sufficient for our purposes.
> 
> my questions are :
> 
>       *       do you know of any work along these lines, 3rd party or
>               otherwise?

Andrew Janke <address@hidden> proposed implementing something
similar in the Java interface, which you can read about in this reply
from John Caron to his question:

  http://www.unidata.ucar.edu/cgi-bin/mfs/70/4439

Glenn Davis also once implemented an experimental memory-mapped
version of the i/o layer for netCDF 3.4 built on top of Unix mmap(2),
which is described here:

  http://www.unidata.ucar.edu/cgi-bin/mfs/70/2756

and available in this file:

  ftp://ftp.unidata.ucar.edu/pub/netcdf/contrib/mmapio.c

that saves some disk I/O.

>       *       am i completely off base in thinking the IO layer could be
>               replaced with an in memory IO layer
> 
> any input appreciated, i was planning on cc'ing john caron on this as well
> but have lost his email.

The I/O layer could be replaced with an in-memory layer.  But if you
just want to access a small slice of data from a huge file, using
random-access to read from the disk is cheaper than reading the whole
file into memory first before you try to access the data slice from
the in-memory copy.  But it sounds like that's not a concern for your
application.

By the way, Charlie Zender has developed a new C++ interface to
netCDF, known as libnco_c++, which he describes in this netcdfgroup
posting:

  http://www.unidata.ucar.edu/glimpse/netcdfgroup-list/1484

which emphasizes easy migration from the C interface ...

--Russ