Re: 961108: CDL support in netCDF library

>To: address@hidden
>From: Robert Fischer <address@hidden>
>Subject: CDL support in netCDF library
>Organization: Harvard
>Keywords: 199611082101.AA14359

Hi Bob,

> i have recently started using netCDF, and i am very impressed with it.
> i am using it for large arrays in scientific computation, as is normal.
> 
> i would also like to use netCDF for small files, which may contain only a
> little bit of data, or even just attributes.  i'd like these files to be
> easily creatable by the user.  this is good because it provides a standard
> file format for configuration files used by a program.
> 
> curretnly, i can do this with netCDF by creating CDL files and converting
> them to netCDF format.  however, i would like to be able to just read an
> ASCII CDL file just as i would read a binary CDF file.  when the
> application says ncopen(), netCDF would open the CDL file, and essentially
> run ncgen on it, storing the entire resulting CDF file in memory.  when
> the app later asks for parts of the CDF file, netCDF would just take them
> from memory.  since this would only be used for small files, it should not
> be a problem storing the entire CDF file in memory.
> 
> any plans to do such a thing?

It's an interesting idea, but it's not in our current plans.

I do something similar in one of my programs that generates names of a
netCDF files to write.  For each such file, the program first checks to
see whether the file (say "foo.nc") already exists.  If not, the program
checks whether a corresponding CDL file ("foo.cdl") exists, using a list
of paths for where to look for such CDL files.  Then it invokes

   ncgen -b -o foo.nc foo.cdl

to make sure the desired netCDF file exists.  Then I write into the file
without caring whether it already existed or had to be created on the
fly.  If you'd like to see the code for this, I can supply it.

There is currently no support for in-memory netCDF datasets, except as
provided by virtual memory; the data is ultimately accessed with open(),
lseek(), read(), and write() calls.  If I were going to implement
something like what you suggest, I'd probably create the file in /tmp/
rather than in memory (some systems keep /tmp/ in swap space anyway) and
write a wrapper around ncopen() that would invoke system() to have ncgen
create the netCDF file from the CDL file much as I do currently in the
program described above.  The wrapper would be able to open either CDL
or netCDF files and in the former case return a valid netCDF file ID to
the transient file in /tmp/.

But I'm not convinced this would be a good thing to add to the core
netCDF library, because on Unix systems it can be implemented above the
level of the library, and it would be difficult to implement portably
within the library.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu