[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: netcdf dods parsing help! :)

Tennessee James Leeuwenburg wrote:


Hi John,

My name is Tennessee. I've been chatting to Nathan Potter and James
Gallagher about writing a custom DODS server. What I've got is a custom
backend database which doesn't support SQL, and doesn't have any network
interface. My access to it is through a command-line C++ program which
handles all the networking in a black-box way.

So, I want to put a dods server in front of my black-box query tool to
provide some fairly standard kind of interface. The black box takes
queries written in its' strange query language, then gives back netcdf
files. (Actually, grib files which I convert using g2nc). The short version of my current problem is as follows: The dods.servers.netcdf code, which I believe is stuff that you have 
written, generates its' DAS and DDS responses by parsing netcdf files
off disk. In my case however, I don't have the netCDF files for my
datasets to hand prior to the dods request for binary data. I've written
a DAS and DDS file for parsing describing the contents of the dataset,
but there are some obvious gotchas here.

Basically, I am fairly certain that the dds / das files I have written
will not be of the same form as that created by the netcdf parser in
that package. What I had hopes to do was use a dummy ServerDDS (maybe
from the testserver) to handle the dds/das requests, then when a dods
request came through, run my database query and create a new netcdf
ServerDDS from the temporary file created by that query.

The netCDF files basically contain model output, being a number of
parameters at regular gridpoints across a number of times and vertical
levels.

I described the data as being made up of Sequences. However,
Nathan/James thought that for various reasons this was a Bad Thing.

Anyway, I was hoping you could talk to me about how best to address the
issue of making sure that my dds/das requests matched the dds that gets
generated by your netcdf parser.

I also realise that maybe it's not going to be possible to use your code
as is for serving my netCDF files.

Sorry if any of this is unclear.

Thanks,
-Tennessee

Hi Tennessee:

sorry its taken me awhile to get to this.
If i understand it, you need to generate an accurate DAS and DDS that reflects what possible data a user can request, but the way the binary data request gets processed (dataset -> grib -> netcdf -> netcdf server) the binary response DAS, DDS probably wont match it correctly. 

So the obvious things you can do:
1. write your own server to map the netcdf file into the DODS response. this isnt all that hard most of the time, because the DODS and Netcdf data models are so similar.
2. modify the netcdf server to do the right thing. here, you have to understand my body of code and then figure out how to modify it without breaking it. This code isnt writtten as well as i wish it was, because i was marrying my code with existing dods java library, and so i was fiddling around a lot. Still it may be faster than starting anew.
3. use NcML to access the netcdf files, which allow you to redefine much of the netcdf file. this may be adequate, but NcML cant do everything. see http://www.unidata.ucar.edu/packages/netcdf/ncml/
I cant tell whats better unless i can see how your DDS looks compared to what the netcdf server returns in the DataDDS