Re: 20040402: netCDF help request

>To: address@hidden
>From: Nithya Nirmal Vijayakumar <address@hidden>
>Subject: Need help
>Organization: UCAR/Unidata
>Keywords: 200404022357.i32NvfCT000994

Hi Nithya,

> I am new user of netCDF from Indiana University and I will need a
> little help getting started. I can find good documentation from the
> website and almost read all of them. But I would like to use netCDF
> to send a data defined by a C structure across the network (from
> provider to receiver). Can I use the usual method of using struct,
> pointers in C to create the data and then use a corresponding netCDF
> format to send it across? or Should I create the data itself as a
> netCDF dataset?  Can you please provide me with a simple
> reader-writer example for sending a record defined by a C structure
> using a corresponding netCDF format?

netCDF may not be appropriate for this application, because netCDF
does not directly support C structs.  This is because netCDF data must
be accessible from Fortran programs through the Fortran interface, and
Fortran 77 does not support structs, only primitives and arrays of
primitives.  It's possible to emulate C structs in netCDF, but that's
a convention that you would have to enforce and interpret without help
from the netCDF interface.

For example, if you wanted to represent an array of structs of the
form

 struct data {
   int id;
   double lat;
   double lon;
   float level[4]
   float temperature[4];
 }

in a netCDF file, you could use a file that was structured something
like (in CDL notation): 

netcdf structdata {
dimensions:
  n = unlimited;  // the record dimension
  level = 4;
variables:
  int id(n);
  double lat(n);
  double lon(n);
  float levels(n, level);
  float temperature(n, level);
}

and each "record" would contain the data corresponding to one element
of the array of structs.

You could certainly create the data as a netCDF dataset and send that
from one host to another, but all netCDF would give you would be a
representation of the data that is portable between machines.  The C
version of netCDF has no support for remote access.  (The Java
interface does support remote access in a couple of different ways,
but I'm assuming you want to use C or C++ for your application.)

Also, from your brief description, it sounds as if you would like to
send one struct at a time from provider to receiver.  With netCDF, you
would have to write all the struct data to a file, send the whole
file, and open the file and read from it on the receiver side.  This
might add latency to your data and would be another reason netCDF is
inappropriate, if you are depending on timely delivery.

A simpler way to do what you want might be to use XDR for representing
the C struct and use RPC (remote procedure calls) to get the struct
from sender to receiver.  If you look at the documentation for the
RPC library ("man rpc" on a Unix system), you will see that the RPC
library automatically uses the XDR representation for data to make it
portable, and can easily handle arbitrary C structs.

Other possibilities include using some other RPC or remote services
framework, although most of the newer ones would have you convert your
data to XML (a text representation) instead of leaving it in a binary
struct, and that has its own problems.

If you need to send a lot of data from one place to another, you also
might look at the Unidata LDM, which supports sending binary data
and small or large products in an event-oriented system:

  http://my.unidata.ucar.edu/content/software/ldm/index.html

For this, you would still need to package your data into some portable
form, possibly using netCDF, XDR, XML, or text.  You would also need
to create product identifiers for each product and write a data
ingester to inject data products into the system (by adding them to an
upstream product queue).  There is a C API for doing this and several
examples with the LDM6 source code.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden          http://www.unidata.ucar.edu/staff/russ