Re: 19990121: Question in variable data structure

>From: raytheon <address@hidden>
>Subject: Question in variable data structure
>Organization: Raytheon
>Keywords: 199901212341.QAA27513 netCDF data storage

Hi Jeff,

Sorry to take so long to reply to your question, it "fell through the
cracks" and we just found it ...

>     If I have a 1k x 1k data array and have taken 300 frames of data.
> What is the best way to store the data in NetCDF.  Should I keep all the
> data together in a data cube consisting of (1000,1000,300), or should I
> create a variable for each frame of data.  Thus having 300 1k x 1k 2D
> array.  Which one will allow me quicker data access to all of the data.

You should be able to access the data just as quickly either way,
since netCDF uses direct access.  But there are advantages to storing
all the data in a single array, because you can access multiple frames
or cross-sections of the data with a single call, whereas if the data
is stored in multiple variables, all the data you can access in a
single call must come from the same variable.

You didn't say whether you are using the C or Fortran interface (or
maybe the C++, Java, or perl interface?).  The data variable should be
dimensioned so that 300 is the most slowly varying dimension if you
mostly intend to access the data a frame at a time.  That way the
1000x1000 blocks of data will be stored contiguously in the files, so
access will be fast.  If you stored the data with the frame dimension
(300) varying fastest, reading a frame would involve reading every
300th value from the disk, which would likely be much slower.

The CDL notation for this uses the C convention, where the last
dimension varies fastest:

  dimensions:
     frame = 300;
     m = 1000;
     n = 1000;
  variables:
     float data(frame, m, n);
     ...

But if you are reading the associated data into a Fortran array, it
should still be dimensioned following Fortran conventions, with the
first dimension varying fastest.

I hope this answers your questions ...

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu