[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #OPQ-508225]: netCDF General - Record data with two unlimited records

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #OPQ-508225]: netCDF General - Record data with two unlimited records
  • Date: Fri, 23 Mar 2007 14:40:37 -0600
Hi,

Here are a few suggestions:

 - Use indirection, by having "index" and "length" variables for each record, 
providing the location in a single large variable array for where the data 
starts for each frame and how many values there are for that frame:

  dimensions:
    frame=unlimited;
    numvals=1000000; // total number of values for all frames
  variables:
    int index(frame);  // where the values for each frame start
    int length(frame); // how many values for each frame
    float values(numvalues);
   ...

 - Use two passes, storing the data in a different organization as it comes in, 
and later making a separate pass over the raw data to store it as a better 
organized netCDF file.  That way, you will know exactly how many values are in 
each frame and you would not have to use an "unlimited" dimension, because you 
would also know how many frames there were.  For example, instead of one 
variable, you could use three: a small, medium, and large variable, each 
containing all the values for small-sized, medium-sized, and large-sized 
frames, with a separate variable telling for each frame, whether it is small, 
medium, or large and where its values are.
 
 - Try netCDF-4, which is currently available for testing.  It supports both 
multiple unlimited dimensions and a "variable length" user-defined type for 
structures like ragged arrays.  However, you would have to wait for the 
released version of netCDF-4 (expected in a few months) to make sure the format 
was supported and future netCDF programs could read it.

 - use HDF5 instead of netCDF, because HDF5 already supports variable-length 
types and multiple unlimited dimensions.  However, HDF5 is somewhat more 
complex than netCDF-3 or netCDF-4 and is not supported by as much third-party 
software.

You might also look at the solutions and conventions for multi-rate data used 
for aircraft data, since it solves a similar problem with netCDF-3:

  http://www.eol.ucar.edu/raf/Software/netCDF.html

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: OPQ-508225
Department: Support netCDF
Priority: Normal
Status: Closed