Re: Netcdf4 Compound types and Examples

The file  (12 MB I believe) is at:

ftp://orpheus.pfeg.noaa.gov/outgoing/roy/subsurf199808.hdf

This is monthly GTSPP data rather than the World Ocean Database.

HDFView was able to open the file with no problem.  To see how it  
fakes a hierarchy, open the temperature point and look at the Data  
VGroup and the Linkage VGroup.    So all the data is actually in one  
flat file, but the linkage vgroups map to the data in the higher  
level, so it tells you which box2's belong to a box10, and then which  
records in the flat table belong to any box2.  With a true hierarchy  
you can do the same with groups and subgroups

But as I said, we had to have for the observed parameters as many  
elements as any observation had a depth as that could not vary.

HTH and I think I am beginning to see an easy way to do a portable  
interface at least for vlens, I don't know about compound types.

-Roy


On Dec 27, 2007, at 3:26 PM, Russ Rew wrote:

> Roy,
>
> > Okay guys, here is the "example" we are working on.  A long time ago
> > we put the  World Ocean Database and the GTSPP data, all subsurface
> > obs that we have or get, into the old HDF-EOS point structure. The
> > way we had it structured was that each file was a 10-degree square
> > with a name that contained the box number as in COADS.  The HDF-EOS
> > point structure had levels that made what was a series of flat tables
> > look like a hierarchy by having a link field.  So level 0 data would
> > be all the box 2's  in the box 10, and the level 1 data would be the
> > data with the box2 number as the link field.  This allowed you to
> > search for all the data in a box2 efficiently
> >
> > A downside was that the number of rows in the field had to be equal
> > to the sum total of depths at which there were observations.    This
> > meant that there is a huge amount of empty space.
>
> I understand the COADS box numbers for 10 degree or two degree boxes,
> but I don't understand what you're saying about levels and link  
> fields.
> Is there a simpler or more detailed description of this data structure
> somewhere that I could study?  I'd like to understand the requirements
> and whether netCDF-4 could eliminate the wasted space while preserving
> fast access.
>
> >                                               ...  So how might we
> > improve on this - why netcdf4!  We use vlen's to deal with the
> > varying number of depths and a compound data type to combine together
> > the info.  Groups and (sub)-Groups provide the same structure that
> > was "emulated" in the point structure.  Or at least that was what we
> > thought would be a good design given netcdf4's features, but that is
> > what prompted my email about examples and design - there may well be
> > better ways of doing this.
> >
> > However, unless I miraculously find both money and a good C
> > programmer, or get Fortran wrappers to work, it will be sometime
> > before it happens.
> >
> > BTW, at one point I used the hdf4tohdf5 program to convert these
> > files to HDF5, so they can be read using that library.  I don't know
> > that the conversion worked that cleanly to use with the HDF-EOS5
> > library, but it should be possible to both read and write using the
> > libs that are a part of Netcdf4.  I actually had a few days to begin
> > to work on this, hence the series of emails.
> >
> > -Roy
> >
> >
> >
> >
> > **********************
> > "The contents of this message do not reflect any position of the U.S.
> > Government or NOAA."
> > **********************
> > Roy Mendelssohn
> > Supervisory Operations Research Analyst
> > NOAA/NMFS
> > Environmental Research Division 
> > Southwest Fisheries Science Center
> > 1352 Lighthouse Avenue
> > Pacific Grove, CA 93950-2097
> >
> > e-mail: address@hidden (Note new e-mail address)
> > voice: (831)-648-9029
> > fax: (831)-648-8440
> > www: http://www.pfeg.noaa.gov/
> >
> > "Old age and treachery will overcome youth and skill."

**********************
"The contents of this message do not reflect any position of the U.S.  
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division 
Southwest Fisheries Science Center
1352 Lighthouse Avenue
Pacific Grove, CA 93950-2097

e-mail: address@hidden (Note new e-mail address)
voice: (831)-648-9029
fax: (831)-648-8440
www: http://www.pfeg.noaa.gov/

"Old age and treachery will overcome youth and skill."