[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #QEA-741463]: NetCDF4 Structural Overhead

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #QEA-741463]: NetCDF4 Structural Overhead
  • Date: Thu, 10 Apr 2008 11:15:38 -0600
> Hi Russ:
>
> The 12 Mb is in line with what I am seeing, as I have 16,200
> subgroups.  What we are trying to do is have a file that for example
> contains a months worth of subsurface profiles  (hence the ragged
> arrays).  So for quicker access, we have a group for each 10-degree
> box, and then for each 10-degree box we have 25 2-degree subgroups.
> When we write the data into the file, we place each profile into the
> correct subgroup.  With lots of profiles, that can speed up search
> time enormously, as there are routines to find the box10 and box2 for
> any lat-lon combination.

Howdy Roy!

As you will see, I have posed your question to the HDF5 team, and also asked
them if there would be a better way to store your data.

It might be naive, but have you tried just storing all your profiles in one big
group and seeing if it really is slow? Seems like they've got some mighty fancy
optimizations and they might do better than you expect. Anyway, I have asked
them if there is a better way and we will see what they say...

My rough experiments show a per-group overhead of about 300 bytes. (With
nothing in the groups and group names of about 10 characters.)

How do you store the two ints? As separate attributes? Should you not be able
to encode those in the names of the groups?

> I can put the file on ftp tomorrow if you want, but the figures add
> up with what you say.  When I put data into it, we will see if it
> worth the effort.  Also, other than the things I have been too dumb
> to figure out, the Fortran 90 interface has been working fine.  I
> told Ed I would test it, and I haven't seen any errors per se.

I can't tell you how helpful this has all been Roy! You are my only real-life
fortran user, so you stand in for the entire netCDF fortran community. (Try not
to let it scare you. ;-)

I went to add your name to our coveted "Credits" page, but you are already
there.

I am also planning life-sized marble statues in the new Unidata Ufuzi
Workspace, to be built in place of the NCAR pink palace (which will be leveled
to make room for the new Unidata building.)

Thanks!

Ed



Ticket Details
===================
Ticket ID: QEA-741463
Department: Support netCDF
Priority: Normal
Status: Closed