[netCDF #CBQ-986702]: Why is nc_put_varm unbearable slow?

Florian-

The difference in speed is (probably)
due to the layout of the data on disk.
Netcdf-4 is built over the HDF5 library,
and HDF5 stores its data using a form of
b-tree. Netcdf classic, on the other hand
lays out the data in, more-or-less, a linear
form on the disk.

Hence, to write a single variable element in
netcdfclassic requires basically a disk seek
to the position of the element followed by a
write of the single data element. Accessing the data
thru a b-tree is inherently slowert. However, when writing
many consecutive values, HDF5 is substantially more efficient,
hence you should see much less difference when using nc_put_vara.

> ...Do you mean that the loop within nc_put_varm occurs when writing the 
> elements to
> disk? I would have expected that you allocate a dedicated buffer into which 
> the
> elements are presorted before being written to disk in chunks via 
> nc_put_vara. This 
> should be quite fast actually.
Potentially, but it is rather complex to do this. If you care to rewrite the
varm code to do it, we would certainly be grateful.



=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: CBQ-986702
Department: Support netCDF
Priority: Normal
Status: Closed