[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #CBQ-986702]: Why is nc_put_varm unbearable slow?

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #CBQ-986702]: Why is nc_put_varm unbearable slow?
  • Date: Sun, 17 Mar 2013 17:58:02 -0600
Florian-

The difference in speed is (probably)
due to the layout of the data on disk.
Netcdf-4 is built over the HDF5 library,
and HDF5 stores its data using a form of
b-tree. Netcdf classic, on the other hand
lays out the data in, more-or-less, a linear
form on the disk.

Hence, to write a single variable element in
netcdfclassic requires basically a disk seek
to the position of the element followed by a
write of the single data element. Accessing the data
thru a b-tree is inherently slowert. However, when writing
many consecutive values, HDF5 is substantially more efficient,
hence you should see much less difference when using nc_put_vara.

> ...Do you mean that the loop within nc_put_varm occurs when writing the 
> elements to
> disk? I would have expected that you allocate a dedicated buffer into which 
> the
> elements are presorted before being written to disk in chunks via 
> nc_put_vara. This 
> should be quite fast actually.
Potentially, but it is rather complex to do this. If you care to rewrite the
varm code to do it, we would certainly be grateful.



=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: CBQ-986702
Department: Support netCDF
Priority: Normal
Status: Closed