[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #ZFB-587742]: Reading variable with strides very slow

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #ZFB-587742]: Reading variable with strides very slow
  • Date: Fri, 09 Aug 2013 14:44:11 -0600
Patrick,

> On the netcdf-3 vs netcdf-4 issue I can at the moment
> only speculate. As a rule, reading small quantities of data
> with netcdf-4 is always slower than netcdf-3 because the
> underlying HDF5 file format is based on b-trees rather than the
> linear disk layout of netcdf-3. Since vars reads a single
> element at a time, that overhead can, I suspect, be significant.
> I am, however surprised that it is as large as you show.

In this case, no b-trees are involved, because the data storage is
contiguous, not chunked (according to ncdump -h -s).  So I'm
surprised how slow the strided netCDF access is, and suspect there
might be a performance bug in how netCDF-4 uses the HDF5 API for
strided access.

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: ZFB-587742
Department: Support netCDF
Priority: Normal
Status: Closed