[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #KVY-259447]: Memory error writing chunks of large variable in

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #KVY-259447]: Memory error writing chunks of large variable in
  • Date: Thu, 29 Jan 2009 15:09:25 -0700
Karen,

Oops, I didn't wait long enough after testing the "workaround"
provided in the last reply.  It turns out the code I suggested still
eventually got an HDF5 error when the variable size exceeded 2**31
bytes on a Sun platform, due to a known HDF5 bug on Suns when writing
large files.  We're waiting on a fix for that from the HDF5 Group.  I
don't think you said what platform you were testing on, but if it was a
Solaris/Sun, then what you saw was just a symptom of that large file
bug in HDF5.

So then I tried your original program, without specifying chunk
shapes, on a Linux x86 platform, and it ran to completion without
getting the HDF5 error you are seeing:

  $ cc -g -Dnetcdf4 -I/shecky/russ/nc4/include -on4err n4err.c
  -L/shecky/russ/nc4/lib -lnetcdf -L/upc/share/ed/local/shecky/lib -lm
  -lhdf5_hl -lhdf5 -lz
  ./n4err -d . -r 10
  Using netcdf4
  Time to define: 1
  number of chunks 16
  Time step0
  Time to write step: 270
  Time to write data: 274
  Time to close: 0
  *** SUCCESS writing example file ./r10.nc!

Then just to see if using smaller chunks would make it faster, I
inserted the suggested modification specifying the chunk shapes as
1 x 100 x 655360, with the following result:

  ./n4err -d . -r 10
  Using netcdf4
  Time to define: 0
  number of chunks 16
  Time step0
  Time to write step: 88
  Time to write data: 91
  Time to close: 0
  *** SUCCESS writing example file ./r10.nc!

So it works on the Linux system even with the single large chunk,
but it's about 3 times faster to write if you explicitly set
the chunks to match your access pattern.

I used the 4.0.1-beta2 release of netCDF and the 1.8.2 release of
HDF5, but I don't think the results would differ using the 4.0
version of netCDF or the 1.8.1 version of HDF5.

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: KVY-259447
Department: Support netCDF
Priority: Emergency
Status: Closed