This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
David, Sorry to hear that didn't work. We may actually be able to work on this again after Tuesday next week, since that's when our AMS paper is due. In the meantime, if it would be any help at all, I could send the C source that duplicates your example with chunking, so you could verify the speedup. You probably know this already, but just to make sure, writing the 256x256x64 variables in 32x64x64 sections is slow because when the data is stored contiguously, writing each section involves 64x64=4096 calls to a low-level write function, each call writing only 32 values. Also each write is preceded by a seek call to start writing at the right location, and this seek call flushes I/O buffers. It's my unverified theory that the occasional erratic slowdown by two orders of magnitude is due to all these small writes getting queued up until there's no more space to queue more pending writes, so the process gets blocked and swapped out until the pending I/O pipe gets drained, and when the process is swapped in again, all its buffers have to get reestablished. One way to prove something like this is going on would be to write a program that just used low-level I/O (not netCDF) to write data in the same pattern of 4096 writes of 32 4-byte values after a seek, to see if the same erratic slowdowns are observed. I may do this at some point to see if my intuition is correct. --Russ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: NNS-961385 Department: Support netCDF Priority: Normal Status: Closed