This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hi Lee, Thanks very much for sending your netCDF performance results and for sending the source of the program you developed to do the timings. This looks useful and we may use if for testing netCDF-4 timings to compare with netCDF-3. Thanks also for your congratulations. The speed of netCDF is in large part a credit to the author of the C library, Glenn Davis, who died trgically in an airplane crash eight years ago. We;ve made a few additional optimizations since then and are happy not to have made things any slower. For example the latest release, netCDF version 3.6.2, improved performance of byte-swapping loops on little-endian platforms by unrolling loops. Some benchmarks show a 20% speedup for reading arrays of floats or ints, and smaller speedups for shorts or doubles, but such speedups may not be seen with compilers that already do loop unrolling as an optimization. > 1. Can you explain why NetCDF would be faster? The core netCDF C library uses a fairly simple buffering scheme that takes advantage of disk caching. The format was designed to support direct access to data, minimizing the number of disk seeks needed to access specified slices of multidimensional arrays, but it still works well for sequential reads and writes. Any advantage it has over Fortran binary writes in that case may be due to the 8Kb buffer size, if that's larger than Fortran uses. It's interesting that the time required for byte-swapping to a portable representation is usually not noticeable, but lately we have had a few complaints about this, reflecting systems that optimize I/O so well that byte-swapping time dominates CPU usage for writing large netCDF files. That problem will go away with netCDF-4, which uses a "reader makes right" strategy supported by the underlying HDF5 library and avoids byte swapping when data is written and read on platforms that have the same byte gender (endianness). > 2. I plan to publish a short note on my results in the ARSC newsletter. Do > you wish to comment before publication? I don't have any particular comments, though you may use anything you like from this email. I would appreciate a URL to the newsletter article when it's published, because it seems like useful information for our community and we might want to link to it from our web site. Thanks again for your efforts and letting us know about the results. --Russ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: NOJ-562667 Department: Support netCDF Priority: Normal Status: Closed