[netCDF #NOJ-562667]: NetCDF Performance

Hi Lee,

Thanks very much for sending your netCDF performance results and for sending 
the source of the program you developed to do the timings.  This looks useful 
and we may use if for testing netCDF-4 timings to compare with netCDF-3.  
Thanks also for your congratulations.  The speed of netCDF is in large part a 
credit to the author of the C library, Glenn Davis, who died trgically in an 
airplane crash eight years ago.  We;ve made a few additional optimizations 
since then and are happy not to have made things any slower.  For example the 
latest release, netCDF version 3.6.2, improved performance of byte-swapping 
loops on little-endian platforms by unrolling loops. Some benchmarks show a 20% 
speedup for reading arrays of floats or ints, and smaller speedups for shorts 
or doubles, but such speedups may not be seen with compilers that already do 
loop unrolling as an optimization.

> 1. Can you explain why NetCDF would be faster?

The core netCDF C library uses a fairly simple buffering scheme that takes 
advantage of disk caching.  The format was designed to support direct access to 
data, minimizing the number of disk seeks needed to access specified slices of 
multidimensional arrays, but it still works well for sequential reads and 
writes.  Any advantage it has over Fortran binary writes in that case may be 
due to the 8Kb buffer size, if that's larger than Fortran uses.  It's 
interesting that the time required for byte-swapping to a portable 
representation is usually not noticeable, but lately we have had a few 
complaints about this, reflecting systems that optimize I/O so well that 
byte-swapping time dominates CPU usage for writing large netCDF files.  That 
problem will go away with netCDF-4, which uses a "reader makes right" strategy 
supported by the underlying HDF5 library and avoids byte swapping when data is 
written and read on platforms that have the same byte gender (endianness).

> 2. I plan to publish a short note on my results in the ARSC newsletter. Do 
> you wish to comment before publication?

I don't have any particular comments, though you may use anything you like from 
this email.  I would appreciate a URL to the newsletter article when it's 
published, because it seems like useful information for our community and we 
might want to link to it from our web site.

Thanks again for your efforts and letting us know about the results.

--Russ



Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: NOJ-562667
Department: Support netCDF
Priority: Normal
Status: Closed