This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
> >From: Olaf Heudecker <address@hidden> > >Subject: Buffersizes > >Organization: Alfred-Wegener-Institute for Polar and Marine Research > >Keywords: 199711120900.CAA08843 netCDF 3.3 Cray > > Dear netCDF team > > I have successfully installed netcdf-3.3.1 on the local Cray T3E and > J90. Now there are several users asking about the memory usage of > netCDF. In a series of tests to investigate the I/O performance > of the T3E we found that netCDF 3.3.1 seems to use a 4MB internal > asynchronous library buffer (besides the bufa:336:2 assign default). > > Could you please tell me whether there is a way to tune this library > buffer size (for specific useres) and how to do it? Do you know the > default sizes on the different architectures? > > Thanks a lot > Olaf R. Heudecker > Alfred-Wegener-Institute for Polar and Marine Research For netcdf I/O on the CRAY, we can identify a couple of levels of buffering. There is a "contract" between the higher layers of netcdf and the "ncio" layer which says "we won't request more than 'chunksize' bytes per request. This controls the maximum size of the buffer used internal to the ncio implementation for system read and write. On the Cray, the ncio implementation is usually 'ffio.c', which uses the cray specific ffio library. This is in contrast to the implementation used on most other systems, 'posixio.c', which used POSIX read(), write(), lseek() calls. The ffio library has all sorts of controls (which I don't pretend to understand), including the bufa directive you cite above. If you are using some sort of assign statement external to netcdf, it is probably being overridden by the following code sequence from ffio.c: ControlString = getenv("NETCDF_FFIOSPEC"); if(ControlString == NULL) { ControlString="bufa:336:2"; } fd = ffopens(path, oflags, 0, 0, &stat, ControlString); EG, if the NETCDF_FFIOSPEC environment variable is not set, use the default shown. Otherwise, use the environment variable. Currently for ffio, the "chunksize" contract is set to the st_oblksize member of struct ffc_stat_s obtained from a fffcntl(). You can check this by putting a breakpoint in ffio.c:blksize(). As you may be able to tell from ffio.c, we have stubbed out interlayer communication to allow this to be set as the *sizehintp argument to ncio_open() and ncio_create(). In netcdf-3.4, we intend to fully implement this and expose this parameter to the user. This is almost complete. If you are interested in early access to this, let me know. -glenn