This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Dave, > To: address@hidden > From: Dave Pierce <address@hidden> > Subject: netCDF v2.4 memory useage under UNICOS > Organization: UCSD/Scripps Institution of Oceanography > Keywords: 199604181925.AA06307 In the above message you wrote: > I've been trying to figure out why my Cray jobs suddenly increased in size > recently and tracked it down to the new version of the netCDF library. > Opening each netCDF file under the old version (2.3) increased the image > size by about 16K on a DEC alpha and about 200K on a Cray J90; but under > version 2.4 of the netCDF library, each opened file increases the image size > almost 3 megabytes on the Cray! With typically 20+ input or output files > simultaneously open in netCDF format, my jobs have increased in size almost > 60 megabytes! > > I saw in the release notes that a bunch of speed optimizations had been > done for the Cray under this release, but it didn't mention the size > penalty. It seems to me that there needs to be some way for the user to > control this space/time tradeoff; if opening a file only adds 16K, as on > the DEC Alpha, you don't really need it, but at 3 megabytes a pop it is > starting to be an important factor on the Cray. > > Do you have any comments, or suggestions for how I can (presumably) set > the buffer sizes smaller? The size increase you noticed is due to the default allocation of I/O buffers in the FFIO layer. This allocation is user-controllable via the NETCDF_FFIOSPEC environment variable. You should read the section in the user's guide that discusses this. There's also a bunch of correspondence in the netCDF support-email archive on this topic Unfortunately, it's not well organized. More unfortunately is the fact that no one here is sufficiently familiar with CRAY FFIO to offer much advice. If all else fails, I suggest posting a query to the netcdfgroup mailing-list asking for help on your particular situation. You should try and describe the I/O structure of your process as completely as possible. I wish I could be more helpful. -------- Steve Emmerson <address@hidden>