This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
>Organization: BMRC >Keywords: 199506020618.AA12340 netCDF ncapt Hi Asri, > Thank you for you suggestion. We ran our model in a higher memory allocation > queue and the job was succesful. > We were using a 15 megaword queue on our Cray YMP and the next step > available was to move to a 28 megaword queue which made the elapse time for > the job an order of magnitude longer. Do you know if changing the netCDF > file structure can reduce memory requirements? > Currently we have substantial amount of data defined as global > attributes. Will dividing them by making them attributes of several > different variable make any difference? I did not appreciate from your original question that the problem with running out of memory was occurring on a Cray. This makes a difference, because the netCDF library is written in C, the Fortran interface is just a thin layer on top of the C interface that makes the calls follow Fortran conventions, and the Cray Fortran interface differs from that for other platforms. When the Fortran interface for the Cray was developed, the Cray did not have any Fortran data types such as BYTE, INTEGER*1, or INTEGER*2 corresponding to the netCDF data types NCBYTE and NCSHORT. Consequently, the Fortran interface allocates extra space in memory to just copy arrays of bytes and shorts from the C interface to Fortran integer arrays. The way it does this is rather crude, just calling malloc() to allocate all the space needed for each I/O operation, rather than doing the packing and unpacking in reasonably-sized chunks. The Fortran interface also copies strings to privately-allocated space, because it has to append a trailing "\0" to such strings for the C interface to be able to deal with them. Thus when you do an ncapt() call with a large string, memory is allocated and a copy is made just to append the NULL character at the end. If this is the cause of the memory problems you are seeing, it would certainly be possible to improve the strategies used in the Fortran interface to make them smarter about using a local statically allocated array and doing the conversion in chunks instead of all at once. Two other alternatives are to use the C interface to avoid the extra data copies, or to see if the Cray Fortran compiler now supports something equivalent to the BYTE, INTEGER*1, or INTEGER*2 declarations that are nonstandard but commonly provided in other compilers. If Cray Fortran now supports such facilities, the memory allocation and copying could be completely eliminated for the Fortran interface for the Cray with a few changes to the netcdf/fortran/jackets.src file in the source distribution. ______________________________________________________________________________ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu