This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Elizabeth: I'm coming into this discussion late, so I'm wondering what the status of this problem is. We received another problem report which may be related. I have attached this. Improved performance on parallel machines is one of the goals of netcdf-3. I would be more than happy to work with you or someone on your staff to enable netcdf-3 to take full advantage of your machines. By working with your experts, it might decrease our support load from separate sites. Our access to CRAY machines is limited to those at NCAR, a t90 at gfdl and a t3e at gfdl. We figure out things as best we can based on this sample, but really have no sense of a recommended "forward compatibility" strategy. A brief outline of the netcdf internal architecture follows. There is an abstract i/o interface used internally which is called "ncio". This interface was designed with multiprocessing in mind. The primitive "read" operation looks like this: int ncio_get(ncio *const nciop, off_t offset, size_t extent, int rflags, void **const vpp); Where 'nciop' is an opaque handle to the lower layer 'offset' is the position in the file where the data begins 'extent' is the amount of data to be read 'rflags' indicates whether this is a read-only operation or read-modify-write operation and 'vpp' is returned containing a pointer to memory which is 'extent' size, containing the data from the file. The returned integer is an error code. This operation can serve to lock a particular region of a file while an the data is examined or modified. Each call to ncio_get() is followed by a call to ncio_rel() when the use of this region is completed: int ncio_rel(ncio *const nciop, off_t offset, int rflags); Where 'nciop' is an opaque handle to the lower layer 'offset' is used to indicate which region to release (this must be the same as used for a previous, unreleased called to ncio_get()) and 'rflags' indicates whether the region was actually modified (ignored if the ncio_get() was read-only). We provide two implementations of the ncio interface with the netcdf package. Once does POSIX read() and write() calls and manages malloc()'ed space. The other uses CRAY ffio in a simple way modeled after Jeff Kuehn's contribution to netcdf-2. There is no reason that operations on distinct regions of a netcdf file could not proceed in parallel. The above _interface_ and it's use by the higher netcdf layer is designed to provide proper barriers. ----> The _implementation_ of ncio we provide does not. <---- It should be straightforward to provide a specialized implementation which would take advantage of parallel architecture. We have done implementation of a similar interface which uses POSIX fcntl() locking to provide safe sharing by separate programs on typical unix. -glenn
--- Begin Message ---
- Subject: Problem reading NetCDF files
- Date: Wed, 18 Jun 1997 13:55:04 -0400 (EDT) address@hidden
Matt- I've narrowed the problem you were having reading netCDF files down to what appears to be an error in the Cray PUTENV function. This function is called to temporarily reset the NETCDF_FFIOSPEC environment variable and reopen the file to access the "time" coordinates efficiently. When these coordinates have been read, the file is closed and NETCDF_FFIOSPEC is reset to its original value, again by calling PUTENV. Ramesh is looking into the problem. In the meantime, I can offer you a temporary get-around in the form of a modified version of "readnc_1.F" in /home/jps/codes/netcdf/v0.7/special/readnc_1.F In this version, I've disabled the environment-variable-resetting. While this will keep you from bombing, it may lead to longer runs times due to the inefficiency of retrieving time coordinate values using larger FFIO cache pages. You may want to temporarily use a larger number of smaller pages (ie, same amount of memory) in your specification of NETCDF_FFIOSPEC, say "cachea:32:16" instead of "cachea:256:2". Let me know how it goes... John
--- End Message ---