Re: 20001221: parallel netcdf (fwd)

>To: address@hidden,
>To: address@hidden
>From: Mary Haley <address@hidden>
>Subject: parallel netcdf (fwd)
>Organization: Oregon State University
>Keywords: 200012212358.eBLNw4o13814 netCDF parallel

Hi Eric,

> I'm trying to find a work-around for using netcdf on the SP3 with my 
> MPI-based 
> model. I rely on netcdf for most of the i/o and write out data via a big 
> temporary array that is the size of one variable for the total model domain.  
> The problem is I'm running out of room on individual processors as my array 
> size 
> gets bigger (I'm assuming this is because the memory is shared with other 
> processes) so that my program won't load.  I'd like to abandon the use of the 
> temporary array, but don't know how that can be done without some sort of 
> parallel netcdf or at least simultaneous direct access to my output file.  Do 
> you have any suggestions?

Unfortunately, netCDF does not support the feature of multiple
concurrent writers because of its implementation that uses user-space
buffering for better performance.  Calls to nc_sync() before and after
each write might work at the expense of slowing things down due to
flushing the buffers, but I think there would still be race conditions
that would make this unreliable, depending on your pattern of writes.

We applied for resources to develop a parallel netCDF last year:

  http://www.unidata.ucar.edu/packages/netcdf/itr/

but the proposal was rejected and we still haven't located resources
to accomplish this work.  (We had intended to exploit the fact that HDF5
uses MPI-I/O for efficient parallel I/O on multiprocessor platforms.)

The only thing I can suggest is to serialize the I/O by having a
separate server process that has the netCDF file open and accepts
requests from the individual processors to read and write the data,
but that might require a redesign and have unacceptable performance
implications.  Another possibility would be to switch to using HDF5
directly and take advantage of its MPI-I/O implementation:

 http://hdf.ncsa.uiuc.edu/HDF5/

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu