[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [address@hidden: netCDF opinion]

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: [address@hidden: netCDF opinion]
  • Date: Wed, 20 Jun 2001 14:23:10 -0600
Hi Bill,

> Can you give me your opinion on something.
> 
> The general logic makeup of my program is something like
> this:
>       1)      Read some data from a netCDF.
>       2)      Interpolate the data to a finer granularity with
>               cubic spline.
>       3)      Check the values for correctness.
>       4)      Write the values to a new netCDF.
> 
> I am currently threading steps 2 and 3 and have seen a fair
> amount of speedup in doing so.
> 
> Now the question:
> I can tell by looking at "top" that the cpu is by far under utilized,
> however when my netCDF process is running it has the IO pegged.
> If I were to thread the putting of data to ( step 4 ) variables would 
> I expect the overall speed of the program to increase?  
> My thinking is that since netCDF appears to be memory mapped I might...
> But on the other hand if I still must go through the same controller 
> maybe it won't.
> 
> I have tried it on a local single CPU system and the result have
> been mixed with neither the single threaded or multi-threaded write
> really showing an advantage and unless there is a compelling reason
> to write the thread ( ie faster in terms of total completion time )
> I'd rather not deal with the additional complexity of have to 
> keep up with mutexes etc.

The standard netCDF I/O library is not thread-safe, so trying to write
the output in parallel from multiple threads may result in data
corruption.  The exception is if you happen to be running on a Cray
T3E, in which case there is a parallel version available.

We submitted a proposal to develop a parallel version that would use
MPI-IO, but that proposal wasn't funded.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu