Re: 20040407: NetCDF I/O Problem

>To: address@hidden
>From: "Kevin W. Thomas" <address@hidden>
>Subject: NetCDF Problem
>Organization: OU/CAPS
>Keywords: 200404071606.i37G6MCT022491 netCDF I/O performance

Kevin,

> >We would like to help, but we don't have much experience on IBM
> >Regattas.  Here's a few questions about this problem:
> 
> I've seen the same behavior on PSC's "rachel", an SMP machine running Tru64
> Unix.

We don't have any logins at the Pittsburgh Supercomputing Center, and
don't have access to an SMP machine running Tru64 Unix for testing, so
I'm not sure that will be of much help in diagnosing the problem, but
if it's really the same problem, it would mean it's not
platform-specific.

> > - Is it easily reproducible, that is, does it always happen in the
> >   same place for the same inputs or does it seem to depend on other
> >   factors (load on computer, order in which things execute, phase of
> >   the moon, etc.)?
> 
> It seems to happen in the same place in the code.  The slowness seems to occur
> the first time a 1D write occurs.

By 1D write, do you mean you're writing to a 1D netCDF variable, or
you're writing a 1D slice of a multidimensional variable?  The latter
could be slow if you're not writing values that are contiguous on
disk.  For example, with netCDF file structured like:

  netcdf example {
  dimensions:
    time = 1000;
    level = 10;
    lon = 360;
    lat = 180;
  variables:
    float rh(time, level, lon, lat);
  }

if you tried to write a 1D slice of varying times, with level, lat,
and lon indices held constant, it would be slow, because you would be
accessing a different disk block for each value, so would require 1000
disk reads and 1000 disk writes.

> > - Do you know what netCDF function is being called for output when
> >   you see the slowdown?
> 
> The code is in NF_PUT_VARA_REAL.

Then it would be useful to know the shape of the variable you are
writing and the start and count arguments to the NF_PUT_VARA_REAL()
function.

> Is there any possibility of someone looking over the I/O part of our code to
> see if we've done something wrong, or maybe forgot to do something.  We're a
> bit new writing netCDF code.

Just send the relevant code showing the NF_PUT_VARA_REAL() call and
enough additional declarations and code to show what the values of all
the parameters to the call are, as well as an excerpt from an ncdump
of the file you are writing that shows the shape of the variable.

Thanks.

--Russ