Re: 20010410: NetCDF: poor performance writing to an NFS volume

>To: "Russ Rew" <address@hidden>
>From: "Mark Hadfield" <address@hidden>
>Subject: Re: 20010410: NetCDF: poor performance writing to an NFS volume
>Organization: NIWA
>Keywords: 200104102239.f3AMddL27187

Hi Mark,

> > Could you make your netCDF file or CDL file available to me to
> > investigate this further?  Putting it on an FTP or HTTP server
> > temporarily until I can copy it would be ideal.
> 
> Here it is:
> 
> http://katipo.niwa.cri.nz/~hadfield/gust/nctest/test.nc

Thanks, I got it.

I was able to run your tests, but couldn't see much of a slowdown
writing to an NFS volume, so the cause of the slowdown you are seeing
remains a mystery to me.

Specifically, here's what I get:

 test$ ncdump test.nc > test.cdl
 test$ ls -l test.cdl
 -rw-r--r--   1 russ     ustaff   3938281 Apr 17 09:47 test.cdl

This is the same size as your CDL file.

 test$ time ncgen test.cdl

 real   0m4.06s
 user   0m4.00s
 sys    0m0.05s

This just checks the time to parse the CDL file, your TEST 4, and your
Compaq Alpha is about 2.5 times as fast as my Sun Ultra2.

 test$ time ncgen -o /buddy/russ/test.nc -b test.cdl # local

 real   0m7.62s
 user   0m4.13s
 sys    0m0.44s

That was your TEST 5, having ncgen write test.nc to a local disk, your
Alpha is about 3.4 times as fast.

 test$ time ncgen -o /scratch/russ/testx.nc -b test.cdl # NFS volume

 real   0m8.32s
 user   0m4.29s
 sys    0m0.39s

Equivalent to your TEST 6, where you get 27.9s, about 3.4 times
slower.

If I just copy the file with cp to an NFS disk versus a local disk, it
takes about twice as long:

 test$ time cp /buddy/russ/test.nc /buddy/russ/test-cp.nc # local to local

 real   0m0.21s
 user   0m0.01s
 sys    0m0.03s

 test$ time cp /buddy/russ/test.nc /scratch/russ/test-cp.nc # local to remote

 real   0m0.43s
 user   0m0.00s
 sys    0m0.02s

Note that ncgen still takes about 20 times as long to parse the CDL
and write a file on an NFS volume using netCDF calls as just copying
the bytes with "cp".  That's because every value must be converted to
XDR, the machine-independent external representation, before it is
written to disk.

I've run all these tests 3 times, with very consistent times.  I've
checked with "df" to make sure the local disk was local and the remote
disk was NFS mounted.  The times I saw are about what I would expect,
which is only a minor slowdown due to NFS and nothing like what you
are seeing.

> Why do you think TEST 6 takes much longer than TEST 3? When ncgen writes
> fill values to a file does it actually write those values, in the same way
> it writes real data?

 test$ time ncgen -o /scratch/russ/test2.nc -b test2.cdl

 real   0m1.32s
 user   0m0.08s
 sys    0m0.04s

This is equivalent to your TEST3, and the times I get for this are not
as consistent, varying from 1.32s to as much as 2.56s.  But the ratio
between TEST6 and TEST3 times (3 to 6) is significantly less than you
are seeing (11.4).  I think much of this time has little to do with
NFS, but is explained by the time it takes to parse 3.2 Mbytes of CDL
versus 12849 bytes of CDL for the case with all the fill values.

The bottom line is that I still can't explain or duplicate the
problems you are seeing with netCDF and NFS writes.  And the fact
that you could tune the I/O to perform reasonably by using nc__create
and a *smaller* chunk size seems counter-intuitive.  

To see if the problem was platform-dependent (Alphas are little-endian
and Suns SPARCS are big-endian), I also tried your timing tests on our
old DEC Alpha.  I've appended the times, which show just parsing the
big CDL file to be *much* slower than on your system.  Maybe the
character I/O on this system is just horrendously slow.  The ncgen was
compiled with -O from the 3.5.0 sources.

--Russ

 VERIFY OS version:

 test$ uname -a
 OSF1 ernie.unidata.ucar.edu V4.0 1091 alpha

 TEST 1:

 test$ time ncgen test2.cdl

 real    0m0.13s
 user    0m0.08s
 sys     0m0.05s

 TEST 2:

 test$ time ncgen -o /tmp/test2.nc -b test2.cdl # local

 real    0m0.90s
 user    0m0.55s
 sys     0m0.20s

 TEST 3:

 test$ time ncgen -o /scratch/russ/test2.nc -b test2.cdl # NFS volume

 real    0m3.40s
 user    0m0.53s
 sys     0m0.30s

 TEST 4:

 test$ time ncgen test.cdl 

 real    0m27.98s
 user    0m27.71s
 sys     0m0.13s

 TEST 5:

 test$ time ncgen -o /tmp/test.nc -b test.cdl # local       

 real    0m36.98s
 user    0m28.56s
 sys     0m2.21s

 TEST 6:

 test$ time ncgen -o /scratch/russ/test.nc -b test.cdl # NFS volume

 real    0m57.56s
 user    0m28.71s
 sys     0m3.15s