[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Support #CUV-251255]: Nccopy extremly slow / hangs

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [Support #CUV-251255]: Nccopy extremly slow / hangs
  • Date: Tue, 21 Jun 2011 11:21:11 -0600
Mark,

> Thanks for the reply. I attach here a copy of ncdump -c foo.nc.
> 
> I've been playing some more, and have been able to make some progress by 
> working in stages:
> 
> nccopy -u -k3 -d0 foo.nc intermediate.nc
> nccopy -c time/1,longitude/6,latitude/7 intermediate.nc intermediate2.nc
> nccopy -c time/1698,longitude/6,latitude/7 intermediate2.nc intermediate3.nc

In looking into your problem, I discovered an embarrassing bug in
nccopy.  For netCDF-4 input files, it was ignoring the -d option.  This
is now fixed in the snapshot, but unfortunately didn't make the 4.1.3
release last Friday.  The fix is fairly simple, just deleting a couple of
lines in ncdump/nccopy.c, which you can get from 

  http://svn.unidata.ucar.edu/repos/netcdf/trunk/ncdump/nccopy.c

I don't think this fix helps your problem, but it may clear up questions
about why nccopy doesn't seem to be working as documented on netCDF-4
input files when parameters are specified to change the compression or
chunking in the output copy.

I'm now digging into how to solve your problem, and I'm pretty sure it will
involve providing better variable chunk cache parameters for both the input
and output variable.  This is a very good example for seeing how using the
default chunk cache settings doesn't perform very well on this kind of large
multi-dimensional transpose operation.  I'll let you know when I have some
progress to report ...

--Russ
 
> Its the last line that it stalls on. I have been able to get nccopy to run to 
> completion using
> 
> nccopy -c time/2 intermediate2.nc intermediate3.nc
> 
> but time/1698 fails - and even takes the rest of the machine with it 
> sometimes! :-(

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: CUV-251255
Department: Support netCDF
Priority: Normal
Status: Closed