[netCDF #ACI-624328]: Experiences from rechunking with nccopy

Hi Joe,

Thanks for the multi-stage rechunking example.  I've downloaded the
example file you provided and will study it as I get time.

--Russ

> Hi Russ,
> 
> Thank you for your answer!
> 
> If it works like you say, that each input chunk is read just once and spread 
> out over the necessary output chunks, shouldn't there be an almost insatiable 
> need for chunk cache in the case where I'm rechunking from e.g. 1x360x720 to 
> 36890x1x1 (in other words changing from a map oriented layout to a time 
> series oriented one, each map is read in and spread out over every single 
> output chunk).
>
> Here's an example where rechunking in steps helps (I've made the example 
> original file available below):
> 
> # First as one step:
> [joe@rita temp]$ time nccopy -c 'time/365,lon/1,lat/1' -m 1G -h 1G 
> Tair_daily_WFD_TYX_format_detrended_1901.nc  Tair.nc
> real  102m14.108s
> 
> 
> # Then several:
> [joe@rita temp]$ time nccopy -c 'time/60,lon/60,lat/60' -m 1G -h 1G 
> Tair_daily_WFD_TYX_format_detrended_1901.nc  Tair_60_60_60.nc
> real  0m17.023s
> 
> [joe@rita temp]$ time nccopy -c 'time/183,lon/5,lat/5' -m 1G -h 1G 
> Tair_60_60_60.nc  Tair_183_5_5.nc
> real  0m44.065s
> 
> [joe@rita temp]$ time nccopy -c 'time/365,lon/1,lat/1' -m 1G -h 1G 
> Tair_183_5_5.nc Tair_365_1_1.nc
> real  0m44.717s
> 
> 
> The example file is available here if you wish to try it yourself (73 MB):
> 
> http://stormbringer.nateko.lu.se/~joe/Tair_daily_WFD_TYX_format_detrended_1901.nc
> 
> Note that it's compressed, I think a lot of the time in the first step above 
> is spent just compressing/decompressing (there's almost 100% CPU utilization).
> 
> Thanks and best regards,
> - Joe
> 
> ________________________________________
> From: Unidata netCDF Support [address@hidden]
> Sent: 08 April 2014 15:51
> To: Joe Siltberg
> Cc: address@hidden
> Subject: [netCDF #ACI-624328]: Experiences from rechunking with nccopy
> 
> Hi Joe,
> 
> > Thank you for that detailed answer!
> 
> No problem.  I'm at a conference all this week, so can't provide much detail 
> until next week.
> However, I'm very interested in chunking issues, and would like to get back 
> to finding a
> good solution for your use case.
> 
> > So the chunk cache is only used to cache output chunks? Then it makes sense 
> > that it doesn't need much.
> 
> Right, nccopy reads the input one chunk at (if it's chunked), and writes to 
> the output,
> spreading the values among as many output chunks as needed.  So if there's 
> insufficient
> cache for holding all the output chunks that have values from one input 
> chunk, it will
> have to write the same chunk multiple times, for each chunk that gets kicked 
> out of
> the chunk cache to make room for other chunks.
> 
> > So in the example file you looked at, rechunked from 1x360x720 to 60x60x60, 
> > nccopy will read in the first 60 time slices in order to create the first 
> > output chunk, and then restart and read them again (hopefully from OS disk 
> > cache next time) to create the next output chunk?
> 
> No, I think it works as described above, only reading each input chunk once.  
> It might work
> faster in some cases to read input values multiple times in order to write 
> output chunks
> fewer times, but nccopy isn't smart enough to know how to do that yet.
> 
> > My real data files are 100 times larger than the one I sent you (~7GB 
> > compressed, ~70GB decompressed). I can easily rechunk them to fairly square 
> > chunks, takes 20 minutes or so, but with very large chunk sizes on the time 
> > dimension (say 4000x1x1) it takes hours. Currently my strategy has been to 
> > do it in several steps, which seems to help. Is that a reasonable approach 
> > for large files or am I doing something wrong if it's that slow?
> 
> I'm very interested in knowing about an example that works faster by 
> rechunking in several
> nccopy steps.  I've suspected that might sometimes be a better strategy, but 
> haven't figured
> out a good example that demonstrates it.
> 
> --Russ
> 
> > I've been rechunking compressed files, and it seems to be CPU bound, if 
> > there's no input chunk cache (except the OS disk cache) that makes sense 
> > since it would need to decompress the same chunks over and over.
> >
> > Thanks and have a nice weekend!
> >
> > - Joe
> > ________________________________________
> > From: Unidata netCDF Support [address@hidden]
> > Sent: 03 April 2014 19:25
> > To: Joe Siltberg
> > Cc: address@hidden
> > Subject: [netCDF #ACI-624328]: Experiences from rechunking with nccopy
> >
> > Hi Joe,
> >
> > > I've been doing some rechunking of large files with nccopy, and have
> > > encountered two potential issues.
> > >
> > > The first is that the -h switch seems to work only up to around 1.7
> > > GB. Specifying -h 2G or 5G doesn't seem to affect memory usage of
> > > the process, or its performance. This was tested with 4.3.1.1 on 64
> > > bit CentOS, and also with your prebuilt 64 bit Windows binaries (also
> > > 4.3.1.1).
> >
> > What you're seeing is that the performance isn't limited by the chunk
> > cache size, it's limited by the actual I/O.  Although you requested 5
> > GBytes of chunk cache, the library doesn't malloc that much, but only
> > mallocs what it needs up to that amount.  So it doesn't need more than
> > 1.7 GBytes for the output file chunk cache in your case.
> >
> > Here's a demonstration you can see from the example file you sent,
> > where you want to rechunk the variable Tair:
> >
> > dimensions:
> > lon = 720 ;
> > lat = 360 ;
> > time = UNLIMITED ; // (365 currently)
> > ...
> > double Tair(time, lat, lon) ;
> >
> > from 365 x 360 x 720 to 60 x 60 x 60.  I renamed your input file
> > "htest.nc" and time each of the following commands after purging the
> > OS disk buffers, so the data is actually read from the disk rather
> > than from OS cache.  If I run (and time)
> >
> > $ ./purge; time nccopy -c 'time/60,lat/60,lon/60' -m 5g -h 5g htest.nc 
> > tmp.nc
> > real  0m28.33s
> >
> > But I get essentially the same time if I only reserve enough chunk
> > cache for one output chunk, 60*60*60*8 bytes:
> >
> > $ ./purge; time nccopy -c 'time/60,lat/60,lon/60' -m 5g -h 1.728m htest.nc 
> > tmp.nc
> > real  0m27.89s
> >
> > And I don't actually need to reserve any memory at all for the input
> > buffer, because all it needs is enough for one chunk of the input, and
> > it allocates that much by default:
> >
> > $ ./purge; time nccopy -c 'time/60,lat/60,lon/60' -h 1.728m htest.nc tmp.nc
> > real  0m24.86s
> >
> > But if I don't have enough chunk cache to hold at least one chunk of
> > the output file, it takes *much* longer:
> >
> > $ ./purge; time nccopy -c 'time/60,lat/60,lon/60' -h 1.727m htest.nc tmp.nc
> > real  16m20.36s
> >
> > Your results will vary with more variables and more times.  For
> > example, with 2 variables of the size of Tair, you need to hold two
> > chunks of output (about 3.5m), but in that case it turns out to be
> > good to specify a larger input file buffer:
> >
> > $./purge; time nccopy -c 'time/60,lat/60,lon/60' -m 100m -h 3.5m htest2.nc 
> > tmp.nc
> > real  2m49.04s
> >
> > Incidentally, I get better performance on the small example by using
> > the "-w" option to use a diskless write and keep the output in memory
> > until the output is closed, not using -h at all:
> >
> > $ ./purge; time nccopy -c 'time/60,lat/60,lon/60' -m 100m -w htest2.nc 
> > tmp.nc
> > real  2m13.14s
> >
> > --Russ
> >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: ACI-624328
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
> 
> Russ Rew                                         UCAR Unidata Program
> address@hidden                      http://www.unidata.ucar.edu
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: ACI-624328
> Department: Support netCDF
> Priority: Normal
> Status: Closed
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: ACI-624328
Department: Support netCDF
Priority: High
Status: Closed