[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #VEG-361627]: possible nccopy bug, causes files size to explode

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #VEG-361627]: possible nccopy bug, causes files size to explode
  • Date: Fri, 30 Sep 2016 15:01:11 -0600
Hello Donald,

To answer your final question first, yes, this is potentially expected 
behavior; the underlying file I/O is handled by libhdf5, and the files being 
written are larger than those written natively by netCDF3.

There are several things that can cause netCDF4 files to be much larger when 
converted from netCDF3.  Fill values and chunks sizes are the first things that 
come to mind; depending on the default chunk size and the amount of data in the 
file, the size of the final file may be inflated when the fill value is written 
out to populate any under-filled chunk.  As you've discovered, unlimited 
dimensions can also be a source of unexpected increase in file size.

If you can provide a sample file I'd be happy to look and see if there's a way 
that it can be made smaller using nccopy; otherwise, ncks may be a better 
option.  I'm not certain what ncks is doing, but I will take a look to see if 
it is doing something that can be integrated into nccopy.

Thanks!

-Ward




> If I use nccopy (nccopy -k netCDF-4 or nccopy -4 or nccopy -7)  to convert
> a file from version 3 to version 4, that file increases in size by a factor
> of 5 in one case and a factor of 20 in another. If I use ncks -7, the file
> size is within 6% of the original. If I run  nccopy with the -u flag to
> remove unlimited dimensions, it is also within a few percent.
> 
> Can you explain this? Is this expected behavior?
> 
> --
> Donald Stark
> NCAR RAL JNT/DTC
> Address: FL3-1042
> Phone: 303.497.2472
> Email: address@hidden
> Web: www.rap.ucar.edu/~stark
> 
> 


Ticket Details
===================
Ticket ID: VEG-361627
Department: Support netCDF
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.