[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #AIQ-275071]: [netcdf-hdf] Unexpected overall file size jump

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #AIQ-275071]: [netcdf-hdf] Unexpected overall file size jump
  • Date: Mon, 10 Jan 2011 10:03:34 -0700
James,

By the way, it turns out that using compression even with the default chunking
works as well as compression with optimized chunking on your example, so you 
can get the same results of smaller files without even bothering with
customizing the chunk sizes.

Here's the specifics.  With your original netCDF classic file, converting to 
a netCDF-4 or netCDF-4 classic model file turns on default chunking for all 
the variables that use an unlimited dimension, and the resulting chunk sizes 
waste lots of space with some chunks that are mostly empty (missing values):

  $ ls -l Fluid_Meas.snc
  -rw-rw-r-- 1 russ ustaff 42186296 Jan 10 09:52 Fluid_Meas.snc
  $ nccopy -k4 Fluid_Meas.snc tmp.nc; ls -l tmp.nc
  -rw-rw-r-- 1 russ ustaff 95528413 Jan 10 09:53 tmp.nc

But just compressing with deflation level 1 and shuffling results
in a smaller file than the original, even using the default chunk sizes:

  $ nccopy -s -d1 -k4 Fluid_Meas.snc tmp.nc; ls -l tmp.nc
  -rw-rw-r-- 1 russ ustaff 27864220 Jan 10 09:53 tmp.nc

You can do as well with smaller custom chunk sizes, but it may not be 
worth the trouble:

  $ nccopy -s -d1 -k4 -c npoints/76375 Fluid_Meas.snc tmp.nc; ls -l tmp.nc
  -rw-rw-r-- 1 russ ustaff 27864220 Jan 10 09:53 tmp.nc

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: AIQ-275071
Department: Support netCDF
Priority: Normal
Status: Closed