Re: 200000806: Compressing netcdf files

>From: Glenn Carver <address@hidden>
>Subject: Compressing netcdf files
>Organization: Centre for Atmospheric Science, Cambridge University, Chemistry 
>Dept.,
>Keywords: 200008062032.e76KWJT17722 netCDF compression bzip2

Hi Glenn,

> I thought I would write highlighting the excellent compression of netcdf
> files offered by the new compression tool 'bzip2'. bzip2 is a high quality
> data compressor which is freely available. The website is :
> "http://sourceware.cygnus.com/bzip2/";
>
> As an example with some of my netcdf files 200Mb uncompressed, bzip2
> squashes these down to 20Mb whereas gzip can only manage 160Mb (both on
> default settings).

Thanks for the information.  I just tried bzip2 on some netCDF model
output files, and while I didn't get as impressive an improvement over
gzip as you saw, the improvement is still quite significant.  Here's
the compression factors I got (uncompressed_size/compressed_size)
using the three compression utilities compress, gzip, and bzip2:

 size  compress  gzip  bzip2
 31Mb   2.54     4.13  6.04
  4Mb   3.63     4.47  7.82
 16Mb   3.58     4.32  7.36

Maybe the 10 to 1 compression factors you are seeing are due to some
repeated values or smaller ranges of values for some variables, or
perhaps bzip2 just performs better on larger files.

> I hope this is of use to people on the netcdf mailing list.

I could forward your note to the netcdfgroup mailing list with your
permission, or just write something myself from what you sent.  I
assume since you didn't post to netcdfgroup that you'd rather we let
netCDF users know about bzip2, which I'm happy to do.  Thanks again
for the information.

--Russ