[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[netCDF #AMR-714212]: netcdf file size for limited vs unlimited
- Subject: [netCDF #AMR-714212]: netcdf file size for limited vs unlimited
- Date: Wed, 24 Mar 2010 13:32:43 -0600
Oscar,
I wrote:
> I'll post a table I'm constructing of the space used by all the cases you
> are considering, and I think it will be clear from that that netcdf-4
> would not be suitable to save space in this case, unless you used
> compression.
Consider a case of two short variables, each with 2000 values. Here's
the size needed for various techniques of storing this as netCDF data.
Size of
netCDF file Technique Format
8116 bytes Fixed dimension, size 2000 netCDF classic
16116 bytes Unlimited dimension, size 2000 netCDF classic
8136 bytes 2-d unlimited variables, 1000x2 netCDF classic
13364 bytes Fixed dimension, size 2000 netCDF-4 format
164470 bytes Unlimited dimension, size 2000 netCDF-4 format
113647 bytes 2-d unlimited variables, 1000x2 netCDF-4 format
You can try similar experiments pretty quickly using the ncgen utility
that comes with netCDF-4, which can generate netCDF classic or
netCDF-4 files fairly easily from CDL text files. ncgen can also
generate C code to create the files, which would be suitable for
inserting a calls to use compression on each of the variables to see
how much space that would save.
If you use compression, there is a spece/time tradeoff, in that more
time is required to store or access the compressed data, but this can
be tuned for performance by adjusting "chunking" parameters.
--Russ
Russ Rew UCAR Unidata Program
address@hidden http://www.unidata.ucar.edu
Ticket Details
===================
Ticket ID: AMR-714212
Department: Support netCDF
Priority: Normal
Status: Closed