[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #AMR-714212]: netcdf file size for limited vs unlimited

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #AMR-714212]: netcdf file size for limited vs unlimited
  • Date: Wed, 24 Mar 2010 13:32:43 -0600
Oscar,

I wrote:
> I'll post a table I'm constructing of the space used by all the cases you
> are considering, and I think it will be clear from that that netcdf-4
> would not be suitable to save space in this case, unless you used
> compression.

Consider a case of two short variables, each with 2000 values.  Here's
the size needed for various techniques of storing this as netCDF data.

 Size of 
 netCDF file      Technique                        Format   

  8116 bytes      Fixed dimension, size 2000       netCDF classic
 16116 bytes      Unlimited dimension, size 2000   netCDF classic
  8136 bytes      2-d unlimited variables, 1000x2  netCDF classic
 13364 bytes      Fixed dimension, size 2000       netCDF-4 format
164470 bytes      Unlimited dimension, size 2000   netCDF-4 format
113647 bytes      2-d unlimited variables, 1000x2  netCDF-4 format

You can try similar experiments pretty quickly using the ncgen utility
that comes with netCDF-4, which can generate netCDF classic or
netCDF-4 files fairly easily from CDL text files.  ncgen can also
generate C code to create the files, which would be suitable for
inserting a calls to use compression on each of the variables to see
how much space that would save.  

If you use compression, there is a spece/time tradeoff, in that more
time is required to store or access the compressed data, but this can
be tuned for performance by adjusting "chunking" parameters.

--Russ


Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: AMR-714212
Department: Support netCDF
Priority: Normal
Status: Closed