[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #SJD-391075]: compression and parallel Netcdf4

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [netCDF #SJD-391075]: compression and parallel Netcdf4
  • Date: Mon, 02 Nov 2015 09:58:36 -0700
Hello Reto,

Our compression/parallel I/O functionality for netCDF4 is provided via the hdf5 
library.  Currently, I don't believe they have any plans for adding that 
functionality to their library, and we (unfortunately) do not have the 
resources to re-implement compression/MPI in a way that they work together.  I 
agree with you, however, that it would be a great feature to offer, and if it 
is something we are able to provide in the future, we most assuredly will.  Or, 
if somebody were to implement it and provide it as a patch via a 'Pull Request' 
on our GitHub page, that would also be graciously accepted :).

I'm sorry I can't provide a more positive response,

-Ward

> Dear NetCDF Support
> 
> On your online documentation
> 
> http://www.unidata.ucar.edu/software/netcdf/workshops/2011/nc4chunking/CompressionAdvice.html
> 
> it is written that compression cannot be used with parallel I/O.
> 
> I've tested the feature and it really does not work :-)
> So, my question goes:
> 
> Is it planned to include per-variable compression also with MPI parallel
> I/O support? It would possibly speed up the define and write of big data
> quite extensively depending on data structure, and it would decrease
> the file size of parallel NetCDF 4 files very much.
> 
> Reto
> 

Ticket Details
===================
Ticket ID: SJD-391075
Department: Support netCDF
Priority: Normal
Status: Closed