[netCDF #MME-788623]: Ending access to a NetCDF4 variable

Hi Nico,

> We are using NetCDF4 to write our data products. These data products
> are variable in size, but typically:
> 
> * Total file sizes ranging from 10GB to several hundreds of GB
> * Tens to hundreds of groups, in 3-4 levels
> * Hundreds to thousands of variables, ranging in size from several
> kB to several GB
> * Each variable having several attributes
> 
> The problem we are facing now is the following: Due to chunking, each
> variable is assigned a chunking cache. Currently we have found no way to
> end the access to a NetCDF variable. Due to the large number of variables
> this has the following effects:
> 
> * The total chunking cache size is very large, causing large memory
> footprint of our software.
> * A lot of data will only be written to the disk when the file is closed
> 
> It is possible to force writing data to disk with nc_sync, but this works
> at a file level rather than variable level and can thus result in a huge
> performance penalty, as this would also force (re-)writing chunks in other
> datasets that are not yet complete. Moreover, it will not release the
> chunking cache and does thus not affect the software's memory footprint.
> 
> Closing and reopening the file would probably release all resources,
> but this will cause a huge overhead and a lot of additional code to find
> and reopen groups and variables again.
> 
> Is it possible with NetCDF4 to explicitly end the access to a variable
> (like it is for example possible in HDF5, using H5Dclose()), thus freeing
> all resources associated with it and forcing all cached data for the
> variable to be written to disk?

No, unfortunately there is no such function in the netCDF-4, but you have 
made a convincing justification of the need for such a function.

I've created an issue in our bug tracking system for the addition of such 
a function, so we don't lose track of it and so you can follow any progress
on the issue or comment on it in the future:

https://bugtracking.unidata.ucar.edu/browse/NCF-292

Thanks for reporting the problem.

--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: MME-788623
Department: Support netCDF
Priority: Normal
Status: Closed