This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hi Nico, > We are using NetCDF4 to write our data products. These data products > are variable in size, but typically: > > * Total file sizes ranging from 10GB to several hundreds of GB > * Tens to hundreds of groups, in 3-4 levels > * Hundreds to thousands of variables, ranging in size from several > kB to several GB > * Each variable having several attributes > > The problem we are facing now is the following: Due to chunking, each > variable is assigned a chunking cache. Currently we have found no way to > end the access to a NetCDF variable. Due to the large number of variables > this has the following effects: > > * The total chunking cache size is very large, causing large memory > footprint of our software. > * A lot of data will only be written to the disk when the file is closed > > It is possible to force writing data to disk with nc_sync, but this works > at a file level rather than variable level and can thus result in a huge > performance penalty, as this would also force (re-)writing chunks in other > datasets that are not yet complete. Moreover, it will not release the > chunking cache and does thus not affect the software's memory footprint. > > Closing and reopening the file would probably release all resources, > but this will cause a huge overhead and a lot of additional code to find > and reopen groups and variables again. > > Is it possible with NetCDF4 to explicitly end the access to a variable > (like it is for example possible in HDF5, using H5Dclose()), thus freeing > all resources associated with it and forcing all cached data for the > variable to be written to disk? No, unfortunately there is no such function in the netCDF-4, but you have made a convincing justification of the need for such a function. I've created an issue in our bug tracking system for the addition of such a function, so we don't lose track of it and so you can follow any progress on the issue or comment on it in the future: https://bugtracking.unidata.ucar.edu/browse/NCF-292 Thanks for reporting the problem. --Russ Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: MME-788623 Department: Support netCDF Priority: Normal Status: Closed