[netCDF #OHL-714122]: NetCDF external links

Hello Eugen,

First, I want to apologize for the delayed response!

The work you have done harnessing the VDS functionality is very interesting.  
We don't currently have any concrete plans to implement the new features from 
the HDF5 library (1.10.x+) until we have completed a fix for the 
backwards-binary-file-compatibility issue it introduced; once we have nailed 
that down in a full release, we will investigate the new features (such as the 
VDS) functionality supported by HDF5 1.10.x+.   

I wonder if you'd be interested in announcing and discussing this on the netCDF 
mailing list (information here: 
http://www.unidata.ucar.edu/software/netcdf/mailing-lists.html).  While we're 
not entirely certain about the use case for spreading the netCDF data between 
several files, we would be very interested to participate in a broader 
discussion about it with our community.  If it turns out there is a lot of 
interest in this and similar functionality, we will certainly make it a higher 
priority internally.

Thank you for your time, and again, my sincere apologies for the delayed 
response! Have a great day,

-Ward

> Dear NetCDF-Support-Team,
> 
> we have been working on NetCDF external link functionality. This allows
> NetCDF applications to create dimension variables which values are
> stored in an external file. Therefore, it uses the HDF5 virtual dataset
> (VDS) functionality. This is useful for, e.g., climate applications that
> rely on a variable per file and timestep configuration. The idea is to
> store the grid in a separate file and link our data to this grid. We
> already have our first working version. You find the patch and the
> examples on our page:
> 
> http://wr.informatik.uni-hamburg.de/research/projects/bullio/netcdf_external_links/start
> 
> Under the hood it uses HDF5 virtual datasets. VDS has the advantage of
> being compatible to the functions that are supported by oridinary
> datasets. Therefore, files containing VDS should be supported by the
> most software.
> 
> There is a minor issue related to HDF5, the call H5F_try_close function
> fails, when ncdump trys to read data from an external dimension. So far
> we found a workaround, but we will fix this issue.
> 
> It would be great if external link functionality could be supported by
> netCDF at some timepoint. We would like to improve our patch and for
> that reason we need your feedback. If you have some idea to the issue
> above, we would be grateful for each hint.
> 
> Regards,
> Eugen Betke
> 
> 
> --
> Eugen Betke
> Abteilung Forschung
> Deutsches Klimarechenzentrum GmbH (DKRZ)
> Bundesstraße 45a • D-20146 Hamburg • Germany
> 
> Phone:  +49 40 460094-146
> Fax: +49 40 460094-270
> E-mail: address@hidden
> URL: http://www.dkrz.de
> 
> Geschäftsführer: Prof. Dr. Thomas Ludwig
> Sitz der Gesellschaft: Hamburg
> Amtsgericht Hamburg HRB 39784
> 
> 


Ticket Details
===================
Ticket ID: OHL-714122
Department: Support netCDF
Priority: Emergency
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.