[netCDFJava #MAO-189374]: Reading HDF5 files containing 'hdf5 external links'

Hi Didier:

We use NcML Aggregation for this kind of thing. The main limitation is that its 
only used in Netcdf-Java, though it _may_ eventually get added to the C 
library. But if you are using java, this is a viable option now:

http://www.unidata.ucar.edu/software/netcdf/ncml/


> Hi,
> 
> 
> 
> Thanks for your answer and information.
> 
> 
> 
> To be more precise, what I'm trying to do with external links is to bring 
> together some data with different criteria.
> 
> 
> 
> I send you a more explicit example to illustrate what I'm trying to do.
> 
> 
> 
> I have some data netcdf4 files (it could be also hdf5 files) that contains 
> some data :
> 
> 
> 
> J1.nc
> 
> J2.nc
> 
> J3.nc
> 
> J4.nc
> 
> J5.nc
> 
> 
> 
> Now I want to organise them with different criteria, for instance by 'Cycle' 
> and by 'Pass'.
> 
> 
> 
> So, I create two hdf5 files that contain links to the source data files 
> J1.nc, J2.nc .... :
> 
> 
> 
> The first hdf5 file dataByCyle.h5 contains 3 groups : Cycle001, Cycle002, 
> Cycle003. An each group contains references to source data files:
> 
> Cycle001
> 
> External link --> J1.nc
> 
> External link --> J2.nc
> 
> Cycle002
> 
> External link --> J3.nc
> 
> External link --> J4.nc
> 
> Cycle003
> 
> External link --> J5.nc
> 
> 
> 
> The second hdf5 file dataByPass.h5 contains 2 groups : Pass001, Pass002. An 
> each group contains references to source data files:
> 
> Pass001
> 
> External link --> J1.nc
> 
> External link --> J5.nc
> 
> Pass002
> 
> External link --> J2.nc
> 
> External link --> J3.nc
> 
> External link --> J4.nc
> 
> 
> 
> This allow to classified data (to have a different view), because I need to 
> access data either by 'cycle' or by 'pass' (in this example).
> 
> 
> 
> That's a way that it seems to me rather efficient.
> 
> 
> 
> I first thought to use only Netcdf4 file format, but 'external links' doesn't 
> exists in Netcdf format. And the 'external link' offers the possibility to 
> have different view of the data without to duplicate them (because of the 
> amount of data).
> 
> 
> 
> I also plan to test the possibility to use these files into a TDS. (TDS 4.0).
> 
> 
> 
> I hope my explaination will clarify the purpose of using 'external links'. 
> I'm also always open to other suggestions.
> 
> 
> 
> HDF5ExtLinkExample.zip as attached file contains files to illustrate this 
> example.
> 
> 
> 
> Best regards,
> 
> 
> 
> Didier.
> 
> 
> 
> 
> 
> -----Message d'origine-----
> De : Unidata netCDF Java Support [mailto:address@hidden]
> Envoyé : lundi 23 février 2009 16:48
> À : didier earith
> Cc : address@hidden
> Objet : [netCDFJava #MAO-189374]: Reading HDF5 files containing 'hdf5 
> external links '
> 
> 
> 
> Hi Didier:
> 
> 
> 
> Thanks, we are evaluating the feasibility of supporting this feature. Im 
> wondering how you intend to use this? I see that the hdf5 file contains 
> nothing but the link to the netcdf4 file.
> 
> 
> 
> Also, in terms of CF conformance, I would recommend that you put the 
> attribute alt:positive = "up" on the alt variable, per section 4.3:
> 
> 
> 
> "For this reason the attribute positive as defined in the COARDS standard is 
> required if the vertical axis units are not a valid unit of pressure"
> 
> 
> 
> 
> 
> > Hi,
> 
> >
> 
> > I send you some files as example :
> 
> >
> 
> > - JA2_IPR_2PcP021_001_20090126_085038_20090126_094651.nc is the file that 
> > contains data. It's a netcdf 4 file format (it could also be a HDF5 file 
> > format),
> 
> >
> 
> > - extlink_source.h5 is the hdf5 file that contains a reference (external 
> > link) to JA2_IPR_2PcP021_001_20090126_085038_20090126_094651.nc file.
> 
> >
> 
> > - dump.log is the result of a h5dump (-H) of the extlink_source.h5 file.
> 
> >
> 
> > extlink_source.h5 file has been created with C API of the hdf5 1.8.2 
> > library.
> 
> >
> 
> > Thanks,
> 
> >
> 
> > Didier.
> 
> >
> 
> > -----Message d'origine-----
> 
> > De : Unidata netCDF Java Support [mailto:address@hidden]
> 
> > Envoyé : vendredi 20 février 2009 17:43
> 
> > À : didier earith
> 
> > Objet : [netCDFJava #MAO-189374]: Reading HDF5 files containing 'hdf5 
> > external links '
> 
> >
> 
> > Im sorry Didier, we dont support "external links". If you send me the 
> > files, I will evaluate if we can add that feature in the future.
> 
> >
> 
> > > Full Name: Didier Earith
> 
> > > Email Address: address@hidden
> 
> > > Organization: CLS
> 
> > > Package Version: 4.0.34
> 
> > > Operating System:
> 
> > > Hardware:
> 
> > > Description of problem: Hi,
> 
> > >
> 
> > > I'm try to use Netcdf-java to read HDF5 files with 'HDF5 external
> 
> > > links'.
> 
> > >
> 
> > > I'm using Netcdf-java 4.0.34.
> 
> > >
> 
> > > It seems that Netcdf-Java can't read data which are declared as
> 
> > > 'external link' in the HDF5 files.
> 
> > >
> 
> > > When reading the HDF5 file, I get no errors, but there no variable are
> 
> > > found.
> 
> > >
> 
> > > If I dump the HDF5 file with h5dump all is fine.
> 
> > >
> 
> > > Is it possible to use Netcdf-java library to read HDF5 files containing
> 
> > > 'HDF5 external links' ?
> 
> > >
> 
> > > Thanks.
> 
> > >
> 
> > > Didier.
> 
> > >
> 
> > >
> 
> >
> 
> >
> 
> > Ticket Details
> 
> > ===================
> 
> > Ticket ID: MAO-189374
> 
> > Department: Support netCDF Java
> 
> > Priority: Normal
> 
> > Status: Open
> 
> >
> 
> >
> 
> >
> 
> > Cliquez sur l'url suivante
> 
> > https://www.mailcontrol.com/sr/ZLTfjrpxYGXTndxI!oX7UoWpENiwadNGkUjoCoATthjAy59twyYb95UxruDQLd2!kpERGUhe2VHC8dmstCo9uQ==
> 
> > si ce message est indésirable (pourriel).
> 
> >
> 
> 
> 
> 
> 
> Ticket Details
> 
> ===================
> 
> Ticket ID: MAO-189374
> 
> Department: Support netCDF Java
> 
> Priority: Normal
> 
> Status: Open
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: MAO-189374
Department: Support netCDF Java
Priority: Critical
Status: Closed