[netCDF #MJT-105642]: Is this file netCDF4 API compliant?

Hi Charlie,

> The forwarded message contains information related to the support request...

Thanks, that should help.

I've been corresponding with Joe Lee and trying some experiments with h5py to 
try to determine under what conditions this bug occurs. I think it's something
more specific than just use of the unlimited dimension in HDF5 files.
 
I expect you have probably tested netCDF-4 on lots more HDF5 data than we
have. Have you tested it successfully with HDF5 data (not created using the 
netCDF-4 library) that uses one or more HDF5 unlimited dimensions, either
with or without dimension scales?

I'll continue looking at this problem, but don't have a good handle on it yet.

Part of the problem is that the HDF5 reading facilities in netCDF-4 were 
developed
in order to read some specific HDF5 NASA datasets that are in common use, but
without determining or documenting exactly what HDF5 files can be read with 
netCDF-4. 

We list which HDF5 features we know can't be read through netCDF-4:

  
http://www.unidata.ucar.edu/netcdf/docs/ncFAQ.html#How-can-I-convert-HDF5-files-into-netCDF-4-files

but I don't know don't know whether that list is complete or whether there
are combinations of HDF5 features not in that list that make a file unreadable
through the netCDF-4 API, such as the example Joe Lee has constructed. I
also don't yet know whether that example merely demonstrates a simple bug
in the netCDF-4 C library, or some missing code that would need to be 
developed ...

--Russ

> -------- Forwarded Message --------
> Subject: RE: [DIWG] Suggestions to The Six Recommendations of the 2013
> HDF5WG
> Date: Wed, 9 Sep 2015 18:11:36 +0000
> From: Joe Lee <address@hidden>
> To: Charlie Zender <address@hidden>
> CC: Kent Yang <address@hidden>, Peter Leonard
> <address@hidden>, Aleksandar Jelenak <address@hidden>, Ted
> Habermann <address@hidden>
> 
> Hi, Charlie!
> 
> The file was created with HDF Product Designer (HPD). Please see the
> attached screenshot.
> 
> You can reproduce it with the attached Python script (h5py) that was
> generated by HPD.
> 
> I followed the DIWG recommendation by attaching dimension scales.
> 
> My observation is that unlimited dimension causes the trouble for
> Unidata software (both NetCDF-C and NetCDF-Java).
> 
> 
> -----Original Message-----
> From: Charlie Zender [mailto:address@hidden]
> Sent: Wednesday, September 09, 2015 12:38 PM
> To: Joe Lee
> Cc: Kent Yang; Peter Leonard
> Subject: Re: [DIWG] Suggestions to The Six Recommendations of the 2013
> HDF5WG
> 
> [Please reduce DIWG list e-mail traffic by only CC'ing DIWG on the most
> important emails]
> 
> Dear Joe,
> 
> I am attempting to ascertain why the file you sent is not read correctly
> by ncks or ncdump. You never explicitly said that whoever created this
> dataset followed interoperability best practices for creating
> netCDF4-readable HDF files.
> I have no way of knowing from the file itself how it was created, and
> I'm not skilled enough reading h5dump to understand all the info.
> Please verify that the dataset was created using best practices
> (dimension scales etc.), and please send a reproducible script/program/
> method to create the dataset so we can understand why the best practices
> did not result in a file readable by the netCDF4-API.
> 
> Thanks!
> Charlie
> 
> On 09/03/2015 10:19 AM, Joe Lee wrote:
> > Hi, all!
> >
> > I don't know why people are so confident about the use of NetCDF-4 that 
> > relies on HDF5 dimension scale and keep recommending its use along with 
> > shared dimensions.
> > Please look at the attached HDF5 template.
> >
> > You'll find that HDF5-C (correct) and  NetCDF-Java/NetCDF-C (wrong) reports 
> > the output differently.
> >
> > H5dump  - it reports correctly z(unlimited, 9):
> >
> >     DATASET "z" {
> >        DATATYPE  H5T_IEEE_F32LE
> >        DATASPACE  SIMPLE { ( 0, 9 ) / ( H5S_UNLIMITED, 9 ) }
> >        DATA {
> >        }
> >
> > Ncks --cdl (netCDF-C) - it reports as z(unlimited, unlimited):
> >
> > C:\nco>ncks --cdl test_unlimited.h5
> > netcdf test_unlimited {
> >    dimensions:
> >      m = UNLIMITED ; // (9 currently)
> >      u = UNLIMITED ; // (0 currently)
> >
> >    variables:
> >      float k(u,m) ;
> >      float m(m) ;
> >      float u(u) ;
> >      float z(m,m) ;
> >
> > z(m,9)
> >
> > THREDDS (netcDF-Java) - it doesn't show even z:
> >
> > Dataset {
> >      Float32 m[m = 0];
> >      Float32 u[u = 0];
> > } testAll/hpd/test_unlimited.h5;
> >
> >
> > -----Original Message-----
> > From: address@hidden
> > [mailto:address@hidden] On Behalf Of
> > Charlie Zender
> > Sent: Thursday, September 03, 2015 11:50 AM
> > To: address@hidden
> > Subject: Re: [DIWG] Suggestions to The Six Recommendations of the 2013
> > HDF5WG
> >
> > The wording of Recommendation #1 is intentional.
> > Only HDF5 features accessible through the netCDF4 API should be used in 
> > future NASA-distributed datasets.
> > This means dimensions must use dimension scales etc.
> > I am unaware of any dataset content that _requires_ using an HDF5 feature 
> > that cannot be made accessible through the netCDF4 API with a little effort 
> > on the part of the HDF5 dataset designer.
> >
> > On 09/03/2015 09:32 AM, Lynnes, Christopher S. (GSFC-5860) wrote:
> >>
> >>> On Sep 3, 2015, at 11:36 AM, Kent Yang <address@hidden> wrote:
> >>>
> >>> Reasons:
> >>> 1) NetCDF-4 is a subset of HDF5. Not all HDF5 features can be supported 
> >>> by NetCDF-4.  If a NASA data product is suitable to use those features, 
> >>> even though this product cannot be accessed by NetCDF-4, it should be 
> >>> fine. These cases may be rare. But it is still possible.
> >>
> >> It would sure be nice to enumerate said features, and produce a 
> >> recommendation to the effect “if you want your data to be readable by 
> >> netCDF tools, then do not use these HDF5 features:…”.
> >> —
> >> Christopher Lynnes   NASA/GSFC    301-614-5185
> >> “I love it when a plan comes together.” — Hannibal Smith
> >>
> >>
> >
> > --
> > Charlie Zender, Earth System Sci. & Computer Sci.
> > University of California, Irvine 949-891-2429 )'(
> >
> 
> --
> Charlie Zender, Earth System Sci. & Computer Sci.
> University of California, Irvine 949-891-2429 )'(
> 
> 
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: MJT-105642
Department: Support netCDF
Priority: Normal
Status: Closed