Re: netcdf-Java error reading HDF5 with large attributes

John,

I sent you the e-mail below back in late September regarding a problem using 
netCDF-Java to open HDF datasets that have large attributes, i.e., 
character-string attributes with very long length. Although my earlier e-mail 
indicated that the threshold seemed to be at 64 kB, I recently heard from 
someone that he is having trouble with an HDF dataset where one of the 
attributes is 54 kB in length.

This problem is being brought to my attention by a group at NASA-JPL involved 
with an upcoming Earth-observing mission. They have so far been using Panoply 
(and thus netCDF-Java) to work with test datasets but will be working with 
“real” data in a few months. Use of these long attributes is apparently 
specified by mission requirements, so it’s not something they are going to stop 
doing.

In my prior e-mail I mentioned this problem occurring with NJ 4.3.22. I have 
since updated Panoply to work with the NJ 4.5.* branch and that the problem 
occurs there also. It does look like there has been some (minor?) change 
between 4.3.22. and 4.5.3 as the line number where the exception happens has 
shifted a bit. Using NJ 4.5.3, I now see

java.io.IOException: java.io.IOException: Negative seek offset
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:431)
        at 
ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:744)
        at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:426)
        at ucar.nc2.dataset.NetcdfDataset.acquireDataset(NetcdfDataset.java:527)
        at ucar.nc2.dataset.NetcdfDataset.acquireDataset(NetcdfDataset.java:504)
        at gov.nasa.giss.data.nc.NcDataset.initMe(NcDataset.java:109)

Caused by: java.io.IOException: Negative seek offset
        at ucar.unidata.io.RandomAccessFile.seek(RandomAccessFile.java:404)
        at ucar.nc2.iosp.hdf5.FractalHeap.readDirectBlock(FractalHeap.java:444)
        at ucar.nc2.iosp.hdf5.FractalHeap.<init>(FractalHeap.java:174)
        at 
ucar.nc2.iosp.hdf5.H5header$DataObject.processAttributeInfoMessage(H5header.java:2392)
        at ucar.nc2.iosp.hdf5.H5header$DataObject.<init>(H5header.java:2380)
        at ucar.nc2.iosp.hdf5.H5header$DataObject.<init>(H5header.java:2230)
        at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2077)
        at ucar.nc2.iosp.hdf5.H5header.access$600(H5header.java:70)
        at 
ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.<init>(H5header.java:2123)
        at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:3998)
        at ucar.nc2.iosp.hdf5.H5header.access$900(H5header.java:70)
        at ucar.nc2.iosp.hdf5.H5header$H5Group.<init>(H5header.java:2203)
        at ucar.nc2.iosp.hdf5.H5header$H5Group.<init>(H5header.java:2168)
        at ucar.nc2.iosp.hdf5.H5header.readSuperBlock2(H5header.java:355)
        at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:203)
        at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:130)
        at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1529)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:821)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:428)
        ... 15 more

I realize that updating NJ to deal with this sort of HDF dataset might be 
complex and potentially time-consuming, but do you have any idea if doing so is 
something that will happen in the near or foreseeable future?

Thanks,
rbs







On Sep 23, 2014, at 17:30, Robert Schmunk <address@hidden> wrote:

> 
> John,
> 
> A group at NASA JPL is gearing up to begin serving a new collection of Earth 
> observations data next year and plan to provide HDF5 datasets. They have run 
> into a problem accessing test datasets using my Panoply app, and the 
> exception traces back to code in netCDF-Java’s reading of the HDF5 header .
> 
> One thing that the JPL group is doing that is out of the ordinary is that 
> they are stuffing a very large XML string into a single global attribute. In 
> some cases, the string is longer than 64 kB. When NJ tries to read the file 
> header for those cases, a "Negative seek offset” exception gets thrown. The 
> stack trace that results looks like the following...
> 
> java.io.IOException: java.io.IOException: Negative seek offset
>       at ucar.nc2.NetcdfFile.open(NetcdfFile.java:425)
>       at 
> ucar.nc2.dataset.NetcdfDataset.openOrAcquireFile(NetcdfDataset.java:699)
>       at ucar.nc2.dataset.NetcdfDataset.openDataset(NetcdfDataset.java:421)
>       at ucar.nc2.dataset.NetcdfDataset.acquireDataset(NetcdfDataset.java:516)
>       at ucar.nc2.dataset.NetcdfDataset.acquireDataset(NetcdfDataset.java:493)
>       at gov.nasa.giss.data.nc.NcDataset.initMe(NcDataset.java:107)
> 
> Caused by: java.io.IOException: Negative seek offset
>       at ucar.unidata.io.RandomAccessFile.seek(RandomAccessFile.java:402)
>       at ucar.nc2.iosp.hdf5.FractalHeap.readDirectBlock(FractalHeap.java:463)
>       at ucar.nc2.iosp.hdf5.FractalHeap.<init>(FractalHeap.java:180)
>       at 
> ucar.nc2.iosp.hdf5.H5header$DataObject.processAttributeInfoMessage(H5header.java:2344)
>       at ucar.nc2.iosp.hdf5.H5header$DataObject.<init>(H5header.java:2332)
>       at ucar.nc2.iosp.hdf5.H5header$DataObject.<init>(H5header.java:2180)
>       at ucar.nc2.iosp.hdf5.H5header.getDataObject(H5header.java:2027)
>       at ucar.nc2.iosp.hdf5.H5header.access$600(H5header.java:70)
>       at 
> ucar.nc2.iosp.hdf5.H5header$DataObjectFacade.<init>(H5header.java:2073)
>       at ucar.nc2.iosp.hdf5.H5header.readGroupNew(H5header.java:3911)
>       at ucar.nc2.iosp.hdf5.H5header.access$900(H5header.java:70)
>       at ucar.nc2.iosp.hdf5.H5header$H5Group.<init>(H5header.java:2153)
>       at ucar.nc2.iosp.hdf5.H5header$H5Group.<init>(H5header.java:2118)
>       at ucar.nc2.iosp.hdf5.H5header.readSuperBlock2(H5header.java:354)
>       at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:206)
>       at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:128)
>       at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1521)
>       at ucar.nc2.NetcdfFile.open(NetcdfFile.java:813)
>       at ucar.nc2.NetcdfFile.open(NetcdfFile.java:422)
>       ... 15 more
> 
> Note: The above comes from NJ 4.3.22. I have not yet switched up to NJ 4.5.X, 
> although I plan to do so after the next Panoply release.
> 
> rbs




--
Robert B. Schmunk
Webmaster / Senior Systems Programmer
NASA Goddard Institute for Space Studies
2880 Broadway, New York, NY 10025