[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: NJ and HDF

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: NJ and HDF
  • Date: Mon, 12 Mar 2012 20:32:01 -0500
On Mar 12, 2012, at 20:22, John Caron wrote:

> On 3/12/2012 2:04 PM, Schmunk, Robert B. (GISS-611.0)[SIGMA SPACE 
> CORPORATION] wrote:

>> First, the HDF5 spec allows for an offset superblock. A Panoply user was
>> trying to open an HDF file in which the superblock was offset 1,048,576
>> (512*2048) bytes, and the NJ library was throwing back an exception that
>> the dataset did not look like valid CDM. In poking around the NJ source code,
>> I found that the ucar.nc2.iosp.hdf5.H5header class is coded to look for an
>> offset superblock, but that it is also coded to stop checking beyond
>> maxHeaderPos = 500,000 bytes. Is there a particular reason why this value
>> has been chosen? Or that a max offset has been specified at all?
> 
> isValidFile() has to be fast, because its applied to all files. so 
> scanning through arbitrarily large files is not a good idea.
> 500K is already, i think 11 disk seeks ( which is probably already too 
> long, is you assume an average of 10ms seek time).
> 
> 500K its just a guess as to how big an offset is actually in use.

It's an odd guess, though, when you think about it. The superblock could
start at offset 262,144 or 524,288. So if you were looking to halt the
tests after, e.g., 11 disk accesses, then something slightly larger than
262,144 would have made more apparent sense.

> could you add an alternative interface where the user specifies the file 
> type? that could solve the problem, with some code tweaking.

Hmmmm, I guess I would have to some exploring of the API. I've just been
using NetcdfDataset.acquireDataset and letting the NJ library figure out
whether the dataset was good (whether or not its content actually matched
the extension).

>> Second, in a different case, the NJ library was throwing back just a
>> NullPointerException, with stack trace …

>> In this latter case, do you think this is a bug in NJ's handling of the
>> DIMENSION_LIST attribute, or have the dataset writers done something they
>> shouldn't have?
> 
> not sure, can you send me the file?

I'll send that along separately.

rbs




--
Robert B. Schmunk
NASA Goddard Institute for Space Studies, 2880 Broadway, New York, NY 10025
212-678-5535