[netCDF #XYO-599400]: A possble netCDF-4 bug?

Hi Lynton,

> I attach a small C-code that falls over for me when I run it.
> If I make MEMSIZE3 smaller then it will run OK.
> The problem appears to be connected with fill values?
> Do you get a similiar problem, and is there a way to overcome the
> problem?

Yes, I can reproduce the problem and on my platform it seems to succeed with 
MEMSIZE3 < 8192 and fail when MEMSIZE3 >= 8192, where of course 8192 is 2**13.
I configure with --enable-logging to see what the error is and got the following
HDF5 error:

HDF5-DIAG: Error detected in HDF5 (1.8.3-snap2) thread 0:
  #000: H5Ddeprec.c line 170 in H5Dcreate1(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #001: H5Dint.c line 430 in H5D_create_named(): unable to create and link to 
dataset
    major: Dataset
    minor: Unable to initialize object
  #002: H5L.c line 1639 in H5L_link_object(): unable to create new link to 
object
    major: Links
    minor: Unable to initialize object
  #003: H5L.c line 1862 in H5L_create_real(): can't insert link
    major: Symbol table
    minor: Unable to insert object
  #004: H5Gtraverse.c line 877 in H5G_traverse(): internal path traversal failed
    major: Symbol table
    minor: Object not found
  #005: H5Gtraverse.c line 703 in H5G_traverse_real(): traversal operator failed
    major: Symbol table
    minor: Callback failed
  #006: H5L.c line 1685 in H5L_link_cb(): unable to create object
    major: Object header
    minor: Unable to initialize object
  #007: H5O.c line 2596 in H5O_obj_create(): unable to open object
    major: Object header
    minor: Can't open object
  #008: H5Doh.c line 293 in H5O_dset_create(): unable to create dataset
    major: Dataset
    minor: Unable to initialize object
  #009: H5Dint.c line 1148 in H5D_create(): can't update the metadata cache
    major: Dataset
    minor: Unable to initialize object
  #010: H5Dint.c line 852 in H5D_update_oh_info(): unable to update new fill 
value header message
    major: Dataset
    minor: Unable to initialize object
  #011: H5Omessage.c line 188 in H5O_msg_append_oh(): unable to create new 
message in header
    major: Attribute
    minor: Unable to insert object
  #012: H5Omessage.c line 228 in H5O_msg_append_real(): unable to create new 
message
    major: Object header
    minor: No space available for allocation
  #013: H5Omessage.c line 1940 in H5O_msg_alloc(): unable to allocate space for 
message
    major: Object header
    minor: Unable to initialize object
  #014: H5Oalloc.c line 1032 in H5O_alloc(): object header message is too large
    major: Object header
    minor: Unable to initialize object
line 87 of lynton.c: HDF error

so I think the error is caused by exceeding an HDF5 limit, but haven't looked 
further
into this except to note these two macros in HDF5 source files (which may be red
herrings):

src/H5FDcore.c:#define H5FD_CORE_INCREMENT              8192
hl/src/H5LTanalyze.c:#define YY_READ_BUF_SIZE 8192

> On a separate issue, I have been investigating how effective compression
> is. It appears that it doesn't work at all with Vlen data types. Is this
> correct?, and if not do you have any counter examples?

You are right, compression is not supported for Vlen data types.  This is a
restriction inherited from HDF5, but I can't locate the HDF5 documentation
for this at the moment.  I also don't know if there are HDF5 plans to remedy
this restriction in a future release.

It should be prominently explained in the netCDF-4 documentation also,
but I was surprised to note that it appears to have been overlooked,
both in the User Guide and language-specific API documentation!  I
will make sure we document this restriction of the vlen types both in
the sections on compression and the sections on variable-length types.

Thanks for pointing out this problem.

--Russ



Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: XYO-599400
Department: Support netCDF
Priority: Normal
Status: Closed