[netCDF #KVY-259447]: Memory error writing chunks of large variable in

Hi Karen,

Sorry to take so long responding to your question.

We have reproduced the problem here, and it is a bug.  Your program
demonstrates it well, so thanks for the effort that went in to
isolating code that would show the bug.

If you get the most recent beta code, there is a new option "-s" for
ncdump that shows "special" virtual attributes with properties like
the chunk sizes.  In your case, the output is:

  $ ncdump -h -s r10.nc
  netcdf r10 {
  dimensions:
        time = UNLIMITED ; // (1 currently)
        char_len = 10 ;
        level = 100 ;
        grid_cells = 10485762 ;
  variables:
        float pressure(time, level, grid_cells) ;
                pressure:_Storage = "chunked" ;
                pressure:_ChunkSizes = 1, 100, 10485762 ;

  // global attributes:
                :_Format = "netCDF-4" ;
  }

As you can see, the default chunksizes here define a single large
chunk for each record, in this case of size 4194304800 bytes.  Even
though that size fits in 32 bytes unsigned, HDF5 gets an error
when trying to allocate a chunk that size in memory.

We're in the process of rethinking setting the default chunk sizes for
the upcoming 4.0.1 release, but in the meantime, you can set the
chunk sizes explicitly in your netCDF-4 code to work around this
problem.  Insert the following before the nc_enddef() call to set
the chunks to 1 x 100 x 655360, and you will no longer get an HDF5
error:

   {
       /* Set chunk shape for variable */
       size_t chunks[] = {1, 100, 655360};
       retval = nc_def_var_chunking(ncid, var_varid, NC_CHUNKED, chunks);
       if (retval)
           ERR(retval);
   }

Note that I declared chunks to be a "size_t" above.  Our current netCDF
snapshot uses that type for specifying the shapes of chunks for each
variable, but earlier versions used "int" instead.  Either will work
in this case, but the correct type is size_t.

Also chunks[2} is set to 655360 above.  It should instead be 655361 if
you actually intend to use the last two cell indexes for pres_out[][],
because otherwise you will have an extra chunk with only two elements
in it along the cell dimension.

Thanks again for reporting the problem and for providing a small test
case that demonstrates the bug.

--Russ


Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: KVY-259447
Department: Support netCDF
Priority: Emergency
Status: Closed