[netCDF #HSP-129967]: [netcdfgroup] possible bug in 3.6.2 with

Hi Mario,

I've developed what I think is a more complete fix for the problem you
reported, as well as a related problem discovered while investigating
this bug.  I'm sending this fix before posting or announcing it, in
case you have time to try it or can see any additional problems with
it.  The fix will also be in the next snapshot release of netCDF-3 and
netCDF-4.  My testing seems to show that it fixes these problems:

 1. On 64-bit platforms (with size_t an unsigned 64-bit type):
   -  for byte variables, if the product of dimensions is
      greater than 2**32, an assertion violation occurs
         Assertion failed: *ulp <= X_SIZE_MAX, file ncx.c, line 1810
   -  for short variables, if the product of dimensions is
      greater than 2**31, the same assertion violation occurs
         Assertion failed: *ulp <= X_SIZE_MAX, file ncx.c, line 1810

 2. On 32-bit platforms (with size_t an unsigned 32-bit type):
   -  for short variables, if the product of dimensions is
      greater than 2**31, a different assertion violation occurs
         Assertion failed: remaining > 0, file putget.c, line 347
   -  for any type of variable, if the product of dimensions is exactly
      2**32 or exactly any higher multiple of 2**32, a divide by zero
      occurs
         Arithmetic Exception(coredump)

Although we had some large file tests for byte and short variables,
they did not duplicate any of the conditions above, so were clearly
inadequate.  Most of our large byte variable testing was on 32-bit
platforms, where no assertion violation such as you reported occurs.

I'm glad all these problems resulted in assertion violations or
arithmetic exceptions so they resulted in a program halting, rather
than bad data being written or existing files corrupted.  If you
encounter any additional problems, especially of the latter sort, we
will try to handle them with the high priority they deserve.

Here's the replacement code for lines 394-426 in libsrc/var.c that
seems to fix all of the above problems:

        for(shp = varp->shape + varp->ndims -1,
                                dsp = varp->dsizes + varp->ndims -1;
                        shp >= varp->shape;
                        shp--, dsp--)
        {
                if(!(shp == varp->shape && IS_RECVAR(varp)))
                {
                        if( *shp <= X_UINT_MAX / product )
                        {
                                product *= *shp;
                        } else
                        {
                                product = X_UINT_MAX ;
                        }
                }
                *dsp = product;
        }


out :
    if( varp->xsz <= (X_UINT_MAX - 1) / product ) /* if integer multiply will
    not overflow */
        {
                varp->len = product * varp->xsz;
                switch(varp->type) {
                case NC_BYTE :
                case NC_CHAR :
                case NC_SHORT :
                        if( varp->len%4 != 0 )
                        {
                                varp->len += 4 - varp->len%4; /* round up */
                /*              *dsp += 4 - *dsp%4; */
                    }
                    break;
                default:
                        /* already aligned */
                        break;
                }
        } else
        {       /* OK for last var to be "too big", indicated by this special
        len */
                varp->len = X_UINT_MAX;
        }

You also reported:

> It is reliable if you stick to very strict constraints. I would actually
> rather say 'It works, but the constraints are not easy to enforce'.
>   - only one variable > 4GB per file, and this must be the last variable
>     in the file.
>   - no attributes stored for variables > 2GB.

We are not aware of any problems with attributes for variables > 2GB.
Because variable attributes are all stored in the file header, before
any of the data, it's hard to see how the size for a variable could
affect its attributes.  If you have more details about this problem or
a program that demonstrates it, we would be very interested in
understanding and fixing the problem.

A problem you reported with the C++ interface:

>   - When creating new files, use NcFile::Replace rather than NcFile::New
>     even if the file does not exist, because NcFile::New ignores
>     NcFile::Offset64Bits, so the files have no large file support.

was fixed in our development code last April, so is only available in
the current "snapshot" release from

  ftp://ftp.unidata.ucar.edu/pub/netcdf/snapshot/netcdf-daily.tar.gz

but the fix involved changing the line 378 in cxx/netcdf.cpp from

        mode = NC_WRITE;

to

        mode |= NC_WRITE;

By the way, your name has been added to the list acknowledging netCDF
contributors at:

  http://www.unidata.ucar.edu/software/netcdf/credits.html

Thanks again for reporting the problems and for being persistent when we
inadvertently neglected to notice what you had found.

Regards,
--Russ

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: HSP-129967
Department: Support netCDF
Priority: Normal
Status: Closed