[netCDF #DFM-455316]: Looks like a bug in the netCDF NC_64BIT_OFFSET variables

Hi Kari,

> Actually, I have a workaround: I use unlimited for time dimension and I
> have verified that the big cubes work.
> 
> I was just testing this setup for reading speed. We now are serving
> modis4 data directly out of 85 GB netcdf file at
> http://128.252.202.19:8080/NASA
> So the reading speed is quite important, and I found this bug while
> benchmarking.

We found the cause of the bug and a fix for it.  The fix will be in
the next snapshot release.  This bug has been in the code since 2004,
when the 64-bit offset format was first added to netCDF 3.6.0, so it's
great that you discovered it and sent in a reproducible test case that
demonstrated it.

The bug occurs if all of the following conditions are true:

  - writing a variable of any type with more than 2**32 values
  - the variable must have 2 or more dimensions
  - the platform on which the file is written must be 32-bit
  - the file can be either classic format or 64-bit offset format
  - the values to be written must begin beyond the first 2**32 values

In this case, a file offset is computed for writing using 32-bit
arithmetic for a multiply, but it should be using 64-bit arithmetic.
This results in an undetected integer overflow and can overwrite data
earlier in the file, without an error or warning.

The one-line fix is to change a statement in libsrc/putget.c from

                    lcoord += *up * *ip;
to
                    lcoord += (off_t)(*up) * (off_t)(*ip);

Thanks again for reporting this bug!

--Russ
 
> I'm already using HDF5 version 1.8.4-patch1
> 
> I compiled my library from
> ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf.tar.gz for win32.
> 
> Kari
> 
> > Howdy Kari!
> >
> > Looks like you found a bug of some kind. I have reproduced your problem in 
> > the netCDF build with new test libsrc4/tst_large2.c.
> >
> > Now we will work on a fix. I will let you know when we have it. Meanwhile, 
> > as a workaround, I note that the NC_NETCDF4 format works fine for your test 
> > program.
> >
> > If you are going to use netCDF-4, make sure you get the HDF5-1.8.4-patch1 
> > release, and the recent 4.1.1 release of netcdf.
> >
> > Thanks,
> >
> > Ed
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: DFM-455316
> > Department: Support netCDF
> > Priority: Normal
> > Status: Open
> >
> >
> >
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: DFM-455316
Department: Support netCDF
Priority: Critical
Status: Closed