[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 980218: netCDF buffer pointer problems with large files

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: 980218: netCDF buffer pointer problems with large files
  • Date: Wed, 18 Feb 1998 14:49:44 -0700
> From: address@hidden (Tom Umeda)
> Subject: netCDF buffer pointer problems with large files
> Keywords: 199802182040.NAA24984 netCDF large files
>
> Hi,
>
> I have encounter a arithmetic overflow problem when I write out large
> netCDF files.
>
> I have traced the problem to the ncio_px_rel routine
> line 222 in  "posixio.c".
>
> The instruction is:
>
>       assert(pxp->bf_offset <= offset
>                  && offset < pxp->bf_offset + (off_t) pxp->bf_extent);
>
> dbx prints:
>
>  (dbx 25) p pxp->bf_offset
>   pxp->bf_offset = 2147450880
>
>   (dbx 26) p (off_t) pxp->bf_extent
>   (off_t ) pxp->bf_extent = 32768
>
>   (dbx 27) p 2147450880+32768
>   2147450880+32768 = -2147483648
>
> This overflow causes assert to fail and program to abort.
>
> Is there a work around?
>
> Thank you
> Tom Umeda

Tom:

You don't say what type of system you are running on or
which version of netcdf-3 you are using.

On most unix systems, the size of netcdf files is limited by
the range of the type off_t. 'off_t' is required to be a signed
integral type, on most unix systems it is declared
typedef long            off_t;

On a few systems, notably IRIX 6 and beyond, this is a 64 bit type,
so the largest offset is 2^63. On most other systems, it is a 32 bit
type, so the largest offset is 2^31. You can obtain a 64bit environment
on recent versions of Solaris as well, using certain compile time options.
See 'man lfcompile'

We did not use off_t consistantly for offset math until fairly recently,
(August 1997). In fact, the version which has the fixes for correct access
to large files has not been generally available. If you grab
ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-3.4a.tar.Z
it has the appropriate fixes. I must emphasize that this is only useful if the
system has 64 bit offsets.

It is interesting that you are actually executing the assert() statement.
I would think that your version of the library would have been compiled
-DNDEBUG, which would turn assert() statements into no-ops.

-glenn