Re: 20020627: netCDF 2 GB Size Limit

>To: "'address@hidden'" <address@hidden>
>From: Sweeney Rick <address@hidden>
>Subject: 2 GB Size Limit
>Organization: PRC
>Keywords: 200206271146.g5RBkau08000 netCDF file size

Hi Rick,

> We use NETCDF to store weather data for our AWIPS application. We have
> a particular program with the Air Force that requires an unusually
> large volume of data.  I ran into a condition where the size of the
> file reached 2 GB and never got any larger. Any attempt to write more
> data using the extent that caused the limit to be reached failed.
> 
> I looked on your web site and it said the solution to the problem was
> to build with NDEBUG turned on. I have compiled NETCDF 3.5.0 with no
> debug and linked and it did not solve my problem.

It also says that the underlying file system has to have Large File
Support.  This is not true of all file systems, so you should make
sure your file system supports files larger than 2 Gbytes.

In addition, depending on the platform you have to provide special
compile flags to compile and link with the right libraries to enable
Large File Support.  There are examples in the installation document
at
    http://www.unidata.ucar.edu/packages/netcdf/INSTALL.html

for what flags are needed for AIX, HP-UX, IRIX64, Linux, and Solaris.
If you have a different platform, you'll need to consult the vendor
documentation for whether Large File Support is available and use the
recommended flags.  If you find something that works that we haven't
documented, please send us the details and we'll include it.

It appears that our web documentation doesn't make it clear that these
additional flags are required for Large File Support except in the
INSTALL document, so I'll remedy that soon.

Note that there are additional constraints on netCDF files larger than
2 Gbytes, as discussed here:

  http://www.unidata.ucar.edu/packages/netcdf/faq.html#lfs

Even with Large File Support, the netCDF format doesn't support
arbitrary large files.  For example, you cannot have more than 2
variables that don't use the unlimited dimension each exceeding 2
Gbytes in size.

> Last night I have attempted again to fix the problem. I not only built
> with no debug, but I removed the offending assert statement:
> 
> assert (offset < X_INT_MAX);
> 
> on line 277 of posixio.c
> 
> I have rebuilt the NETCDF library and I have even compiled my
> application with no debug.  I don't know if it works yet because my
> office and my email are temporarily in different locations. If this
> doesn't work, I will next grab the BETA version that you advertise on
> the web site.
> 
> Have you guys really fixed this problem? Do you have any advice on how
> to get around this problem? This is really getting a lot of high
> visibility.

We have successfully written and read 6 Gbyte netCDF files on 32-bit Linux,
AIX, and 64-bit Solaris platforms, so we believe the problem is
fixed.  If you have an example where you have used the recommended
compile flags and it doesn't work, please send us the details so we
can reproduce the problem here and fix it.

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu