[netCDF #ZCE-849683]: clarification on large file support

Fabio,

> Please, I need just one more clarification.
> I would like to use the netcdf library on windows 64 bit platform.
> In order to activate very large file support, is it sufficient to compile
> the library specifying a 64bit environment and activate the 64bit file
> offset. Doing so, does the size_t type automatically switch to a 64 bit
> unsigned integer variable in such a way not to limit memory and thus data
> set allocation. I am using your code and Visual studio 2008 and I'm
> specifing a 64 bit target environment.

With a 64-bit development environment, there is no need to "activate
the 64bit file offset", and this depends on the off_t type, not
size_t.

All versions of netCDF since 3.6.0 include support for reading and
writing 64-bit offset netCDF files, whether compiled with a 32-bit or
64-bit development environment, if the size of the C off_t type is at
least 8 bytes. The size of the off_t type for your 64-bit Windows
development environment is surely 8 bytes, but you can test this by
printing "sizeof(off_t)" in a small C program, and you might as well 
also print the size of size_t to answer your other question:

#include <sys/types.h>
#include <stdio.h>
int main(void) {
    printf("Size of off_t is %d bytes.\n", sizeof(off_t));
    printf("Size of size_t is %d bytes.\n", sizeof(size_t));
    return 0;
}

If either of these is printed as a value less than 8, make sure you 
use whatever C compiler flags are needed for your compiler to specify 
a 64-bit development environment.

Note that there are still some limits to dataset size using the netCDF
64-bit offset format, as described in the Users Guide and in the FAQ on
large file support, in particular the answers to the three questions:

  Have all netCDF size limits been eliminated?
  Why are variables still limited in size?
  How can I write variables larger than 4 GiB?

  http://www.unidata.ucar.edu/netcdf/docs/faq.html#Large%20File%20Support10

As indicated in the answer to the second question above, the reason variable
sizes are still limited has to do with a desire to make sure that netCDF
64-bit offset files files are still portable, even  to 32-bit platforms.

--Russ

> 
> 
> -----Original Message-----
> From: Unidata netCDF Support [mailto:address@hidden]
> Sent: Monday, July 12, 2010 7:28 PM
> To: address@hidden
> Cc: address@hidden; address@hidden;
> address@hidden
> Subject: [netCDF #ZCE-849683]: clarification on large file support
> 
> Fabio,
> 
> A little more clarification is needed to my last reply.  I said:
> 
> Each variable in the file cannot exceed 4GB (not 2GB), in netCDF
> versions after 3.6.1, including the current netCDF 4.1.1. The actual
> maximum size of a variable on a 32-bit platform is (2^32 - 4) bytes.
> Part of the confusion is a documentation error here:
> 
> 
> http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Classic-Limitat
> ions
> 
> which I just discovered hasn't been updated since the size limit on a
> single variable was changed from 2GB (2^31 - 4) to 4GB (2^32 - 4) in
> versions since netCDF 3.6.1.
> 
> I was confused and the original documentation is correct.  The netCDF
> classic format limits all but the last variable to 2GB in size.  It is the
> 64-bit offset file format that permits all variables to be 4GB in size, and
> the last variable to be even larger.  The netCDF-4/HDF5 format variant has
> no 4GB limits on the size of any variable.  Sorry for the confusion!
> 
> --Russ
> 
> 
> 
> Russ Rew                                         UCAR Unidata Program
> address@hidden                      http://www.unidata.ucar.edu
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: ZCE-849683
> Department: Support netCDF
> Priority: Normal
> Status: Closed
> 
> 
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: ZCE-849683
Department: Support netCDF
Priority: Normal
Status: Closed