Re: netCDF question (fwd)

------- Forwarded Message

Date:    Mon, 29 Oct 2001 12:46:51 -0700
From:    Russ Rew <address@hidden>
To:      Dennis Shea <address@hidden>
Subject: Re: netCDF question 

Hi Dennis,

> I was answering a question in "comp.lang.fortran"
> on netCDF vs HDF5
> 
> I gave some pros-n-cons of each. One "limitation" of netCDF
> which I listed was:
> 
> >   (2) 2 GB limit on file size 
> 
> One person sent the following "nitpick" on this comment:
> 
> >  The limit only applies to a single record. Compiled with the right
> >  flags on a 64-bit architecture, the filesize can exceed 2^32 bytes,
> >  however, single records still have that limit.
> 
> Is this true? Would it not compromise one of netCDF's main
> selling points: architecture independence?

It is true that you can write netCDF files that exceed 2 GB on
platforms that have "Large File Support" (LFS), and that doesn't even
necessarily mean a 64-bit architecture.  For example, on a 32-bit
Linux system I was able to write and read a 6 GB netCDF file, from
either Fortran or C.  This file would be platform-independent for any
LFS platform.  But if you tried to call nc_open on such a file on a
platform without LFS, you would get a "file too large" error.  That's
the same thing that would happen if you wrote a large text file on an
LFS platform and tried to use it on a platform that didn't have LFS.
But I might still call such text files "platform-independent".

As far as I know, all 64-bit architectures have LFS, but now some
32-bit architectures do also (such as Linux).

I should write an FAQ on this subject soon, giving all the gory
details.

The constraints are a little more complicated than just "single
records can't exceed 2 Gbytes".  If you don't use the UNLIMITED
dimension, only one variable can exceed 2 Gbytes in size, but it can
be as large as the underlying system permits.  If you use the
UNLIMITED dimension,  any number of record variables can exceed 2
Gbytes in size, as long as the offset of the start of each record
variable within a record is less than 2 Gbytes.  But the size of a
record can actually also exceed 2 Gbytes, if the last variable in the
record has very large record slices.

One of these days, I'll try to explain this all more clearly in an
FAQ, and give some examples.  I also need to update the User's Guides,
since they also say that netCDF files are limited to 2 Gbytes in size,
even though they haven't been since netCDF 3.4.

--Russ


------- End of Forwarded Message