[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Apparent filesize limit?

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: Apparent filesize limit?
  • Date: Sun, 17 Sep 2006 22:02:46 -0600
Hi John,

> Hello, I've created a NetCDF convention to amass into one 
> outputfile results contained in many (e.g. 50) NetCDF files
> (of the AnDI for spectrometry type).  Data in the new 
> convention include two 3-D arrays, with their unlimited 
> dimension being the number of input files. But these 
> conventions/types may be unimportant to my question.
> 
> After some coding false starts, I can now iterative through
> a list of input files and load data into the output file
> without obvious memory problems. But all stops at iteration  
> 38, when the output file just exceeds 2 gigabytes, 
> specifically, when it is 2,097,144 kilobytes. 
> 
> My question:  Have I run up against a NetCDF file size 
> Limitation?  Or is this some glitch in R or in Windows?

Sorry about the delay in responding, I've been away from my email for
a couple of weeks.

Before version 3.6 of netCDF, file size was essentially limited to
2**31 bytes (denoted 2 GiB).  But Large File Support was added to
netCDF with version 3.6.0 and is in the current netCDF 3.6.1 release.
For more information about this, see the Frequently Asked Questions
section on Large File Support:

  http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#lfs

In short, netCDF files can now be much larger, but you may have to
wait for third-party utilities and packages such as R to relink or
adapt to netCDF version 3.6 so you can take advantage of this
support.  As one of the FAQs says:

  Should I start using the new 64-bit offset format for all my netCDF
  files?

  No, we discourage users from making use of the new format unless
  they need it for very large files. It may be some time until
  third-party software that uses the netCDF library is upgraded to 3.6
  or later versions that support the new large file facilities, so we
  advise continuing to use the classic netCDF format for data that
  doesn't require huge file offsets. The library makes this
  recommendation easy to follow, since the default for file creation
  is the classic format.

Please let us know if you have more questions not answered by the FAQs
referenced above.

--Russ
_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu