This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
>To: address@hidden >From: Steve Hankin <address@hidden> >Subject: 2 Gig limit of netCDF files >Organization: UCAR/Unidata >Keywords: 200205102009.g4AK9Ia05392 Hi Steve, > I was looking at the Web page "What other future work on > netCDF is planned?" > (http://www.unidata.ucar.edu/packages/netcdf/faq.html#plans) > and I see that lifting the 2 gigabyte limit is listed as a > plan for netCDF V4. I'll have to update that web page to better reflect reality one of these days. The current netCDF V3 actually supports files much larger than 2 Gbytes in several ways, summarized in a new section of the User's Guide on "Large File Support", which will soon make its way into new editions of the User's Guides. I've appended a text version, or you can read it in the Fortran90 User's Guide: http://www.unidata.ucar.edu/packages/netcdf/f90/Documentation/f90-html-docs/guide9.html#2236524 > My question: can you estimate the availability date for > this? > > If not, then might the >2 gig limit feature be made > available earlier as a patch or a beta version? Anything more than the current Large File Support requires a new format for netCDF data, not just code changes. So it cannot be done with a patch. That's the idea of the proposal to base netcdf4 on HDF-5, as in a proposal I submitted last year (URL available on request). Until something like that ambitious effort gets funded, I hope you can get along with the existing Large File Support and the constraints it imposes. --Russ 9.3 Large File Support It is possible to write netCDF files that exceed 2 GB on platforms that have "Large File Support" (LFS). Such files would be platform-independent to other LFS platforms, but if you call nc_open to access data from such a file on an older platform without LFS, you would expect a "file too large" error. There are important constraints on the structure of large netCDF files that result from the 32-bit relative offsets that are part of the netCDF file format: * If you don't use the unlimited dimension, only one variable can exceed 2 Gbytes in size, but it can be as large as the underlying file system permits. It must be the last variable in the dataset, and the offset to the beginning of this variable must be less than about 2 Gbytes. For example, the structure of the data might be something like: netcdf bigfile1 { dimensions: x=2000; y=5000; z=10000; variables: double x(x); // coordinate variables double y(y); double z(z); double var(x, y, z); // 800 Gbytes } * If you use the unlimited dimension, any number of record variables may exceed 2 Gbytes in size, as long as the offset of the start of each record variable within a record is less than about 2 Gbytes. For example, the structure of the data in a 2.4 Tbyte file might be something like: netcdf bigfile2 { dimensions: x=2000; y=5000; z=10; t=UNLIMITED; // 1000 records, for example variables: double x(x); // coordinate variables double y(y); double z(z); double t(t); // 3 record variables, 2.4 Gbytes per record double var1(t, x, y, z); double var2(t, x, y, z); double var3(t, x, y, z); }