This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
> I'm quite sorry, > Actually I'm writing much more than what I stated in my previous email > > I'm trying to dump on disk about 9GB. But It does not work and provides me > the error I already said. ... > I'm still trying to write a large file on a 64bit platform. > > I fixed the variables type and now I get 8 bytes for both size_t and off_t > system variables. > > ... I get an error: 22 > ("invalis argument") from the nc_endded function. > > What is the cause? The netCDF error code for "Invalid Argument" is (-36), so the error you are getting of 22 is from the operating system, in /usr/include/errno.h: #define EINVAL 22 /* Invalid argument */ This may still be a netCDF library problem, but there are other reasons related to your file system or operating system that may be the cause. See the answer to the FAQ "Why do I get an error message when I try to create a file larger than 2 GiB with the new library?": http://www.unidata.ucar.edu/netcdf/docs/faq.html#Large%20File%20Support12 Can you create large files with any other program on your system? For example, do you get an error from the operating system trying to write a 5GB file with a program that doesn't use the netCDF library, such as the following little C program? If so, try compiling it with -D_FILE_OFFSET_BITS=64 flag. If that still doesn't work, you'll need to configure your file system for large file support. --Russ /* * See if we can write a 5GB file, to make sure large file system * supported. Note: may need to compile this with * -D_FILE_OFFSET_BITS=64 flag on 32-bit systems to get an 8-byte * off_t type. */ #include <stdlib.h> #include <stdio.h> #include <string.h> #include <errno.h> extern int errno; #define MILLION 1000000 int main() { int i, j, k; float data[MILLION]; size_t written; FILE *f; f = fopen("bigfile", "w"); if(!f) { printf("error opening file: %s\n", strerror(errno)); return errno; } for(i=0; i < 1250; i++) { written = fwrite(data, sizeof(float), MILLION, f); if (written < MILLION) { printf("error writing, errno=%d, %s\n", errno, strerror(errno)); return errno; } } return 0; } > -----Original Message----- > From: Unidata netCDF Support [mailto:address@hidden] > Sent: Friday, July 23, 2010 6:26 PM > To: address@hidden > Cc: address@hidden; address@hidden; > address@hidden; address@hidden > Subject: [netCDF #ZCE-849683]: clarification on large file support > > > I did as you suggested me. > > > > But I get the following outcome: > > > > Size of Size_t = 8 ok > > Size of Off_t = 4 !unbelievable > > > > How does it come? > > We expect both of quantities to be equal to 8 bytes on a 64 bit > > platform, don't they? > > I'm surprised that the default size of off_t is only 4, but maybe you need > to set a C macro to get an 8-byte off_t. Try compiling and running this C > program: > > #define _FILE_OFFSET_BITS 64 > #include <sys/types.h> > #include <stdio.h> > int main(void) { > printf("Size of off_t is %d bytes.\n", sizeof(off_t)); printf("Size of > size_t is %d bytes.\n", sizeof(size_t)); return 0; } > > If that shows sizeof(off_t) is now 8 bytes, then you can rebuild netCDF > setting the CFLAGS environment variable, for example > > CFLAGS='-D_FILE_OFFSET_BITS=64' > > > Maybe some particular preprocessor have to be defined before compiling > > the netcdf lib? > > I can find the following preprocessors in my solution: > > _WINDOWS;_USRDLL;NETCDF_EXPORTS;VERSION=3.6.1-beta1;DLL_EXPORT;VISUAL_ > > CPLUSP > > LUS;_FILE_OFFSET_BITS=64 > > > > I removed the NETCDF_DLL preprocessor since I need to create a static > > library rather then a dynamic one. > > > > Can you give me some more suggestions to get the library work properly > > with large file support? > > I wish I knew more about the Windows compilation environment for 64-bit > support. We may get these problems solved in the next six months with our > planned port to Windows that was recently announced on the netcdfgroup > mailing list: > > > http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2010/msg00257 > .html > > --Russ > > > -----Original Message----- > > From: Unidata netCDF Support [mailto:address@hidden] > > Sent: Monday, July 19, 2010 6:15 PM > > To: address@hidden > > Cc: address@hidden; address@hidden; > > address@hidden; address@hidden > > Subject: [netCDF #ZCE-849683]: clarification on large file support > > > > Fabio, > > > > > Please, I need just one more clarification. > > > I would like to use the netcdf library on windows 64 bit platform. > > > In order to activate very large file support, is it sufficient to > > > compile the library specifying a 64bit environment and activate the > > > 64bit file offset. Doing so, does the size_t type automatically > > > switch to a 64 bit unsigned integer variable in such a way not to > > > limit memory and thus data set allocation. I am using your code and > > > Visual studio 2008 and I'm specifing a 64 bit target environment. > > > > With a 64-bit development environment, there is no need to "activate > > the 64bit file offset", and this depends on the off_t type, not size_t. > > > > All versions of netCDF since 3.6.0 include support for reading and > > writing 64-bit offset netCDF files, whether compiled with a 32-bit or > > 64-bit development environment, if the size of the C off_t type is at > > least 8 bytes. The size of the off_t type for your 64-bit Windows > > development environment is surely 8 bytes, but you can test this by > > printing "sizeof(off_t)" in a small C program, and you might as well > > also print the size of size_t to answer your other question: > > > > #include <sys/types.h> > > #include <stdio.h> > > int main(void) { > > printf("Size of off_t is %d bytes.\n", sizeof(off_t)); printf("Size of > > size_t is %d bytes.\n", sizeof(size_t)); return 0; } > > > > If either of these is printed as a value less than 8, make sure you > > use whatever C compiler flags are needed for your compiler to specify > > a 64-bit development environment. > > > > Note that there are still some limits to dataset size using the netCDF > > 64-bit offset format, as described in the Users Guide and in the FAQ > > on large file support, in particular the answers to the three questions: > > > > Have all netCDF size limits been eliminated? > > Why are variables still limited in size? > > How can I write variables larger than 4 GiB? > > > > http://www.unidata.ucar.edu/netcdf/docs/faq.html#Large%20File%20Suppor > > t10 > > > > As indicated in the answer to the second question above, the reason > > variable sizes are still limited has to do with a desire to make sure > > that netCDF 64-bit offset files files are still portable, even to 32-bit > platforms. > > > > --Russ > > > > > > > > > > > -----Original Message----- > > > From: Unidata netCDF Support > > > [mailto:address@hidden] > > > Sent: Monday, July 12, 2010 7:28 PM > > > To: address@hidden > > > Cc: address@hidden; address@hidden; > > > address@hidden > > > Subject: [netCDF #ZCE-849683]: clarification on large file support > > > > > > Fabio, > > > > > > A little more clarification is needed to my last reply. I said: > > > > > > Each variable in the file cannot exceed 4GB (not 2GB), in netCDF > > > versions after 3.6.1, including the current netCDF 4.1.1. The actual > > > maximum size of a variable on a 32-bit platform is (2^32 - 4) bytes. > > > Part of the confusion is a documentation error here: > > > > > > > > > http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Classic > > > -L > > > imitat > > > ions > > > > > > which I just discovered hasn't been updated since the size limit on > > > a single variable was changed from 2GB (2^31 - 4) to 4GB (2^32 - 4) > > > in versions since netCDF 3.6.1. > > > > > > I was confused and the original documentation is correct. The > > > netCDF classic format limits all but the last variable to 2GB in > > > size. It is the 64-bit offset file format that permits all > > > variables to be 4GB in size, and the last variable to be even > > > larger. The netCDF-4/HDF5 format variant has no 4GB limits on the > > > size of any variable. Sorry for > > the confusion! > > > > > > --Russ > > > > > > > > > > > > Russ Rew UCAR Unidata Program > > > address@hidden http://www.unidata.ucar.edu > > > > > > > > > > > > Ticket Details > > > =================== > > > Ticket ID: ZCE-849683 > > > Department: Support netCDF > > > Priority: Normal > > > Status: Closed > > > > > > > > > > > > > > > > Russ Rew UCAR Unidata Program > > address@hidden http://www.unidata.ucar.edu > > > > > > > > Ticket Details > > =================== > > Ticket ID: ZCE-849683 > > Department: Support netCDF > > Priority: Normal > > Status: Closed > > > > > > Russ Rew UCAR Unidata Program > address@hidden http://www.unidata.ucar.edu > > > > Ticket Details > =================== > Ticket ID: ZCE-849683 > Department: Support netCDF > Priority: Normal > Status: Closed > > Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: ZCE-849683 Department: Support netCDF Priority: Normal Status: Closed