[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[netCDF #ZCE-849683]: clarification on large file support
- Subject: [netCDF #ZCE-849683]: clarification on large file support
- Date: Wed, 28 Jul 2010 10:34:26 -0600
> I'm quite sorry,
> Actually I'm writing much more than what I stated in my previous email
>
> I'm trying to dump on disk about 9GB. But It does not work and provides me
> the error I already said.
...
> I'm still trying to write a large file on a 64bit platform.
>
> I fixed the variables type and now I get 8 bytes for both size_t and off_t
> system variables.
>
> ... I get an error: 22
> ("invalis argument") from the nc_endded function.
>
> What is the cause?
The netCDF error code for "Invalid Argument" is (-36), so the error
you are getting of 22 is from the operating system, in /usr/include/errno.h:
#define EINVAL 22 /* Invalid argument */
This may still be a netCDF library problem, but there are other reasons
related to your file system or operating system that may be the cause. See
the answer to the FAQ "Why do I get an error message when I try to create a
file larger than 2 GiB with the new library?":
http://www.unidata.ucar.edu/netcdf/docs/faq.html#Large%20File%20Support12
Can you create large files with any other program on your system? For example,
do you get an error from the operating system trying to write a 5GB file with a
program that doesn't use the netCDF library, such as the following little C
program? If so, try compiling it with -D_FILE_OFFSET_BITS=64 flag. If that
still doesn't work, you'll need to configure your file system for large file
support.
--Russ
/*
* See if we can write a 5GB file, to make sure large file system
* supported. Note: may need to compile this with
* -D_FILE_OFFSET_BITS=64 flag on 32-bit systems to get an 8-byte
* off_t type.
*/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
extern int errno;
#define MILLION 1000000
int
main() {
int i, j, k;
float data[MILLION];
size_t written;
FILE *f;
f = fopen("bigfile", "w");
if(!f) {
printf("error opening file: %s\n", strerror(errno));
return errno;
}
for(i=0; i < 1250; i++) {
written = fwrite(data, sizeof(float), MILLION, f);
if (written < MILLION) {
printf("error writing, errno=%d, %s\n", errno, strerror(errno));
return errno;
}
}
return 0;
}
> -----Original Message-----
> From: Unidata netCDF Support [mailto:address@hidden]
> Sent: Friday, July 23, 2010 6:26 PM
> To: address@hidden
> Cc: address@hidden; address@hidden;
> address@hidden; address@hidden
> Subject: [netCDF #ZCE-849683]: clarification on large file support
>
> > I did as you suggested me.
> >
> > But I get the following outcome:
> >
> > Size of Size_t = 8 ok
> > Size of Off_t = 4 !unbelievable
> >
> > How does it come?
> > We expect both of quantities to be equal to 8 bytes on a 64 bit
> > platform, don't they?
>
> I'm surprised that the default size of off_t is only 4, but maybe you need
> to set a C macro to get an 8-byte off_t. Try compiling and running this C
> program:
>
> #define _FILE_OFFSET_BITS 64
> #include <sys/types.h>
> #include <stdio.h>
> int main(void) {
> printf("Size of off_t is %d bytes.\n", sizeof(off_t)); printf("Size of
> size_t is %d bytes.\n", sizeof(size_t)); return 0; }
>
> If that shows sizeof(off_t) is now 8 bytes, then you can rebuild netCDF
> setting the CFLAGS environment variable, for example
>
> CFLAGS='-D_FILE_OFFSET_BITS=64'
>
> > Maybe some particular preprocessor have to be defined before compiling
> > the netcdf lib?
> > I can find the following preprocessors in my solution:
> > _WINDOWS;_USRDLL;NETCDF_EXPORTS;VERSION=3.6.1-beta1;DLL_EXPORT;VISUAL_
> > CPLUSP
> > LUS;_FILE_OFFSET_BITS=64
> >
> > I removed the NETCDF_DLL preprocessor since I need to create a static
> > library rather then a dynamic one.
> >
> > Can you give me some more suggestions to get the library work properly
> > with large file support?
>
> I wish I knew more about the Windows compilation environment for 64-bit
> support. We may get these problems solved in the next six months with our
> planned port to Windows that was recently announced on the netcdfgroup
> mailing list:
>
>
> http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2010/msg00257
> .html
>
> --Russ
>
> > -----Original Message-----
> > From: Unidata netCDF Support [mailto:address@hidden]
> > Sent: Monday, July 19, 2010 6:15 PM
> > To: address@hidden
> > Cc: address@hidden; address@hidden;
> > address@hidden; address@hidden
> > Subject: [netCDF #ZCE-849683]: clarification on large file support
> >
> > Fabio,
> >
> > > Please, I need just one more clarification.
> > > I would like to use the netcdf library on windows 64 bit platform.
> > > In order to activate very large file support, is it sufficient to
> > > compile the library specifying a 64bit environment and activate the
> > > 64bit file offset. Doing so, does the size_t type automatically
> > > switch to a 64 bit unsigned integer variable in such a way not to
> > > limit memory and thus data set allocation. I am using your code and
> > > Visual studio 2008 and I'm specifing a 64 bit target environment.
> >
> > With a 64-bit development environment, there is no need to "activate
> > the 64bit file offset", and this depends on the off_t type, not size_t.
> >
> > All versions of netCDF since 3.6.0 include support for reading and
> > writing 64-bit offset netCDF files, whether compiled with a 32-bit or
> > 64-bit development environment, if the size of the C off_t type is at
> > least 8 bytes. The size of the off_t type for your 64-bit Windows
> > development environment is surely 8 bytes, but you can test this by
> > printing "sizeof(off_t)" in a small C program, and you might as well
> > also print the size of size_t to answer your other question:
> >
> > #include <sys/types.h>
> > #include <stdio.h>
> > int main(void) {
> > printf("Size of off_t is %d bytes.\n", sizeof(off_t)); printf("Size of
> > size_t is %d bytes.\n", sizeof(size_t)); return 0; }
> >
> > If either of these is printed as a value less than 8, make sure you
> > use whatever C compiler flags are needed for your compiler to specify
> > a 64-bit development environment.
> >
> > Note that there are still some limits to dataset size using the netCDF
> > 64-bit offset format, as described in the Users Guide and in the FAQ
> > on large file support, in particular the answers to the three questions:
> >
> > Have all netCDF size limits been eliminated?
> > Why are variables still limited in size?
> > How can I write variables larger than 4 GiB?
> >
> > http://www.unidata.ucar.edu/netcdf/docs/faq.html#Large%20File%20Suppor
> > t10
> >
> > As indicated in the answer to the second question above, the reason
> > variable sizes are still limited has to do with a desire to make sure
> > that netCDF 64-bit offset files files are still portable, even to 32-bit
> platforms.
> >
> > --Russ
> >
> > >
> > >
> > > -----Original Message-----
> > > From: Unidata netCDF Support
> > > [mailto:address@hidden]
> > > Sent: Monday, July 12, 2010 7:28 PM
> > > To: address@hidden
> > > Cc: address@hidden; address@hidden;
> > > address@hidden
> > > Subject: [netCDF #ZCE-849683]: clarification on large file support
> > >
> > > Fabio,
> > >
> > > A little more clarification is needed to my last reply. I said:
> > >
> > > Each variable in the file cannot exceed 4GB (not 2GB), in netCDF
> > > versions after 3.6.1, including the current netCDF 4.1.1. The actual
> > > maximum size of a variable on a 32-bit platform is (2^32 - 4) bytes.
> > > Part of the confusion is a documentation error here:
> > >
> > >
> > > http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Classic
> > > -L
> > > imitat
> > > ions
> > >
> > > which I just discovered hasn't been updated since the size limit on
> > > a single variable was changed from 2GB (2^31 - 4) to 4GB (2^32 - 4)
> > > in versions since netCDF 3.6.1.
> > >
> > > I was confused and the original documentation is correct. The
> > > netCDF classic format limits all but the last variable to 2GB in
> > > size. It is the 64-bit offset file format that permits all
> > > variables to be 4GB in size, and the last variable to be even
> > > larger. The netCDF-4/HDF5 format variant has no 4GB limits on the
> > > size of any variable. Sorry for
> > the confusion!
> > >
> > > --Russ
> > >
> > >
> > >
> > > Russ Rew UCAR Unidata Program
> > > address@hidden http://www.unidata.ucar.edu
> > >
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: ZCE-849683
> > > Department: Support netCDF
> > > Priority: Normal
> > > Status: Closed
> > >
> > >
> > >
> > >
> >
> > Russ Rew UCAR Unidata Program
> > address@hidden http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: ZCE-849683
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
>
> Russ Rew UCAR Unidata Program
> address@hidden http://www.unidata.ucar.edu
>
>
>
> Ticket Details
> ===================
> Ticket ID: ZCE-849683
> Department: Support netCDF
> Priority: Normal
> Status: Closed
>
>
Russ Rew UCAR Unidata Program
address@hidden http://www.unidata.ucar.edu
Ticket Details
===================
Ticket ID: ZCE-849683
Department: Support netCDF
Priority: Normal
Status: Closed