[netCDF #ANI-357083]: Errors writing large files

Good afternoon Mitch,

> I downloaded NetCDF-4.2.dev-NC3-32-shared.exe two days ago(2012-12-17)
> from http://www.unidata.ucar.edu/software/netcdf/win_netcdf/#binaries
> 
> That version doesn't crash when I try to create a large file, but it
> doesn't work either.
> 
> With the same source code as before, building the Release build...
> 
> - with NC_64BIT_OFFSET but not NC_SHARE, nc_enddef fails with error 22
> (Invalid argument). The resultant file is 2G (2,147,475,456 bytes).
> 
> - with NC_64BIT_OFFSET and NC_SHARE, it works OK up to 4G, but fails for
> larger files.
> 
> I get the same results building with Visual Studio 2005 on XP, or VS2010
> on Windows 7.
> 
> Any ideas when we might have a NetCDF for Windows that works correctly
> with large files (>4GB)
> 

I've spent most of this afternoon investigating this issue, and it appears to 
be related to how large file support is handled in 32-bit Windows.  Fixing this 
is going to take a little bit of planning, but in the meantime I can offer two 
solutions that worked with your test program on my Windows 7 machine.

1. Use the 64-bit libraries:  If you were to download and install 
NetCDF-4.2.dev-NC3-64-shared.exe from the Windows NetCDF website, you should be 
able to generate the larger file.  

2. Use the netCDF4 file format: If you were to download and install one of the 
netCDF-4 packages (NC4), you would be able to invoke nc_create with the 
NC_NETCDF4 flag; this worked for me in both 32 and 64 bit Windows environments. 
 

Thanks for your help bringing this to my attention; this is an issue which will 
need to be addressed to offer full large file support on Windows in both 32 and 
64 bit environments.

Have a great evening,

-Ward
> Thanks
> 
> Mitch Ames
> 
> From: Ames, Mitch
> Sent: 2012-11-30, Friday 17:07
> To: 'address@hidden'
> Subject: Errors writing large files
> 
> Further to my original e-mail below ...
> 
> If set NC_NOFILL after creating the file, eg:
> 
> err = nc_create("big.nc", NC_CLOBBER | NC_64BIT_OFFSET /*|
> NC_SHARE*/, &ncid);
> 
> CHECK("nc_create")
> 
> err = nc_set_fill(ncid, NC_NOFILL, NULL);
> 
> CHECK("nc_set_fill")
> 
> ...
> 
> nc_enddef() succeeds but the program crashes (with the same errors as
> before) when writing the data as the file gets up to 2GB or 4GB.
> 
> From: Ames, Mitch
> Sent: 2012-11-30, Friday 16:01
> To: 'address@hidden'
> Subject:
> 
> I'm running into problems creating large (> 4GB) NetCDF files.
> 
> I've downloaded nc3_32_dll.zip
> <http://www.unidata.ucar.edu/software/netcdf/win_netcdf/nc3_32_dll.zip>
> from http://www.unidata.ucar.edu/software/netcdf/win_netcdf/#binaries,
> and I'm building and running a Debug build.
> 
> Attached is the test code that I use to reproduce the problem.
> 
> In all cases I'm creating the file with NC_64BIT_OFFSET and three
> fixed-size variables, the first two of which are relatively small
> (<1MB), the third large (>4 GB). From my reading of the documentation,
> this should be supported. (There are no record variables.)
> 
> In all cases where it crashes, the program crashes in nc_enddef().
> 
> To change the size of the file created, I just change the size of
> x_columns and y_rows.
> 
> If I create the file without NC_SHARE, nc_enddef() crashes if the file
> exceeds 2GB, with these errors:
> 
> Microsoft Visual C++ Runtime Library
> 
> Assertion failed!
> 
> Program: c:\mitch\netcdf\largenetcdf\debug\LargeNetCDF.exe
> 
> File: ../../libsrc/posixio.c
> 
> Line: 363
> 
> pxp->bf_offset + (off_t) pxp->bf_extent
> 
> (then if that is ignored ...)
> 
> Microsoft Visual C++ Runtime Library
> 
> Assertion failed!
> 
> Program: c:\mitch\netcdf\largenetcdf\debug\LargeNetCDF.exe
> 
> File: ../../libsrc/posixio.c
> 
> Line: 365
> 
> Expression: pIf(fIsSet(rflags, RGN_MODIFIED),
> fIsSet(pxp->bf_rflags, RGN_WRITE))
> 
> (then if that is ignored an unhandled Win32 exception [3040]...)
> 
> With NC_SHARE it works up to 4GB but fails above that with:
> 
> Microsoft Visual C++ Runtime Library
> 
> Assertion failed!
> 
> Program: c:\mitch\netcdf\largenetcdf\debug\LargeNetCDF.exe
> 
> File: ../../libsrc/posixio.c
> 
> Line: 286
> 
> Expression: *posp == OFF_NONE || *posp == lseek(nciop->fd,
> 0, SEEK_CUR)
> 
> (then if that is ignored an unhandled Win32 exception [3120]...)
> 
> Similar results occur with the same source code and netcdf libraries,
> using
> 
> - Visual Studio 2005 SP2 on Windows XP SP3
> 
> - Visual Studio 2010 SP1 on Windows 7 SP1
> 
> I can't run the Release build at all - it simply crashes with an
> unhandled Win32 exception [2216].
> 
> I can't build with nc3_32_static, as explained in my e-mail of
> 2012-11-27 (your Ticket ID: IHX-989868)
> 
> Some help resolving this problem would be appreciated.
> 
> Thank-you
> 
> Mitch Ames
> 
> 
> 

Ticket Details
===================
Ticket ID: ANI-357083
Department: Support netCDF
Priority: Normal
Status: Open