Re: Netcdf

Hi Jeremy,

> Good morning, John Dutton and I are doing some work with WRF and we've 
> come across some issues with very large netcdf files.  He told me to 
> send you a message since you are the netcdf guru. 
> 
> I have a netcdf output file from WRF that is close to 20GB in size.  The 
> problem I have is that many of the programs I use to extract information 
> from this file are having issues accessing the data.  I get errors like:
> 
> posixio.c:309: px_rel: Assertion `pxp->bf_offset <= offset && offset < 
> pxp->bf_offset + (off_t) pxp->bf_extent' failed.
> 
> Do you have any idea of what could be causing this problem?  I am using 
> the libraries from Netcdf version 3.6.1.  I read that beginning with 
> 3.6.0 that files greater than 4GB can be read/written to, and my 
> programs had no problem with a file 4.6GB in size.   I'd appreciate it 
> if you could give me some general info on whether there is a limitation 
> with Netcdf or if I need to modify the software I am using to access the 
> data in the file.  

I don't know what could be causing this problem, but I'd be interested
in tracking it down.  As far as I know, anything that works for files
greater than 4,294,967,296 bytes in size should work fine for files up
to the maximum the file system supports.  Most of what I know about
large files I've put into the FAQ on Large File Support:

  
http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#Large%20File%20Support

There have been a few minor fixes since 3.6.1 that we have
incorporated into a 3.6.2-beta release that you could try, but I
expect it won't fix the problem you describe:

  ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-3.6.2-beta2.tar.gz

Looking at the assertion failure you got, it would be useful to know
the values of pxp->bf_extent (is it negative? this should never
happen), and of pxp->bf_offset and offset.  I don't know how easy it
would be for you to determine these values in a debugger, but that
would certainly help in determining what is going on.

I don't think we ever tested netCDF with files bigger than about 12
GBytes, because we thought the 4.29 GByte limit was the only important
threshold.  You may have discovered a bug for larger file sizes, and
if so, I'd like to diagnose it and fix it.  

Having us run WRF to produce an output file is probably not a
practical way to find the bug.  Assuming there is a bug, we don't know
whether it's in the writing of the files by WRF or in reading the
files later.  We may have to ask you to make one of these 20GB files
available for examination, to see whether we can determine whether the
file is corrupt and if so, why.

Do you know which version of the netCDF library is in use by the WRF
model producing the 20 GB outputs?

Is it convenient to try building and linking either your reading
programs or your WRF model against the netcdf-3.6.2-beta2 release
referenced above?

Could you make an example 20 GByte file that causes the failure
available for us to look at?  (I may have to clear some space to store
it, and I could imagine it might take a long time to transfer over the
network.)

--Russ