Re: DODS, netCDF and Large File Support

Russ Rew wrote:

> > To: address@hidden, address@hidden, address@hidden, address@hidden,
> >       address@hidden
> > From: Rick Grubin <address@hidden>
> > Subject: Re: 20040707: DODS, netCDF and Large File Support
> > Organization: NCAR SCD
> > Keywords:200407071325.i67DPXaW025904 dods lfs linux
> >    
>
> Hi Rick,
>
>  
>
> > [All of the following refers to Linux hosts, particularly RH9 and FC1]
> >
> > I have a client application (NCL http://ngwww.ucar.edu/ncl) which uses
> > netCDF as its base file structure.  
> >
> > Earlier this year, I was able to successfully integrate DODS client
> > libraries into NCL, giving users access to remote files.
> >
> > Recently, users have been asking for large file support ("lfs", >2Gb)
> > within NCL.  I've also been able to integrate this support.
> >
> > In each of the two cases noted above, the feature in question (DODS and 
> > lfs) was integrated independent of the other, and each worked/works well.
> >
> > When trying to build both into my client application at the same time, I
> > note that DODS continues to work as expected, but access to large files
> > fails.
> >
> > My netCDF libraries (v3.5.1) are built with lfs, as are my DODS libraries
> > (v3.4.3).  By "built" I mean I compile with:
> >
> >    -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
> >
> > in order to 'turn on' lfs on a 32-bit host.  When I try to access a large
> > file, I see the following message:
> >
> >    lnetcdf/lposixio.c:269: px_get: Assertion `offset >= 0' failed.
> >
> > I suspect that this is a file size error, and for some reason a file size
> > is a negative number.  I expect I'll build the netCDF libs with debugging
> > and trace this through.
> >
> > My questions:
> >
> > 1.  Are DODS and lfs mutually exclusive?
> > 1a. Is lfs supported with DODS file access?
> > 2.  If 'no' to #1/#1a, what (obvious?) item am I overlooking?
> >    
>
> I think DODS and LFS should work together.  Charlie Zender's NCO
> package
>
>  http://nco.sourceforge.net/
>
> claims to support both LFS and NCO, though the LFS support has just
> been added recently.  The LFS support in netCDF 3.5.1 is somewhat
> limited (only one non-record variable over 2 GB allowed, etc.).  We
> are adding better LFS support to netCDF 3.6.0, which will be released
> later this month.  
>
> None of this explains the bug you are seeing, which I haven't seen
> before.  The assertion in lposixio.c/px_get is from DODS code rather
> than the netCDF library, since netCDF has know "lposixio.c" file.
>
> If you discover more by tracing this, please let us know what you
> find. 
>
> --Russ
>
>  
Hi all,

I'm CCing James Gallagher on this to get another OPeNDAP/DODS perspective.

The netcdf file libsrc/posixio.c is used as the OPeNDAP file 
nc3-dods/lnetcdf/posixio.c with very few changes. The assertion that 
fails above is the same in both versions of the file and none of the 
other changes jump out at me as being likely to affect this assertion. 
However, I haven't looked at any differences in other files between the 
netCDF code and the code as it is in OPeNDAP.

James, any thoughts on OPeNDAP and LFS?

Ethan

--
Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    address@hidden
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
---------------------------------------------------------------------------