[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: DODS, netCDF and Large File Support

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: DODS, netCDF and Large File Support
  • Date: Wed, 07 Jul 2004 09:06:46 -0600
>To: address@hidden, address@hidden, address@hidden, address@hidden,
>        address@hidden
>From: Rick Grubin <address@hidden>
>Subject: Re: 20040707: DODS, netCDF and Large File Support
>Organization: NCAR SCD
>Keywords:200407071325.i67DPXaW025904 dods lfs linux

Hi Rick,

> [All of the following refers to Linux hosts, particularly RH9 and FC1]
> 
> I have a client application (NCL http://ngwww.ucar.edu/ncl) which uses
> netCDF as its base file structure.  
> 
> Earlier this year, I was able to successfully integrate DODS client
> libraries into NCL, giving users access to remote files.
> 
> Recently, users have been asking for large file support ("lfs", >2Gb)
> within NCL.  I've also been able to integrate this support.
> 
> In each of the two cases noted above, the feature in question (DODS and 
> lfs) was integrated independent of the other, and each worked/works well.
> 
> When trying to build both into my client application at the same time, I
> note that DODS continues to work as expected, but access to large files
> fails.
> 
> My netCDF libraries (v3.5.1) are built with lfs, as are my DODS libraries
> (v3.4.3).  By "built" I mean I compile with:
> 
>     -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
> 
> in order to 'turn on' lfs on a 32-bit host.  When I try to access a large
> file, I see the following message:
> 
>     lnetcdf/lposixio.c:269: px_get: Assertion `offset >= 0' failed.
> 
> I suspect that this is a file size error, and for some reason a file size
> is a negative number.  I expect I'll build the netCDF libs with debugging
> and trace this through.
> 
> My questions:
> 
> 1.  Are DODS and lfs mutually exclusive?
> 1a. Is lfs supported with DODS file access?
> 2.  If 'no' to #1/#1a, what (obvious?) item am I overlooking?

I think DODS and LFS should work together.  Charlie Zender's NCO
package

  http://nco.sourceforge.net/

claims to support both LFS and NCO, though the LFS support has just
been added recently.  The LFS support in netCDF 3.5.1 is somewhat
limited (only one non-record variable over 2 GB allowed, etc.).  We
are adding better LFS support to netCDF 3.6.0, which will be released
later this month.  

None of this explains the bug you are seeing, which I haven't seen
before.  The assertion in lposixio.c/px_get is from DODS code rather
than the netCDF library, since netCDF has know "lposixio.c" file.

If you discover more by tracing this, please let us know what you
find. 

--Russ