Re: TDS File Size limit?

Hi John,

I can reproduce this with a number of different files larger than 2Gb.

This is using HTTPServer (I have not tried OpenDAP.)

I'll check the logs for errors and get back to you.

Thanks,

-Eric

John Caron wrote:
> Eric Nienhouse wrote:
> > Hi John,
> >
> > We may be hitting a file size limit in TDS 4.1.  We've begun 
> > publishing a number of climate model output NetCDF files that are in 
> > excess of 3GB.  When downloading these files from the TDS, we're 
> > getting "shorter" files returned.  For example a file of ~5GB results 
> > in an file returned of ~680MB in size:
> >
> > Actual file size:  4975605540 bytes
> > Returned file size: 680638244 bytes
> >
> > This looks suspiciously like an issue related to the storage type and 
> > possible int rollover as the actual file size exceeds Java's 
> > Integer.MAX_VALUE and the resulting size is exactly the actual size 
> > truncated to 32-bits.
> >
> > I've looked at the headers returned from the HTTP request for the 
> > file and the Content-Length reflects the "shortened" 680638244 byte 
> > size.  If I use an HTTP client like WGet and force the Content-Length 
> > to be ignored, the resulting stream is still only 680638244 bytes in 
> > length.
> >
> > I reviewed the Thredds list and Googled this topic, but I have not 
> > found anything specific as to this problem.  Has this come up before?
> >
> > We've typically been working with files of less the 2GB in the past, 
> > and as far as I know, this is the first we've experienced this 
> > problem.  (We're running TDS 4.1.2)  This will become a great need 
> > for us once the AR5/CMIP5 datasets start flowing (which may be as 
> > early as mid June!)  These data files will likely be in the 2GB - 4GB 
> > size range.
> >
> > I do know the Java Servlet specification defines the setContentLength 
> > parameter as an "int" which is a known limitation.  This may be related.
> >
> > Please let me know if you need further details.
> >
> > Thanks,
> >
> > -Eric
>
> Hi Eric:
>
> "When downloading these files from the TDS" : 1) are you using 
> HTTPserver or Opendap? 2) is it reproducible? 3) can you look in the 
> threddsServlet.log to see if there's an error message?