Hi John,
We may be hitting a file size limit in TDS 4.1. We've begun
publishing a number of climate model output NetCDF files that are in
excess of 3GB. When downloading these files from the TDS, we're
getting "shorter" files returned. For example a file of ~5GB results
in an file returned of ~680MB in size:
Actual file size: 4975605540 bytes
Returned file size: 680638244 bytes
This looks suspiciously like an issue related to the storage type and
possible int rollover as the actual file size exceeds Java's
Integer.MAX_VALUE and the resulting size is exactly the actual size
truncated to 32-bits.
I've looked at the headers returned from the HTTP request for the
file and the Content-Length reflects the "shortened" 680638244 byte
size. If I use an HTTP client like WGet and force the Content-Length
to be ignored, the resulting stream is still only 680638244 bytes in
length.
I reviewed the Thredds list and Googled this topic, but I have not
found anything specific as to this problem. Has this come up before?
We've typically been working with files of less the 2GB in the past,
and as far as I know, this is the first we've experienced this
problem. (We're running TDS 4.1.2) This will become a great need
for us once the AR5/CMIP5 datasets start flowing (which may be as
early as mid June!) These data files will likely be in the 2GB - 4GB
size range.
I do know the Java Servlet specification defines the setContentLength
parameter as an "int" which is a known limitation. This may be related.
Please let me know if you need further details.
Thanks,
-Eric