[THREDDS #SYD-491615]: TDS java exception "java.lang.NegativeArraySizeException" for large netcdf

Katherine,

I'm not entirely sure what's going on, but apparently, the files you are 
creating are not recognized as being netcdf format.  For instance, look 
at this output:

madry@MENDAX /z/testdata/support/SYD-491615/data
$ ncdump -k macav2livneh_was_inmcm4_rcp85_2086_2100_CONUS.nc
c:\Program Files (x86)\NetCDF 4.3.0\bin\ncdump.exe: 
macav2livneh_was_inmcm4_rcp85_2086_2100_CONUS.nc: NetCDF: Unknown file 
format

madry@MENDAX /z/testdata/support/SYD-491615/data

That said, the command I'm using there is from the netcdf-c library 
(v4.3.0).  The netcdf-java library seems to do better with the file, as 
I can open it up in ToolsUI.  Both the netcdf and hdf5 file viewers can 
open the file.  I also was able to serve the file out of a TDS I'm 
running locally using an opendap request.

So, let me ask a couple of quick questions.  Are you running your TDS on 
a 64-bit machine?  Do you know if the java libraries and jre you are 
using are 64-bit?  It's easy to miss an errant 32-bit option, which, 
given the size of the file, will be a problem.  Once you get past 
2-something gigabytes, a 32-bit system won't manage it properly.

-Lansing

On 4/29/2014 11:10 AM, Hegewisch, Katherine (address@hidden) wrote:
> New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" 
> for large netcdf
>
> Luke and Lansing,
>
> Yes, these netcdf files are for the years 1950 - 2100. So those years 
> 2086-2100 you saw in the calendar dates are correct. The years are also in 
> the file names. These are future climate runs.
>
> Yes these are netcdf4 files. I generate them in MATLAB using
> ncid = netcdf.create(FULLFILENAME,'NETCDF4');
>
> Yes these files are compressed. I compress them in MATLAB using
>    netcdf.defVarDeflate(ncid,dataID,true,true,5);
> (the compression is a 5 out of 10... so this should be about 1/2 compression).
>
> I wonder if you need me to make smaller files by using smaller ranges of 
> years.
> Luke - would you like me to make an uncompressed file to see if that works 
> better with THREDDS?
>
> Katherine
> ________________________________________
> From: Sheneman, Lucas (address@hidden)
> Sent: Monday, April 28, 2014 7:22 PM
> To: address@hidden
> Cc: Hegewisch, Katherine (address@hidden)
> Subject: Re: [THREDDS #SYD-491615]: TDS java exception 
> "java.lang.NegativeArraySizeException" for large netcdf
>
> Lansing and Katherine -
>
> Katherine, can you address Lansing's questions about your netcdf
> files?   We are trying to debug thredds.   It works on your smaller
> files, but some things fail on your larger files (>2GB).
>
> Please copy me on your correspondence...
>
> Thanks,
>    -Luke
>
> --
>
> Luke Sheneman, Ph.D
> Technology and Data Services Manager
> Northwest Knowledge Network (NKN) / University of Idaho
> http://www.northwestknowledge.net
>
> address@hidden
> Office: 208.885.4228   Mobile: 208.669.2248
>
>
> On 4/26/14, 8:15 AM, Unidata THREDDS Support wrote:
>> Compressed how?  I can open the file in ToolsUI, another netCDF-Java
>> interface.  If they are compressed somehow, is the 3.1G the compressed
>> or uncompressed size?  Are they actually netcdf4 files?
>>
>> -Lansing
>>
>> On 4/25/2014 6:10 PM, Luke Sheneman wrote:
>>> New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" 
>>> for large netcdf
>>>
>>> Thanks Lansing.
>>>
>>> I believe the netcdfs are compressed.   Perhaps that is playing a role here?
>>>
>>> I look forward to seeing what you come up with.
>>>
>>> Thanks,
>>>      -Luke
>>>
>>> --
>>>
>>> Luke Sheneman, Ph.D
>>> Technology and Data Services Manager
>>> Northwest Knowledge Network (NKN) / University of Idaho
>>> http://www.northwestknowledge.net
>>>
>>> address@hidden
>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>
>>>
>>> On 4/25/14, 3:10 PM, Unidata THREDDS Support wrote:
>>>> Luke,
>>>>
>>>> I finally got my system back online today.  It's not clear what
>>>> happened, but I had to wipe out and reinstall my standard toolkit
>>>> programs.  I fired up a catalog with your file in it, which is 3.1G in
>>>> size.  I noticed that trying to download the entire dataset through the
>>>> ncss service failed because the maximum size limit in threddsConfig.xml
>>>> is set at the default 52428800 (50 Megabytes).  I upped the maximum size
>>>> by editing this parameter in threddsConfig:
>>>>
>>>>        <NetcdfSubsetService>
>>>>          <allow>true</allow>
>>>>          <scour>15 min</scour>
>>>>          <maxAge>30 min</maxAge>
>>>> <!--maxFileDownloadSize>2097152</maxFileDownloadSize-->
>>>> <!--maxFileDownloadSize>52428800</maxFileDownloadSize-->
>>>> <maxFileDownloadSize>10552428800</maxFileDownloadSize>
>>>>        </NetcdfSubsetService>
>>>>
>>>> Then I realized that somehow, the file is being seen as much larger than
>>>> 3.1G, which is an odd behavior...I shouldn't have to specify 10G to
>>>> cover a 3.1G file.  So, I'll have to dig and see what's going on.
>>>>
>>>> Have a good weekend,
>>>>        Lansing
>>>>
>>>> On 4/11/2014 5:31 PM, Luke Sheneman wrote:
>>>>> New Client Reply: TDS java exception 
>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>
>>>>> Thanks Lansing -
>>>>>
>>>>> I've attached the latest log files.
>>>>>
>>>>> I am somewhat familiar with the data.   These are downscaled climate
>>>>> model outputs.  Some of the data are historical and some are
>>>>> predictive.   The one you were looking at is predicting climate
>>>>> scenarios for periods in the future (the years 2086 through 2100).   I
>>>>> don't see anything weird with those dates in the data that you were
>>>>> looking at...
>>>>>
>>>>> Have a great weekend, and I look forward to working with you soon to get
>>>>> the java exception.
>>>>>
>>>>> Thanks,
>>>>>        -Luke
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Luke Sheneman, Ph.D
>>>>> Technology and Data Services Manager
>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>> http://www.northwestknowledge.net
>>>>>
>>>>> address@hidden
>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>
>>>>>
>>>>> On 4/11/14, 3:58 PM, Unidata THREDDS Support wrote:
>>>>>> baby steps...
>>>>>>
>>>>>> When you get a moment, could you repeat the log capture procedure?  I'm 
>>>>>> about to head out for the day, but I'll pull this up again on Monday.  I 
>>>>>> looked at one of the files already, which I downloaded using the http 
>>>>>> server.  Are you familiar with the data, by chance?  I notice that the 
>>>>>> calendar dates (Gregorian, days since 1901-01-01) are for 2086-2100.  Is 
>>>>>> this right, or is something else fishy here?
>>>>>>
>>>>>> Have a good weekend!
>>>>>>
>>>>>> -Lansing
>>>>>>
>>>>>> Ticket Details
>>>>>> ===================
>>>>>> Ticket ID: SYD-491615
>>>>>> Department: Support THREDDS
>>>>>> Priority: Normal
>>>>>> Status: Open
>>>>>>
>>>>> Ticket Details
>>>>> ===================
>>>>> Ticket ID: SYD-491615
>>>>> Department: Support THREDDS
>>>>> Priority: Normal
>>>>> Status: Open
>>>>> Link:  
>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>> Ticket Details
>>>> ===================
>>>> Ticket ID: SYD-491615
>>>> Department: Support THREDDS
>>>> Priority: High
>>>> Status: Open
>>>>
>>> Ticket Details
>>> ===================
>>> Ticket ID: SYD-491615
>>> Department: Support THREDDS
>>> Priority: High
>>> Status: Open
>>> Link:  
>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>
>> Ticket Details
>> ===================
>> Ticket ID: SYD-491615
>> Department: Support THREDDS
>> Priority: High
>> Status: Open
>>
>
>
> Ticket Details
> ===================
> Ticket ID: SYD-491615
> Department: Support THREDDS
> Priority: High
> Status: Open
> Link:  
> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770



Ticket Details
===================
Ticket ID: SYD-491615
Department: Support THREDDS
Priority: High
Status: Open