[THREDDS #SYD-491615]: TDS java exception "java.lang.NegativeArraySizeException" for large netcdf

Well, don't put a nail in that coffin yet, I didn't like how the netcdf 
subset service (ncss) was handling the dataset...I foresee some work on 
that service.

Here's what I'm using:

apache-tomcat 7.0.53 (http://tomcat.apache.org/download-70.cgi)
jdk-7u55-windows-x64 
(http://www.oracle.com/technetwork/java/javase/downloads/jdk7-downloads-1880260.html)

madry@MENDAX ~/jarfiles/toolsui
$ java -version
java version "1.7.0_55"
Java(TM) SE Runtime Environment (build 1.7.0_55-b13)
Java HotSpot(TM) 64-Bit Server VM (build 24.55-b03, mixed mode)

-Lansing

On 4/30/2014 3:19 PM, Luke Sheneman wrote:
> New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" 
> for large netcdf
>
> Okay, thanks Lansing.   We will take a look using the TooksUI.   If your
> dev environment running TDS 4.3.21 but it can successfully serve these
> larger netcdf files, then it is definitely a configuration issue on our
> end.   It sounds like you can serve these files on your debug TDS server.
>
> I can open and view the larger netcdfs using Panoply.
>
> Your screen grab looks right.  That "bleeding over" into Canada is
> intentional to capture portions of the Columbia River basin that reside
> in Canada.
>
> Which version of Java are you using in your test environment?  Are you
> using Oracle or OpenJDK java?
>
> We are using Oracle, 64-bit:
>
> [root@thredds1 sheneman]# /usr/java/jdk1.7.0_51/bin/java -version
> java version "1.7.0_51"
> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
>
> Which version of Tomcat?   We are using Tomcat 6.0.24
>
> Hmmm...
>
>
> Thanks,
>    -Luke
>
>
> --
>
> Luke Sheneman, Ph.D
> Technology and Data Services Manager
> Northwest Knowledge Network (NKN) / University of Idaho
> http://www.northwestknowledge.net
>
> address@hidden
> Office: 208.885.4228   Mobile: 208.669.2248
>
>
> On 4/30/14, 1:43 PM, Unidata THREDDS Support wrote:
>> Luke,
>>
>> So here's what I've done to try to capture your problem:
>>
>> 1) I've been running our current development branch out of my debugger,
>> serving one of the files I grabbed.  The development branch is a little
>> ahead of the stable release branch, so I also
>> 2) have been running the latest stable release of the TDS (4.3.21) from
>> the command line, using the same file and catalog.
>> 3) I've also been using both the 4.3.21 and current development branch
>> of ToolsUI to look at the file and interact with the TDS in a live
>> environment
>>
>> What I've found is that neither TDS generates a stack trace when serving
>> up the large .nc file through OPeNDAP.  I can both look at ASCII (as
>> long as I restrict the request) and download a .dods file.  Using
>> ToolsUI, I can hit the TDS remotely and interact with the dataset.  To
>> that end, I set up a viewer to scroll through time and make a movie of
>> the data - looks like continental US plus a little something bleeding
>> over into Canada near the WA/ID/MT border (see attached screen grab).
>>
>> Do you have ToolsUI?  If not, you should grab the jar file here:
>>
>> https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp
>>
>> It's a handy file exploration tool.  You can run it from the command
>> line using:
>>
>> java -Xmx1g -jar toolsui-4.3.jar
>>
>> Once you have it running, you can navigate to either of two tabs to take
>> a quick look at the file.  The first is the NCDump tab, the second is
>> the Iosp/HDF5/HDF5-Objects tab.  The NCDump output will be obvious.
>> When you open under HDF5, selecting anything in the upper window will
>> cause the lower window to populate with relevant information.
>>
>> If the file comes up fine there, then try the THREDDS tab.  In the
>> catalog URL entry field, enter the address of your thredds server,
>> substituting .xml for .html.  For instance, it looks like this for me
>> (attached). Clicking the Show Dataset button should take you to another
>> pane under FeatureTypes/Grids.  From there, you can select the Daily
>> Mean Near-Surface Wind and click the RedRaw, the little red martian in
>> the upper right.  A graphics display will come up, empty, and if you
>> click RedRaw again, you should see the data populating the screen...
>>
>> On the other hand, if something fails quickly, then I'm going to have to
>> fall back on the configuration issue, looking hard at 32 vs. 64 bit
>> libraries somewhere.
>>
>> By the way, what are the other netcdf tools you mentioned you were using
>> that had no problems looking at the files?  As an aside, Matlab I think
>> uses the netcdf-c libraries, and it might be good to make sure it is
>> using a recent set.
>>
>> Fortunately, if this is a NetCDF issue, I have some friends around the
>> corner who can help us out.
>>
>> -Lansing
>>
>>
>> On 4/30/2014 7:27 AM, Luke Sheneman wrote:
>>> New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" 
>>> for large netcdf
>>>
>>> Katherine,
>>>
>>> Let's see what Lansing says.   THREDDs should be able to handle these
>>> files.  The fact that it seems to break when these files are large
>>> indicates either a serious bug in THREDDs or a problem with our
>>> operating environment.   I am curious if unidata sees this java
>>> exception when they try to serve these files on their TDS
>>> installations?   If so, then it is most likely a bug with TDS itself.
>>> That is my expectation, and should be really easy to test.   I know that
>>> Lansing already downloaded some of the netcdfs.
>>>
>>> -Luke
>>>
>>> --
>>>
>>> Luke Sheneman, Ph.D
>>> Technology and Data Services Manager
>>> Northwest Knowledge Network (NKN) / University of Idaho
>>> http://www.northwestknowledge.net
>>>
>>> address@hidden
>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>
>>>
>>> On 4/29/14, 10:10 AM, Hegewisch, Katherine (address@hidden) wrote:
>>>> Luke and Lansing,
>>>>
>>>> Yes, these netcdf files are for the years 1950 - 2100. So those years 
>>>> 2086-2100 you saw in the calendar dates are correct. The years are also in 
>>>> the file names. These are future climate runs.
>>>>
>>>> Yes these are netcdf4 files. I generate them in MATLAB using
>>>> ncid = netcdf.create(FULLFILENAME,'NETCDF4');
>>>>
>>>> Yes these files are compressed. I compress them in MATLAB using
>>>>       netcdf.defVarDeflate(ncid,dataID,true,true,5);
>>>> (the compression is a 5 out of 10... so this should be about 1/2 
>>>> compression).
>>>>
>>>> I wonder if you need me to make smaller files by using smaller ranges of 
>>>> years.
>>>> Luke - would you like me to make an uncompressed file to see if that works 
>>>> better with THREDDS?
>>>>
>>>> Katherine
>>>> ________________________________________
>>>> From: Sheneman, Lucas (address@hidden)
>>>> Sent: Monday, April 28, 2014 7:22 PM
>>>> To: address@hidden
>>>> Cc: Hegewisch, Katherine (address@hidden)
>>>> Subject: Re: [THREDDS #SYD-491615]: TDS java exception 
>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>
>>>> Lansing and Katherine -
>>>>
>>>> Katherine, can you address Lansing's questions about your netcdf
>>>> files?   We are trying to debug thredds.   It works on your smaller
>>>> files, but some things fail on your larger files (>2GB).
>>>>
>>>> Please copy me on your correspondence...
>>>>
>>>> Thanks,
>>>>       -Luke
>>>>
>>>> --
>>>>
>>>> Luke Sheneman, Ph.D
>>>> Technology and Data Services Manager
>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>> http://www.northwestknowledge.net
>>>>
>>>> address@hidden
>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>
>>>>
>>>> On 4/26/14, 8:15 AM, Unidata THREDDS Support wrote:
>>>>> Compressed how?  I can open the file in ToolsUI, another netCDF-Java
>>>>> interface.  If they are compressed somehow, is the 3.1G the compressed
>>>>> or uncompressed size?  Are they actually netcdf4 files?
>>>>>
>>>>> -Lansing
>>>>>
>>>>> On 4/25/2014 6:10 PM, Luke Sheneman wrote:
>>>>>> New Client Reply: TDS java exception 
>>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>>
>>>>>> Thanks Lansing.
>>>>>>
>>>>>> I believe the netcdfs are compressed.   Perhaps that is playing a role 
>>>>>> here?
>>>>>>
>>>>>> I look forward to seeing what you come up with.
>>>>>>
>>>>>> Thanks,
>>>>>>         -Luke
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Luke Sheneman, Ph.D
>>>>>> Technology and Data Services Manager
>>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>>> http://www.northwestknowledge.net
>>>>>>
>>>>>> address@hidden
>>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>>
>>>>>>
>>>>>> On 4/25/14, 3:10 PM, Unidata THREDDS Support wrote:
>>>>>>> Luke,
>>>>>>>
>>>>>>> I finally got my system back online today.  It's not clear what
>>>>>>> happened, but I had to wipe out and reinstall my standard toolkit
>>>>>>> programs.  I fired up a catalog with your file in it, which is 3.1G in
>>>>>>> size.  I noticed that trying to download the entire dataset through the
>>>>>>> ncss service failed because the maximum size limit in threddsConfig.xml
>>>>>>> is set at the default 52428800 (50 Megabytes).  I upped the maximum size
>>>>>>> by editing this parameter in threddsConfig:
>>>>>>>
>>>>>>>           <NetcdfSubsetService>
>>>>>>>             <allow>true</allow>
>>>>>>>             <scour>15 min</scour>
>>>>>>>             <maxAge>30 min</maxAge>
>>>>>>> <!--maxFileDownloadSize>2097152</maxFileDownloadSize-->
>>>>>>> <!--maxFileDownloadSize>52428800</maxFileDownloadSize-->
>>>>>>> <maxFileDownloadSize>10552428800</maxFileDownloadSize>
>>>>>>>           </NetcdfSubsetService>
>>>>>>>
>>>>>>> Then I realized that somehow, the file is being seen as much larger than
>>>>>>> 3.1G, which is an odd behavior...I shouldn't have to specify 10G to
>>>>>>> cover a 3.1G file.  So, I'll have to dig and see what's going on.
>>>>>>>
>>>>>>> Have a good weekend,
>>>>>>>           Lansing
>>>>>>>
>>>>>>> On 4/11/2014 5:31 PM, Luke Sheneman wrote:
>>>>>>>> New Client Reply: TDS java exception 
>>>>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>>>>
>>>>>>>> Thanks Lansing -
>>>>>>>>
>>>>>>>> I've attached the latest log files.
>>>>>>>>
>>>>>>>> I am somewhat familiar with the data.   These are downscaled climate
>>>>>>>> model outputs.  Some of the data are historical and some are
>>>>>>>> predictive.   The one you were looking at is predicting climate
>>>>>>>> scenarios for periods in the future (the years 2086 through 2100).   I
>>>>>>>> don't see anything weird with those dates in the data that you were
>>>>>>>> looking at...
>>>>>>>>
>>>>>>>> Have a great weekend, and I look forward to working with you soon to 
>>>>>>>> get
>>>>>>>> the java exception.
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>           -Luke
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>>
>>>>>>>> Luke Sheneman, Ph.D
>>>>>>>> Technology and Data Services Manager
>>>>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>>>>> http://www.northwestknowledge.net
>>>>>>>>
>>>>>>>> address@hidden
>>>>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>>>>
>>>>>>>>
>>>>>>>> On 4/11/14, 3:58 PM, Unidata THREDDS Support wrote:
>>>>>>>>> baby steps...
>>>>>>>>>
>>>>>>>>> When you get a moment, could you repeat the log capture procedure?  
>>>>>>>>> I'm about to head out for the day, but I'll pull this up again on 
>>>>>>>>> Monday.  I looked at one of the files already, which I downloaded 
>>>>>>>>> using the http server.  Are you familiar with the data, by chance?  I 
>>>>>>>>> notice that the calendar dates (Gregorian, days since 1901-01-01) are 
>>>>>>>>> for 2086-2100.  Is this right, or is something else fishy here?
>>>>>>>>>
>>>>>>>>> Have a good weekend!
>>>>>>>>>
>>>>>>>>> -Lansing
>>>>>>>>>
>>>>>>>>> Ticket Details
>>>>>>>>> ===================
>>>>>>>>> Ticket ID: SYD-491615
>>>>>>>>> Department: Support THREDDS
>>>>>>>>> Priority: Normal
>>>>>>>>> Status: Open
>>>>>>>>>
>>>>>>>> Ticket Details
>>>>>>>> ===================
>>>>>>>> Ticket ID: SYD-491615
>>>>>>>> Department: Support THREDDS
>>>>>>>> Priority: Normal
>>>>>>>> Status: Open
>>>>>>>> Link:  
>>>>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>>>>> Ticket Details
>>>>>>> ===================
>>>>>>> Ticket ID: SYD-491615
>>>>>>> Department: Support THREDDS
>>>>>>> Priority: High
>>>>>>> Status: Open
>>>>>>>
>>>>>> Ticket Details
>>>>>> ===================
>>>>>> Ticket ID: SYD-491615
>>>>>> Department: Support THREDDS
>>>>>> Priority: High
>>>>>> Status: Open
>>>>>> Link:  
>>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>>> Ticket Details
>>>>> ===================
>>>>> Ticket ID: SYD-491615
>>>>> Department: Support THREDDS
>>>>> Priority: High
>>>>> Status: Open
>>>>>
>>> Ticket Details
>>> ===================
>>> Ticket ID: SYD-491615
>>> Department: Support THREDDS
>>> Priority: High
>>> Status: Open
>>> Link:  
>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>
>>
>> Ticket Details
>> ===================
>> Ticket ID: SYD-491615
>> Department: Support THREDDS
>> Priority: High
>> Status: Open
>
>
> Ticket Details
> ===================
> Ticket ID: SYD-491615
> Department: Support THREDDS
> Priority: High
> Status: Open
> Link:  
> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770



Ticket Details
===================
Ticket ID: SYD-491615
Department: Support THREDDS
Priority: High
Status: Open