[THREDDS #SYD-491615]: TDS java exception "java.lang.NegativeArraySizeException" for large netcdf

Great!  I also just heard back from one of our NetCDF gurus, recently 
back from EGU.  He took a look at the sample file I have, and determined 
that it has characteristics of HDF4.  At the least, he couldn't read it 
using "pure" hdf5 libraries, but had to recompile ncdump including hdf4 
libraries.  I don't know if that is a surprise for you, but I thought 
I'd pass it along.  He was convinced that it was a NetCDF-4 file 
generally, though.

-Lansing

On 5/5/2014 11:18 AM, Hegewisch, Katherine (address@hidden) wrote:
> New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" 
> for large netcdf
>
> These should not be characters, but should be floating point. That is a 
> mistake on my part.
> I can fix this.
>
> Katherine
> ________________________________________
> From: Unidata THREDDS Support <address@hidden>
> Sent: Monday, May 5, 2014 10:15 AM
> To: Sheneman, Lucas (address@hidden)
> Cc: address@hidden; Sheneman, Lucas (address@hidden); Hegewisch, Katherine 
> (address@hidden); Vollmer, David (address@hidden)
> Subject: [THREDDS #SYD-491615]: TDS java exception 
> "java.lang.NegativeArraySizeException" for large netcdf
>
> Luke,
>
> Have you had any luck opening the file using ToolsUI?  Also, in my
> poking around, I ran into an issue in that there are global attributes
> that look like they should be floating point data types (double, I
> think), but are actually character strings:
>
> :geospatial_lat_min = "25.1562";
>     :geospatial_lat_max = "52.8438";
>     :geospatial_lon_min = "-124.5938";
>     :geospatial_lon_max = "-67.0312";
>
> I'll have to figure out a way to manage it better, but is there some
> reason these are being created as chars?
>
> -Lansing
>
> On 4/30/2014 3:19 PM, Luke Sheneman wrote:
>> New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" 
>> for large netcdf
>>
>> Okay, thanks Lansing.   We will take a look using the TooksUI.   If your
>> dev environment running TDS 4.3.21 but it can successfully serve these
>> larger netcdf files, then it is definitely a configuration issue on our
>> end.   It sounds like you can serve these files on your debug TDS server.
>>
>> I can open and view the larger netcdfs using Panoply.
>>
>> Your screen grab looks right.  That "bleeding over" into Canada is
>> intentional to capture portions of the Columbia River basin that reside
>> in Canada.
>>
>> Which version of Java are you using in your test environment?  Are you
>> using Oracle or OpenJDK java?
>>
>> We are using Oracle, 64-bit:
>>
>> [root@thredds1 sheneman]# /usr/java/jdk1.7.0_51/bin/java -version
>> java version "1.7.0_51"
>> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
>>
>> Which version of Tomcat?   We are using Tomcat 6.0.24
>>
>> Hmmm...
>>
>>
>> Thanks,
>>     -Luke
>>
>>
>> --
>>
>> Luke Sheneman, Ph.D
>> Technology and Data Services Manager
>> Northwest Knowledge Network (NKN) / University of Idaho
>> http://www.northwestknowledge.net
>>
>> address@hidden
>> Office: 208.885.4228   Mobile: 208.669.2248
>>
>>
>> On 4/30/14, 1:43 PM, Unidata THREDDS Support wrote:
>>> Luke,
>>>
>>> So here's what I've done to try to capture your problem:
>>>
>>> 1) I've been running our current development branch out of my debugger,
>>> serving one of the files I grabbed.  The development branch is a little
>>> ahead of the stable release branch, so I also
>>> 2) have been running the latest stable release of the TDS (4.3.21) from
>>> the command line, using the same file and catalog.
>>> 3) I've also been using both the 4.3.21 and current development branch
>>> of ToolsUI to look at the file and interact with the TDS in a live
>>> environment
>>>
>>> What I've found is that neither TDS generates a stack trace when serving
>>> up the large .nc file through OPeNDAP.  I can both look at ASCII (as
>>> long as I restrict the request) and download a .dods file.  Using
>>> ToolsUI, I can hit the TDS remotely and interact with the dataset.  To
>>> that end, I set up a viewer to scroll through time and make a movie of
>>> the data - looks like continental US plus a little something bleeding
>>> over into Canada near the WA/ID/MT border (see attached screen grab).
>>>
>>> Do you have ToolsUI?  If not, you should grab the jar file here:
>>>
>>> https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp
>>>
>>> It's a handy file exploration tool.  You can run it from the command
>>> line using:
>>>
>>> java -Xmx1g -jar toolsui-4.3.jar
>>>
>>> Once you have it running, you can navigate to either of two tabs to take
>>> a quick look at the file.  The first is the NCDump tab, the second is
>>> the Iosp/HDF5/HDF5-Objects tab.  The NCDump output will be obvious.
>>> When you open under HDF5, selecting anything in the upper window will
>>> cause the lower window to populate with relevant information.
>>>
>>> If the file comes up fine there, then try the THREDDS tab.  In the
>>> catalog URL entry field, enter the address of your thredds server,
>>> substituting .xml for .html.  For instance, it looks like this for me
>>> (attached). Clicking the Show Dataset button should take you to another
>>> pane under FeatureTypes/Grids.  From there, you can select the Daily
>>> Mean Near-Surface Wind and click the RedRaw, the little red martian in
>>> the upper right.  A graphics display will come up, empty, and if you
>>> click RedRaw again, you should see the data populating the screen...
>>>
>>> On the other hand, if something fails quickly, then I'm going to have to
>>> fall back on the configuration issue, looking hard at 32 vs. 64 bit
>>> libraries somewhere.
>>>
>>> By the way, what are the other netcdf tools you mentioned you were using
>>> that had no problems looking at the files?  As an aside, Matlab I think
>>> uses the netcdf-c libraries, and it might be good to make sure it is
>>> using a recent set.
>>>
>>> Fortunately, if this is a NetCDF issue, I have some friends around the
>>> corner who can help us out.
>>>
>>> -Lansing
>>>
>>>
>>> On 4/30/2014 7:27 AM, Luke Sheneman wrote:
>>>> New Client Reply: TDS java exception 
>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>
>>>> Katherine,
>>>>
>>>> Let's see what Lansing says.   THREDDs should be able to handle these
>>>> files.  The fact that it seems to break when these files are large
>>>> indicates either a serious bug in THREDDs or a problem with our
>>>> operating environment.   I am curious if unidata sees this java
>>>> exception when they try to serve these files on their TDS
>>>> installations?   If so, then it is most likely a bug with TDS itself.
>>>> That is my expectation, and should be really easy to test.   I know that
>>>> Lansing already downloaded some of the netcdfs.
>>>>
>>>> -Luke
>>>>
>>>> --
>>>>
>>>> Luke Sheneman, Ph.D
>>>> Technology and Data Services Manager
>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>> http://www.northwestknowledge.net
>>>>
>>>> address@hidden
>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>
>>>>
>>>> On 4/29/14, 10:10 AM, Hegewisch, Katherine (address@hidden) wrote:
>>>>> Luke and Lansing,
>>>>>
>>>>> Yes, these netcdf files are for the years 1950 - 2100. So those years 
>>>>> 2086-2100 you saw in the calendar dates are correct. The years are also 
>>>>> in the file names. These are future climate runs.
>>>>>
>>>>> Yes these are netcdf4 files. I generate them in MATLAB using
>>>>> ncid = netcdf.create(FULLFILENAME,'NETCDF4');
>>>>>
>>>>> Yes these files are compressed. I compress them in MATLAB using
>>>>>        netcdf.defVarDeflate(ncid,dataID,true,true,5);
>>>>> (the compression is a 5 out of 10... so this should be about 1/2 
>>>>> compression).
>>>>>
>>>>> I wonder if you need me to make smaller files by using smaller ranges of 
>>>>> years.
>>>>> Luke - would you like me to make an uncompressed file to see if that 
>>>>> works better with THREDDS?
>>>>>
>>>>> Katherine
>>>>> ________________________________________
>>>>> From: Sheneman, Lucas (address@hidden)
>>>>> Sent: Monday, April 28, 2014 7:22 PM
>>>>> To: address@hidden
>>>>> Cc: Hegewisch, Katherine (address@hidden)
>>>>> Subject: Re: [THREDDS #SYD-491615]: TDS java exception 
>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>
>>>>> Lansing and Katherine -
>>>>>
>>>>> Katherine, can you address Lansing's questions about your netcdf
>>>>> files?   We are trying to debug thredds.   It works on your smaller
>>>>> files, but some things fail on your larger files (>2GB).
>>>>>
>>>>> Please copy me on your correspondence...
>>>>>
>>>>> Thanks,
>>>>>        -Luke
>>>>>
>>>>> --
>>>>>
>>>>> Luke Sheneman, Ph.D
>>>>> Technology and Data Services Manager
>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>> http://www.northwestknowledge.net
>>>>>
>>>>> address@hidden
>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>
>>>>>
>>>>> On 4/26/14, 8:15 AM, Unidata THREDDS Support wrote:
>>>>>> Compressed how?  I can open the file in ToolsUI, another netCDF-Java
>>>>>> interface.  If they are compressed somehow, is the 3.1G the compressed
>>>>>> or uncompressed size?  Are they actually netcdf4 files?
>>>>>>
>>>>>> -Lansing
>>>>>>
>>>>>> On 4/25/2014 6:10 PM, Luke Sheneman wrote:
>>>>>>> New Client Reply: TDS java exception 
>>>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>>>
>>>>>>> Thanks Lansing.
>>>>>>>
>>>>>>> I believe the netcdfs are compressed.   Perhaps that is playing a role 
>>>>>>> here?
>>>>>>>
>>>>>>> I look forward to seeing what you come up with.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>          -Luke
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Luke Sheneman, Ph.D
>>>>>>> Technology and Data Services Manager
>>>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>>>> http://www.northwestknowledge.net
>>>>>>>
>>>>>>> address@hidden
>>>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>>>
>>>>>>>
>>>>>>> On 4/25/14, 3:10 PM, Unidata THREDDS Support wrote:
>>>>>>>> Luke,
>>>>>>>>
>>>>>>>> I finally got my system back online today.  It's not clear what
>>>>>>>> happened, but I had to wipe out and reinstall my standard toolkit
>>>>>>>> programs.  I fired up a catalog with your file in it, which is 3.1G in
>>>>>>>> size.  I noticed that trying to download the entire dataset through the
>>>>>>>> ncss service failed because the maximum size limit in threddsConfig.xml
>>>>>>>> is set at the default 52428800 (50 Megabytes).  I upped the maximum 
>>>>>>>> size
>>>>>>>> by editing this parameter in threddsConfig:
>>>>>>>>
>>>>>>>>            <NetcdfSubsetService>
>>>>>>>>              <allow>true</allow>
>>>>>>>>              <scour>15 min</scour>
>>>>>>>>              <maxAge>30 min</maxAge>
>>>>>>>> <!--maxFileDownloadSize>2097152</maxFileDownloadSize-->
>>>>>>>> <!--maxFileDownloadSize>52428800</maxFileDownloadSize-->
>>>>>>>> <maxFileDownloadSize>10552428800</maxFileDownloadSize>
>>>>>>>>            </NetcdfSubsetService>
>>>>>>>>
>>>>>>>> Then I realized that somehow, the file is being seen as much larger 
>>>>>>>> than
>>>>>>>> 3.1G, which is an odd behavior...I shouldn't have to specify 10G to
>>>>>>>> cover a 3.1G file.  So, I'll have to dig and see what's going on.
>>>>>>>>
>>>>>>>> Have a good weekend,
>>>>>>>>            Lansing
>>>>>>>>
>>>>>>>> On 4/11/2014 5:31 PM, Luke Sheneman wrote:
>>>>>>>>> New Client Reply: TDS java exception 
>>>>>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>>>>>
>>>>>>>>> Thanks Lansing -
>>>>>>>>>
>>>>>>>>> I've attached the latest log files.
>>>>>>>>>
>>>>>>>>> I am somewhat familiar with the data.   These are downscaled climate
>>>>>>>>> model outputs.  Some of the data are historical and some are
>>>>>>>>> predictive.   The one you were looking at is predicting climate
>>>>>>>>> scenarios for periods in the future (the years 2086 through 2100).   I
>>>>>>>>> don't see anything weird with those dates in the data that you were
>>>>>>>>> looking at...
>>>>>>>>>
>>>>>>>>> Have a great weekend, and I look forward to working with you soon to 
>>>>>>>>> get
>>>>>>>>> the java exception.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>            -Luke
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Luke Sheneman, Ph.D
>>>>>>>>> Technology and Data Services Manager
>>>>>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>>>>>> http://www.northwestknowledge.net
>>>>>>>>>
>>>>>>>>> address@hidden
>>>>>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 4/11/14, 3:58 PM, Unidata THREDDS Support wrote:
>>>>>>>>>> baby steps...
>>>>>>>>>>
>>>>>>>>>> When you get a moment, could you repeat the log capture procedure?  
>>>>>>>>>> I'm about to head out for the day, but I'll pull this up again on 
>>>>>>>>>> Monday.  I looked at one of the files already, which I downloaded 
>>>>>>>>>> using the http server.  Are you familiar with the data, by chance?  
>>>>>>>>>> I notice that the calendar dates (Gregorian, days since 1901-01-01) 
>>>>>>>>>> are for 2086-2100.  Is this right, or is something else fishy here?
>>>>>>>>>>
>>>>>>>>>> Have a good weekend!
>>>>>>>>>>
>>>>>>>>>> -Lansing
>>>>>>>>>>
>>>>>>>>>> Ticket Details
>>>>>>>>>> ===================
>>>>>>>>>> Ticket ID: SYD-491615
>>>>>>>>>> Department: Support THREDDS
>>>>>>>>>> Priority: Normal
>>>>>>>>>> Status: Open
>>>>>>>>>>
>>>>>>>>> Ticket Details
>>>>>>>>> ===================
>>>>>>>>> Ticket ID: SYD-491615
>>>>>>>>> Department: Support THREDDS
>>>>>>>>> Priority: Normal
>>>>>>>>> Status: Open
>>>>>>>>> Link:  
>>>>>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>>>>>> Ticket Details
>>>>>>>> ===================
>>>>>>>> Ticket ID: SYD-491615
>>>>>>>> Department: Support THREDDS
>>>>>>>> Priority: High
>>>>>>>> Status: Open
>>>>>>>>
>>>>>>> Ticket Details
>>>>>>> ===================
>>>>>>> Ticket ID: SYD-491615
>>>>>>> Department: Support THREDDS
>>>>>>> Priority: High
>>>>>>> Status: Open
>>>>>>> Link:  
>>>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>>>> Ticket Details
>>>>>> ===================
>>>>>> Ticket ID: SYD-491615
>>>>>> Department: Support THREDDS
>>>>>> Priority: High
>>>>>> Status: Open
>>>>>>
>>>> Ticket Details
>>>> ===================
>>>> Ticket ID: SYD-491615
>>>> Department: Support THREDDS
>>>> Priority: High
>>>> Status: Open
>>>> Link:  
>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>
>>> Ticket Details
>>> ===================
>>> Ticket ID: SYD-491615
>>> Department: Support THREDDS
>>> Priority: High
>>> Status: Open
>>
>> Ticket Details
>> ===================
>> Ticket ID: SYD-491615
>> Department: Support THREDDS
>> Priority: High
>> Status: Open
>> Link:  
>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>
>
> Ticket Details
> ===================
> Ticket ID: SYD-491615
> Department: Support THREDDS
> Priority: High
> Status: Open
>
>
>
>
> Ticket Details
> ===================
> Ticket ID: SYD-491615
> Department: Support THREDDS
> Priority: High
> Status: Open
> Link:  
> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770



Ticket Details
===================
Ticket ID: SYD-491615
Department: Support THREDDS
Priority: High
Status: Open