[THREDDS #SYD-491615]: TDS java exception "java.lang.NegativeArraySizeException" for large netcdf

Katherine et al.,

I went back to our netcdf folks, who now tell me that the file I have 
from you really is in HDF-5/Netcdf-4 format.  Best guess on the hiccup 
would be waving hands at possible different versions of the libraries 
being used to write and then read the files.  I don't know much about 
the different versions of MATLAB mentioned below, so I can't really say, 
but it looks like R2014A supports netcdf-4.1.3.

Another issue that we'll need to address has to do with a limitation in 
Java that the TDS isn't handling gracefully.  While the file I have is 
"only" 3 gigs or so, it is compressed.  When accessing the data, it is 
very easy to ask for a single variable array that exceeds the maximum 
java array size, which is limited to something like MAX_INT indices.

-Lansing

On 5/6/2014 3:35 PM, Hegewisch, Katherine (address@hidden) wrote:
> New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" 
> for large netcdf
>
> Luke and others,
>
> I use a couple of different versions of MATLAB to make these files: MATLAB 
> R2012a, R2013b, and R2013a.
>
> In each version, I use the same 2 commands for making these:
> %making the netcdf file
>   ncid = netcdf.create(FULLFILENAME,'NETCDF4');
> %compression
>   netcdf.defVarDeflate(ncid,dataID,true,true,5);
>
> Katherine
> ________________________________________
> From: Sheneman, Lucas (address@hidden)
> Sent: Tuesday, May 6, 2014 1:50 PM
> To: Hegewisch, Katherine (address@hidden); address@hidden
> Cc: Vollmer, David (address@hidden)
> Subject: Re: [THREDDS #SYD-491615]: TDS java exception 
> "java.lang.NegativeArraySizeException" for large netcdf
>
> Katherine,
>
> NetCDF4 and HDF5 are pretty much the same thing, and in fact netcdf4 is
> a subset of HDF5, as per unidata documentation:
>
> https://www.unidata.ucar.edu/software/netcdf/docs/netcdf/NetCDF_002d4-Format.html
>
> However if MATLAB is creating netcdf4 files with HDF4 (not HDF5)
> elements, that does point to a potential problem with how MATLAB creates
> netcdf4 files.
>
> Lansing, since these netcdf4 files were created entirely from MATLAB's
> netcdf4 file creation tools - and you guys have a better handle on what
> parts of the file required hdf4 libraries, perhaps Unidata could file a
> bug against MATLAB?
>
> Katherine, what specific version of MATLAB did you use to generate these
> netcdf4 files?
>
> Katherine, I can help you get our THREDDS configuration working with the
> new (smaller) files.   I will work on that this afternoon.  Have they
> been uploaded to NKN?
>
> Thanks,
>    -Luke
>
> --
>
>
> Luke Sheneman, Ph.D
> Technology and Data Services Manager
> Northwest Knowledge Network (NKN) / University of Idaho
> http://www.northwestknowledge.net
>
> address@hidden
> Office: 208.885.4228   Mobile: 208.669.2248
>
>
> On 5/6/14, 11:42 AM, Hegewisch, Katherine (address@hidden) wrote:
>> MATLAB claims that making netcdf files using NETCDF4 option
>> creates a   'NetCDF-4/HDF5 file'
>> http://www.mathworks.com/help/matlab/ref/netcdf.create.html
>> Should I contact MATLAB about this?
>>
>> Luke - if I remade my netcdf files so that they were 10 year ranges (i..e 
>> half the size they are now), would that help you in getting THREDDS going 
>> and all this working.. or are the other problems preventing this too?
>>
>> k
>> ________________________________________
>> From: Unidata THREDDS Support <address@hidden>
>> Sent: Monday, May 5, 2014 12:23 PM
>> To: Sheneman, Lucas (address@hidden)
>> Cc: address@hidden; Sheneman, Lucas (address@hidden); Hegewisch, Katherine 
>> (address@hidden); Vollmer, David (address@hidden)
>> Subject: [THREDDS #SYD-491615]: TDS java exception 
>> "java.lang.NegativeArraySizeException" for large netcdf
>>
>> Great!  I also just heard back from one of our NetCDF gurus, recently
>> back from EGU.  He took a look at the sample file I have, and determined
>> that it has characteristics of HDF4.  At the least, he couldn't read it
>> using "pure" hdf5 libraries, but had to recompile ncdump including hdf4
>> libraries.  I don't know if that is a surprise for you, but I thought
>> I'd pass it along.  He was convinced that it was a NetCDF-4 file
>> generally, though.
>>
>> -Lansing
>>
>> On 5/5/2014 11:18 AM, Hegewisch, Katherine (address@hidden) wrote:
>>> New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" 
>>> for large netcdf
>>>
>>> These should not be characters, but should be floating point. That is a 
>>> mistake on my part.
>>> I can fix this.
>>>
>>> Katherine
>>> ________________________________________
>>> From: Unidata THREDDS Support <address@hidden>
>>> Sent: Monday, May 5, 2014 10:15 AM
>>> To: Sheneman, Lucas (address@hidden)
>>> Cc: address@hidden; Sheneman, Lucas (address@hidden); Hegewisch, Katherine 
>>> (address@hidden); Vollmer, David (address@hidden)
>>> Subject: [THREDDS #SYD-491615]: TDS java exception 
>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>
>>> Luke,
>>>
>>> Have you had any luck opening the file using ToolsUI?  Also, in my
>>> poking around, I ran into an issue in that there are global attributes
>>> that look like they should be floating point data types (double, I
>>> think), but are actually character strings:
>>>
>>> :geospatial_lat_min = "25.1562";
>>>       :geospatial_lat_max = "52.8438";
>>>       :geospatial_lon_min = "-124.5938";
>>>       :geospatial_lon_max = "-67.0312";
>>>
>>> I'll have to figure out a way to manage it better, but is there some
>>> reason these are being created as chars?
>>>
>>> -Lansing
>>>
>>> On 4/30/2014 3:19 PM, Luke Sheneman wrote:
>>>> New Client Reply: TDS java exception 
>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>
>>>> Okay, thanks Lansing.   We will take a look using the TooksUI.   If your
>>>> dev environment running TDS 4.3.21 but it can successfully serve these
>>>> larger netcdf files, then it is definitely a configuration issue on our
>>>> end.   It sounds like you can serve these files on your debug TDS server.
>>>>
>>>> I can open and view the larger netcdfs using Panoply.
>>>>
>>>> Your screen grab looks right.  That "bleeding over" into Canada is
>>>> intentional to capture portions of the Columbia River basin that reside
>>>> in Canada.
>>>>
>>>> Which version of Java are you using in your test environment?  Are you
>>>> using Oracle or OpenJDK java?
>>>>
>>>> We are using Oracle, 64-bit:
>>>>
>>>> [root@thredds1 sheneman]# /usr/java/jdk1.7.0_51/bin/java -version
>>>> java version "1.7.0_51"
>>>> Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
>>>> Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode)
>>>>
>>>> Which version of Tomcat?   We are using Tomcat 6.0.24
>>>>
>>>> Hmmm...
>>>>
>>>>
>>>> Thanks,
>>>>       -Luke
>>>>
>>>>
>>>> --
>>>>
>>>> Luke Sheneman, Ph.D
>>>> Technology and Data Services Manager
>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>> http://www.northwestknowledge.net
>>>>
>>>> address@hidden
>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>
>>>>
>>>> On 4/30/14, 1:43 PM, Unidata THREDDS Support wrote:
>>>>> Luke,
>>>>>
>>>>> So here's what I've done to try to capture your problem:
>>>>>
>>>>> 1) I've been running our current development branch out of my debugger,
>>>>> serving one of the files I grabbed.  The development branch is a little
>>>>> ahead of the stable release branch, so I also
>>>>> 2) have been running the latest stable release of the TDS (4.3.21) from
>>>>> the command line, using the same file and catalog.
>>>>> 3) I've also been using both the 4.3.21 and current development branch
>>>>> of ToolsUI to look at the file and interact with the TDS in a live
>>>>> environment
>>>>>
>>>>> What I've found is that neither TDS generates a stack trace when serving
>>>>> up the large .nc file through OPeNDAP.  I can both look at ASCII (as
>>>>> long as I restrict the request) and download a .dods file.  Using
>>>>> ToolsUI, I can hit the TDS remotely and interact with the dataset.  To
>>>>> that end, I set up a viewer to scroll through time and make a movie of
>>>>> the data - looks like continental US plus a little something bleeding
>>>>> over into Canada near the WA/ID/MT border (see attached screen grab).
>>>>>
>>>>> Do you have ToolsUI?  If not, you should grab the jar file here:
>>>>>
>>>>> https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp
>>>>>
>>>>> It's a handy file exploration tool.  You can run it from the command
>>>>> line using:
>>>>>
>>>>> java -Xmx1g -jar toolsui-4.3.jar
>>>>>
>>>>> Once you have it running, you can navigate to either of two tabs to take
>>>>> a quick look at the file.  The first is the NCDump tab, the second is
>>>>> the Iosp/HDF5/HDF5-Objects tab.  The NCDump output will be obvious.
>>>>> When you open under HDF5, selecting anything in the upper window will
>>>>> cause the lower window to populate with relevant information.
>>>>>
>>>>> If the file comes up fine there, then try the THREDDS tab.  In the
>>>>> catalog URL entry field, enter the address of your thredds server,
>>>>> substituting .xml for .html.  For instance, it looks like this for me
>>>>> (attached). Clicking the Show Dataset button should take you to another
>>>>> pane under FeatureTypes/Grids.  From there, you can select the Daily
>>>>> Mean Near-Surface Wind and click the RedRaw, the little red martian in
>>>>> the upper right.  A graphics display will come up, empty, and if you
>>>>> click RedRaw again, you should see the data populating the screen...
>>>>>
>>>>> On the other hand, if something fails quickly, then I'm going to have to
>>>>> fall back on the configuration issue, looking hard at 32 vs. 64 bit
>>>>> libraries somewhere.
>>>>>
>>>>> By the way, what are the other netcdf tools you mentioned you were using
>>>>> that had no problems looking at the files?  As an aside, Matlab I think
>>>>> uses the netcdf-c libraries, and it might be good to make sure it is
>>>>> using a recent set.
>>>>>
>>>>> Fortunately, if this is a NetCDF issue, I have some friends around the
>>>>> corner who can help us out.
>>>>>
>>>>> -Lansing
>>>>>
>>>>>
>>>>> On 4/30/2014 7:27 AM, Luke Sheneman wrote:
>>>>>> New Client Reply: TDS java exception 
>>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>>
>>>>>> Katherine,
>>>>>>
>>>>>> Let's see what Lansing says.   THREDDs should be able to handle these
>>>>>> files.  The fact that it seems to break when these files are large
>>>>>> indicates either a serious bug in THREDDs or a problem with our
>>>>>> operating environment.   I am curious if unidata sees this java
>>>>>> exception when they try to serve these files on their TDS
>>>>>> installations?   If so, then it is most likely a bug with TDS itself.
>>>>>> That is my expectation, and should be really easy to test.   I know that
>>>>>> Lansing already downloaded some of the netcdfs.
>>>>>>
>>>>>> -Luke
>>>>>>
>>>>>> --
>>>>>>
>>>>>> Luke Sheneman, Ph.D
>>>>>> Technology and Data Services Manager
>>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>>> http://www.northwestknowledge.net
>>>>>>
>>>>>> address@hidden
>>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>>
>>>>>>
>>>>>> On 4/29/14, 10:10 AM, Hegewisch, Katherine (address@hidden) wrote:
>>>>>>> Luke and Lansing,
>>>>>>>
>>>>>>> Yes, these netcdf files are for the years 1950 - 2100. So those years 
>>>>>>> 2086-2100 you saw in the calendar dates are correct. The years are also 
>>>>>>> in the file names. These are future climate runs.
>>>>>>>
>>>>>>> Yes these are netcdf4 files. I generate them in MATLAB using
>>>>>>> ncid = netcdf.create(FULLFILENAME,'NETCDF4');
>>>>>>>
>>>>>>> Yes these files are compressed. I compress them in MATLAB using
>>>>>>>          netcdf.defVarDeflate(ncid,dataID,true,true,5);
>>>>>>> (the compression is a 5 out of 10... so this should be about 1/2 
>>>>>>> compression).
>>>>>>>
>>>>>>> I wonder if you need me to make smaller files by using smaller ranges 
>>>>>>> of years.
>>>>>>> Luke - would you like me to make an uncompressed file to see if that 
>>>>>>> works better with THREDDS?
>>>>>>>
>>>>>>> Katherine
>>>>>>> ________________________________________
>>>>>>> From: Sheneman, Lucas (address@hidden)
>>>>>>> Sent: Monday, April 28, 2014 7:22 PM
>>>>>>> To: address@hidden
>>>>>>> Cc: Hegewisch, Katherine (address@hidden)
>>>>>>> Subject: Re: [THREDDS #SYD-491615]: TDS java exception 
>>>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>>>
>>>>>>> Lansing and Katherine -
>>>>>>>
>>>>>>> Katherine, can you address Lansing's questions about your netcdf
>>>>>>> files?   We are trying to debug thredds.   It works on your smaller
>>>>>>> files, but some things fail on your larger files (>2GB).
>>>>>>>
>>>>>>> Please copy me on your correspondence...
>>>>>>>
>>>>>>> Thanks,
>>>>>>>          -Luke
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Luke Sheneman, Ph.D
>>>>>>> Technology and Data Services Manager
>>>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>>>> http://www.northwestknowledge.net
>>>>>>>
>>>>>>> address@hidden
>>>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>>>
>>>>>>>
>>>>>>> On 4/26/14, 8:15 AM, Unidata THREDDS Support wrote:
>>>>>>>> Compressed how?  I can open the file in ToolsUI, another netCDF-Java
>>>>>>>> interface.  If they are compressed somehow, is the 3.1G the compressed
>>>>>>>> or uncompressed size?  Are they actually netcdf4 files?
>>>>>>>>
>>>>>>>> -Lansing
>>>>>>>>
>>>>>>>> On 4/25/2014 6:10 PM, Luke Sheneman wrote:
>>>>>>>>> New Client Reply: TDS java exception 
>>>>>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>>>>>
>>>>>>>>> Thanks Lansing.
>>>>>>>>>
>>>>>>>>> I believe the netcdfs are compressed.   Perhaps that is playing a 
>>>>>>>>> role here?
>>>>>>>>>
>>>>>>>>> I look forward to seeing what you come up with.
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>            -Luke
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>>
>>>>>>>>> Luke Sheneman, Ph.D
>>>>>>>>> Technology and Data Services Manager
>>>>>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>>>>>> http://www.northwestknowledge.net
>>>>>>>>>
>>>>>>>>> address@hidden
>>>>>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 4/25/14, 3:10 PM, Unidata THREDDS Support wrote:
>>>>>>>>>> Luke,
>>>>>>>>>>
>>>>>>>>>> I finally got my system back online today.  It's not clear what
>>>>>>>>>> happened, but I had to wipe out and reinstall my standard toolkit
>>>>>>>>>> programs.  I fired up a catalog with your file in it, which is 3.1G 
>>>>>>>>>> in
>>>>>>>>>> size.  I noticed that trying to download the entire dataset through 
>>>>>>>>>> the
>>>>>>>>>> ncss service failed because the maximum size limit in 
>>>>>>>>>> threddsConfig.xml
>>>>>>>>>> is set at the default 52428800 (50 Megabytes).  I upped the maximum 
>>>>>>>>>> size
>>>>>>>>>> by editing this parameter in threddsConfig:
>>>>>>>>>>
>>>>>>>>>>              <NetcdfSubsetService>
>>>>>>>>>>                <allow>true</allow>
>>>>>>>>>>                <scour>15 min</scour>
>>>>>>>>>>                <maxAge>30 min</maxAge>
>>>>>>>>>> <!--maxFileDownloadSize>2097152</maxFileDownloadSize-->
>>>>>>>>>> <!--maxFileDownloadSize>52428800</maxFileDownloadSize-->
>>>>>>>>>> <maxFileDownloadSize>10552428800</maxFileDownloadSize>
>>>>>>>>>>              </NetcdfSubsetService>
>>>>>>>>>>
>>>>>>>>>> Then I realized that somehow, the file is being seen as much larger 
>>>>>>>>>> than
>>>>>>>>>> 3.1G, which is an odd behavior...I shouldn't have to specify 10G to
>>>>>>>>>> cover a 3.1G file.  So, I'll have to dig and see what's going on.
>>>>>>>>>>
>>>>>>>>>> Have a good weekend,
>>>>>>>>>>              Lansing
>>>>>>>>>>
>>>>>>>>>> On 4/11/2014 5:31 PM, Luke Sheneman wrote:
>>>>>>>>>>> New Client Reply: TDS java exception 
>>>>>>>>>>> "java.lang.NegativeArraySizeException" for large netcdf
>>>>>>>>>>>
>>>>>>>>>>> Thanks Lansing -
>>>>>>>>>>>
>>>>>>>>>>> I've attached the latest log files.
>>>>>>>>>>>
>>>>>>>>>>> I am somewhat familiar with the data.   These are downscaled climate
>>>>>>>>>>> model outputs.  Some of the data are historical and some are
>>>>>>>>>>> predictive.   The one you were looking at is predicting climate
>>>>>>>>>>> scenarios for periods in the future (the years 2086 through 2100).  
>>>>>>>>>>>  I
>>>>>>>>>>> don't see anything weird with those dates in the data that you were
>>>>>>>>>>> looking at...
>>>>>>>>>>>
>>>>>>>>>>> Have a great weekend, and I look forward to working with you soon 
>>>>>>>>>>> to get
>>>>>>>>>>> the java exception.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>              -Luke
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Luke Sheneman, Ph.D
>>>>>>>>>>> Technology and Data Services Manager
>>>>>>>>>>> Northwest Knowledge Network (NKN) / University of Idaho
>>>>>>>>>>> http://www.northwestknowledge.net
>>>>>>>>>>>
>>>>>>>>>>> address@hidden
>>>>>>>>>>> Office: 208.885.4228   Mobile: 208.669.2248
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 4/11/14, 3:58 PM, Unidata THREDDS Support wrote:
>>>>>>>>>>>> baby steps...
>>>>>>>>>>>>
>>>>>>>>>>>> When you get a moment, could you repeat the log capture procedure? 
>>>>>>>>>>>>  I'm about to head out for the day, but I'll pull this up again on 
>>>>>>>>>>>> Monday.  I looked at one of the files already, which I downloaded 
>>>>>>>>>>>> using the http server.  Are you familiar with the data, by chance? 
>>>>>>>>>>>>  I notice that the calendar dates (Gregorian, days since 
>>>>>>>>>>>> 1901-01-01) are for 2086-2100.  Is this right, or is something 
>>>>>>>>>>>> else fishy here?
>>>>>>>>>>>>
>>>>>>>>>>>> Have a good weekend!
>>>>>>>>>>>>
>>>>>>>>>>>> -Lansing
>>>>>>>>>>>>
>>>>>>>>>>>> Ticket Details
>>>>>>>>>>>> ===================
>>>>>>>>>>>> Ticket ID: SYD-491615
>>>>>>>>>>>> Department: Support THREDDS
>>>>>>>>>>>> Priority: Normal
>>>>>>>>>>>> Status: Open
>>>>>>>>>>>>
>>>>>>>>>>> Ticket Details
>>>>>>>>>>> ===================
>>>>>>>>>>> Ticket ID: SYD-491615
>>>>>>>>>>> Department: Support THREDDS
>>>>>>>>>>> Priority: Normal
>>>>>>>>>>> Status: Open
>>>>>>>>>>> Link:  
>>>>>>>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>>>>>>>> Ticket Details
>>>>>>>>>> ===================
>>>>>>>>>> Ticket ID: SYD-491615
>>>>>>>>>> Department: Support THREDDS
>>>>>>>>>> Priority: High
>>>>>>>>>> Status: Open
>>>>>>>>>>
>>>>>>>>> Ticket Details
>>>>>>>>> ===================
>>>>>>>>> Ticket ID: SYD-491615
>>>>>>>>> Department: Support THREDDS
>>>>>>>>> Priority: High
>>>>>>>>> Status: Open
>>>>>>>>> Link:  
>>>>>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>>>>>> Ticket Details
>>>>>>>> ===================
>>>>>>>> Ticket ID: SYD-491615
>>>>>>>> Department: Support THREDDS
>>>>>>>> Priority: High
>>>>>>>> Status: Open
>>>>>>>>
>>>>>> Ticket Details
>>>>>> ===================
>>>>>> Ticket ID: SYD-491615
>>>>>> Department: Support THREDDS
>>>>>> Priority: High
>>>>>> Status: Open
>>>>>> Link:  
>>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>>>> Ticket Details
>>>>> ===================
>>>>> Ticket ID: SYD-491615
>>>>> Department: Support THREDDS
>>>>> Priority: High
>>>>> Status: Open
>>>> Ticket Details
>>>> ===================
>>>> Ticket ID: SYD-491615
>>>> Department: Support THREDDS
>>>> Priority: High
>>>> Status: Open
>>>> Link:  
>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>> Ticket Details
>>> ===================
>>> Ticket ID: SYD-491615
>>> Department: Support THREDDS
>>> Priority: High
>>> Status: Open
>>>
>>>
>>>
>>>
>>> Ticket Details
>>> ===================
>>> Ticket ID: SYD-491615
>>> Department: Support THREDDS
>>> Priority: High
>>> Status: Open
>>> Link:  
>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770
>>
>> Ticket Details
>> ===================
>> Ticket ID: SYD-491615
>> Department: Support THREDDS
>> Priority: High
>> Status: Open
>>
>
>
>
> Ticket Details
> ===================
> Ticket ID: SYD-491615
> Department: Support THREDDS
> Priority: High
> Status: Open
> Link:  
> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770



Ticket Details
===================
Ticket ID: SYD-491615
Department: Support THREDDS
Priority: Emergency
Status: Open