Luke, So here's what I've done to try to capture your problem: 1) I've been running our current development branch out of my debugger, serving one of the files I grabbed. The development branch is a little ahead of the stable release branch, so I also 2) have been running the latest stable release of the TDS (4.3.21) from the command line, using the same file and catalog. 3) I've also been using both the 4.3.21 and current development branch of ToolsUI to look at the file and interact with the TDS in a live environment What I've found is that neither TDS generates a stack trace when serving up the large .nc file through OPeNDAP. I can both look at ASCII (as long as I restrict the request) and download a .dods file. Using ToolsUI, I can hit the TDS remotely and interact with the dataset. To that end, I set up a viewer to scroll through time and make a movie of the data - looks like continental US plus a little something bleeding over into Canada near the WA/ID/MT border (see attached screen grab). Do you have ToolsUI? If not, you should grab the jar file here: https://www.unidata.ucar.edu/downloads/netcdf/netcdf-java-4/index.jsp It's a handy file exploration tool. You can run it from the command line using: java -Xmx1g -jar toolsui-4.3.jar Once you have it running, you can navigate to either of two tabs to take a quick look at the file. The first is the NCDump tab, the second is the Iosp/HDF5/HDF5-Objects tab. The NCDump output will be obvious. When you open under HDF5, selecting anything in the upper window will cause the lower window to populate with relevant information. If the file comes up fine there, then try the THREDDS tab. In the catalog URL entry field, enter the address of your thredds server, substituting .xml for .html. For instance, it looks like this for me (attached). Clicking the Show Dataset button should take you to another pane under FeatureTypes/Grids. From there, you can select the Daily Mean Near-Surface Wind and click the RedRaw, the little red martian in the upper right. A graphics display will come up, empty, and if you click RedRaw again, you should see the data populating the screen... On the other hand, if something fails quickly, then I'm going to have to fall back on the configuration issue, looking hard at 32 vs. 64 bit libraries somewhere. By the way, what are the other netcdf tools you mentioned you were using that had no problems looking at the files? As an aside, Matlab I think uses the netcdf-c libraries, and it might be good to make sure it is using a recent set. Fortunately, if this is a NetCDF issue, I have some friends around the corner who can help us out. -Lansing On 4/30/2014 7:27 AM, Luke Sheneman wrote: > New Client Reply: TDS java exception "java.lang.NegativeArraySizeException" > for large netcdf > > Katherine, > > Let's see what Lansing says. THREDDs should be able to handle these > files. The fact that it seems to break when these files are large > indicates either a serious bug in THREDDs or a problem with our > operating environment. I am curious if unidata sees this java > exception when they try to serve these files on their TDS > installations? If so, then it is most likely a bug with TDS itself. > That is my expectation, and should be really easy to test. I know that > Lansing already downloaded some of the netcdfs. > > -Luke > > -- > > Luke Sheneman, Ph.D > Technology and Data Services Manager > Northwest Knowledge Network (NKN) / University of Idaho > http://www.northwestknowledge.net > > address@hidden > Office: 208.885.4228 Mobile: 208.669.2248 > > > On 4/29/14, 10:10 AM, Hegewisch, Katherine (address@hidden) wrote: >> Luke and Lansing, >> >> Yes, these netcdf files are for the years 1950 - 2100. So those years >> 2086-2100 you saw in the calendar dates are correct. The years are also in >> the file names. These are future climate runs. >> >> Yes these are netcdf4 files. I generate them in MATLAB using >> ncid = netcdf.create(FULLFILENAME,'NETCDF4'); >> >> Yes these files are compressed. I compress them in MATLAB using >> netcdf.defVarDeflate(ncid,dataID,true,true,5); >> (the compression is a 5 out of 10... so this should be about 1/2 >> compression). >> >> I wonder if you need me to make smaller files by using smaller ranges of >> years. >> Luke - would you like me to make an uncompressed file to see if that works >> better with THREDDS? >> >> Katherine >> ________________________________________ >> From: Sheneman, Lucas (address@hidden) >> Sent: Monday, April 28, 2014 7:22 PM >> To: address@hidden >> Cc: Hegewisch, Katherine (address@hidden) >> Subject: Re: [THREDDS #SYD-491615]: TDS java exception >> "java.lang.NegativeArraySizeException" for large netcdf >> >> Lansing and Katherine - >> >> Katherine, can you address Lansing's questions about your netcdf >> files? We are trying to debug thredds. It works on your smaller >> files, but some things fail on your larger files (>2GB). >> >> Please copy me on your correspondence... >> >> Thanks, >> -Luke >> >> -- >> >> Luke Sheneman, Ph.D >> Technology and Data Services Manager >> Northwest Knowledge Network (NKN) / University of Idaho >> http://www.northwestknowledge.net >> >> address@hidden >> Office: 208.885.4228 Mobile: 208.669.2248 >> >> >> On 4/26/14, 8:15 AM, Unidata THREDDS Support wrote: >>> Compressed how? I can open the file in ToolsUI, another netCDF-Java >>> interface. If they are compressed somehow, is the 3.1G the compressed >>> or uncompressed size? Are they actually netcdf4 files? >>> >>> -Lansing >>> >>> On 4/25/2014 6:10 PM, Luke Sheneman wrote: >>>> New Client Reply: TDS java exception >>>> "java.lang.NegativeArraySizeException" for large netcdf >>>> >>>> Thanks Lansing. >>>> >>>> I believe the netcdfs are compressed. Perhaps that is playing a role >>>> here? >>>> >>>> I look forward to seeing what you come up with. >>>> >>>> Thanks, >>>> -Luke >>>> >>>> -- >>>> >>>> Luke Sheneman, Ph.D >>>> Technology and Data Services Manager >>>> Northwest Knowledge Network (NKN) / University of Idaho >>>> http://www.northwestknowledge.net >>>> >>>> address@hidden >>>> Office: 208.885.4228 Mobile: 208.669.2248 >>>> >>>> >>>> On 4/25/14, 3:10 PM, Unidata THREDDS Support wrote: >>>>> Luke, >>>>> >>>>> I finally got my system back online today. It's not clear what >>>>> happened, but I had to wipe out and reinstall my standard toolkit >>>>> programs. I fired up a catalog with your file in it, which is 3.1G in >>>>> size. I noticed that trying to download the entire dataset through the >>>>> ncss service failed because the maximum size limit in threddsConfig.xml >>>>> is set at the default 52428800 (50 Megabytes). I upped the maximum size >>>>> by editing this parameter in threddsConfig: >>>>> >>>>> <NetcdfSubsetService> >>>>> <allow>true</allow> >>>>> <scour>15 min</scour> >>>>> <maxAge>30 min</maxAge> >>>>> <!--maxFileDownloadSize>2097152</maxFileDownloadSize--> >>>>> <!--maxFileDownloadSize>52428800</maxFileDownloadSize--> >>>>> <maxFileDownloadSize>10552428800</maxFileDownloadSize> >>>>> </NetcdfSubsetService> >>>>> >>>>> Then I realized that somehow, the file is being seen as much larger than >>>>> 3.1G, which is an odd behavior...I shouldn't have to specify 10G to >>>>> cover a 3.1G file. So, I'll have to dig and see what's going on. >>>>> >>>>> Have a good weekend, >>>>> Lansing >>>>> >>>>> On 4/11/2014 5:31 PM, Luke Sheneman wrote: >>>>>> New Client Reply: TDS java exception >>>>>> "java.lang.NegativeArraySizeException" for large netcdf >>>>>> >>>>>> Thanks Lansing - >>>>>> >>>>>> I've attached the latest log files. >>>>>> >>>>>> I am somewhat familiar with the data. These are downscaled climate >>>>>> model outputs. Some of the data are historical and some are >>>>>> predictive. The one you were looking at is predicting climate >>>>>> scenarios for periods in the future (the years 2086 through 2100). I >>>>>> don't see anything weird with those dates in the data that you were >>>>>> looking at... >>>>>> >>>>>> Have a great weekend, and I look forward to working with you soon to get >>>>>> the java exception. >>>>>> >>>>>> Thanks, >>>>>> -Luke >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Luke Sheneman, Ph.D >>>>>> Technology and Data Services Manager >>>>>> Northwest Knowledge Network (NKN) / University of Idaho >>>>>> http://www.northwestknowledge.net >>>>>> >>>>>> address@hidden >>>>>> Office: 208.885.4228 Mobile: 208.669.2248 >>>>>> >>>>>> >>>>>> On 4/11/14, 3:58 PM, Unidata THREDDS Support wrote: >>>>>>> baby steps... >>>>>>> >>>>>>> When you get a moment, could you repeat the log capture procedure? I'm >>>>>>> about to head out for the day, but I'll pull this up again on Monday. >>>>>>> I looked at one of the files already, which I downloaded using the http >>>>>>> server. Are you familiar with the data, by chance? I notice that the >>>>>>> calendar dates (Gregorian, days since 1901-01-01) are for 2086-2100. >>>>>>> Is this right, or is something else fishy here? >>>>>>> >>>>>>> Have a good weekend! >>>>>>> >>>>>>> -Lansing >>>>>>> >>>>>>> Ticket Details >>>>>>> =================== >>>>>>> Ticket ID: SYD-491615 >>>>>>> Department: Support THREDDS >>>>>>> Priority: Normal >>>>>>> Status: Open >>>>>>> >>>>>> Ticket Details >>>>>> =================== >>>>>> Ticket ID: SYD-491615 >>>>>> Department: Support THREDDS >>>>>> Priority: Normal >>>>>> Status: Open >>>>>> Link: >>>>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770 >>>>> Ticket Details >>>>> =================== >>>>> Ticket ID: SYD-491615 >>>>> Department: Support THREDDS >>>>> Priority: High >>>>> Status: Open >>>>> >>>> Ticket Details >>>> =================== >>>> Ticket ID: SYD-491615 >>>> Department: Support THREDDS >>>> Priority: High >>>> Status: Open >>>> Link: >>>> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770 >>> Ticket Details >>> =================== >>> Ticket ID: SYD-491615 >>> Department: Support THREDDS >>> Priority: High >>> Status: Open >>> > > > Ticket Details > =================== > Ticket ID: SYD-491615 > Department: Support THREDDS > Priority: High > Status: Open > Link: > https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=23770 Ticket Details =================== Ticket ID: SYD-491615 Department: Support THREDDS Priority: High Status: Open
Attachment:
NKN_ToolsUI_capture.PNG
Description: PNG image
Attachment:
NKN_ToolsUI_THREDDS.PNG
Description: PNG image
