RE: FW: Netcdf4-Java reading problem

Thanks John. We'll track that ticket.

We're chunking at 100000 for 1Hz data, corresponding roughly to 1 day of data 
we typically read at a time. We haven't explored chunking for the 10Hz data 
because of the size limit we ran into. Reading is pretty darn fast for the 1Hz 
data.

Kim
________________________________________
From: John Caron [address@hidden]
Sent: Friday, August 22, 2014 10:18 AM
To: Edward Hartnett
Cc: address@hidden; Kim Kokkonen; support-netcdf-java
Subject: Re: FW: Netcdf4-Java reading problem

sorry that should be:

Ive added JIRA ticket:

https://bugtracking.unidata.ucar.edu/browse/TDS-594


On Fri, Aug 22, 2014 at 10:17 AM, John Caron 
<address@hidden<mailto:address@hidden>> wrote:
Hi Ed and all:

The situation, AFAIU, is that dimension lengths are limited to 4  bytes in 
netcdf-3, but 8 bytes in netcdf-4. The Java API has not caught up yet, but we 
will.

The C API uses size_t, which is the size of a pointer (?), which to me isnt 
clear why that solves the issue (maybe you remember, Ed?). I will talk with 
Russ ASAP about it.

In the meanwhile, if you need to read through Java API, you are stuck with 
dimension lengths < 2^31 (note not < 2^32 due to the sign problem in Java). 
Because its an API change, it will have to be in a major version like 5.0, so 
thats not for a while. Ive added JIRA ticket:


Perhaps you could write 1 year files for now?

BTW, how are you chunking these?

John


On Wed, Aug 20, 2014 at 3:30 PM, Edward Hartnett 
<address@hidden<mailto:address@hidden>> wrote:
Howdy John!

This is for 10 years of 10 Hz data. The dimension is time.

Is there a better way? Has no one encountered this problem before? I am amazed!

Ed
________________________________________
From: John Caron [address@hidden<mailto:address@hidden>]
Sent: Wednesday, August 20, 2014 1:03 PM
To: Edward Hartnett
Cc: address@hidden<mailto:address@hidden>; Kim Kokkonen; support-netcdf-java
Subject: Re: FW: Netcdf4-Java reading problem

Yes, netcdf-java will fail if any dimension length > 2^31 -1, since we use 
signed ints for dimensions lengths. Can netcdf-C library handle this?

Thats quite a massive file you have there. How big is it?



On Mon, Aug 18, 2014 at 3:53 PM, Edward Hartnett 
<address@hidden<mailto:address@hidden><mailto:address@hidden<mailto:address@hidden>>>
 wrote:
Howdy John!

I hope all is going well at Unidata! ;-)

Can netCDF-java netCDF-4 files handle unlimited dimensions > MAX_INT?

Thanks!
Ed
________________________________________
From: Benjamin Busby
Sent: Monday, August 18, 2014 2:36 PM
To: Edward Hartnett
Cc: Kim Kokkonen
Subject: Netcdf4-Java reading problem

Hi Ed,

I've come across a problem with reading a netcdf file using Java, where a 
dimension whose length is greater than the maximum int value causes the file to 
fail when attempting to open it. Here's the header for the current netcdf file 
that I'm working with:

netcdf AllTimData {
types:
  compound AllTimScienceSamples {
    int64 packetVtcw ;
    int64 sampleVtcw ;
    byte adcLatchup3 ;
    byte adcLatchup2 ;
    byte adcLatchup1 ;
    byte coneActiveStateD ;
    byte coneActiveStateC ;
    byte coneActiveStateB ;
    byte coneActiveStateA ;
    byte relayCone ;
    byte shutterPositionD ;
    byte shutterPositionC ;
    byte shutterPositionB ;
    byte shutterPositionA ;
    byte shutterOvercurrentD ;
    byte shutterOvercurrentC ;
    byte shutterOvercurrentB ;
    byte shutterOvercurrentA ;
    byte dspGainMode ;
    byte dspEclipseMode ;
    byte sequenceCounter ;
    int heaterControl ;
    byte loopDataLengthA ;
    byte loopDataLengthB ;
    int64 MUTimeStamp ;
  }; // AllTimScienceSamples
  compound AllTimScienceDataA {
    int64 sampleVtcw ;
    int64 dataVtcw ;
    int loopData ;
  }; // AllTimScienceDataA
dimensions:
        TSS-sampleVtcw = UNLIMITED ; // (328872712 currently)
        TSS-index = UNLIMITED ; // (3289 currently)
        TSDA-dataVtcw = UNLIMITED ; // (3282309813 currently)
        TSDA-index = UNLIMITED ; // (32824 currently)
variables:
        int64 TSS-sampleVtcw(TSS-sampleVtcw) ;
        AllTimScienceSamples TimScienceSamples(TSS-sampleVtcw) ;
        int64 TSS-index(TSS-index) ;
        int64 TSDA-dataVtcw(TSDA-dataVtcw) ;
        AllTimScienceDataA TimScienceDataA(TSDA-dataVtcw) ;
        int64 TSDA-index(TSDA-index) ;
}

Opening and reading from this file works fine in Python, but attempting to even 
open the file in Java causes this error:

java.io.IOException: java.lang.IllegalArgumentException: Unlimited Dimension 
length =-1012657483 must >= 0
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:430)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:397)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:384)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:372)
        at IndexedNC4Reader.main(IndexedNC4Reader.java:62)
Caused by: java.lang.IllegalArgumentException: Unlimited Dimension length 
=-1012657483 must >= 0
        at ucar.nc2.Dimension.setLength(Dimension.java:433)
        at ucar.nc2.Dimension.<init>(Dimension.java:363)
        at ucar.nc2.iosp.hdf5.H5header.addDimension(H5header.java:801)
        at ucar.nc2.iosp.hdf5.H5header.findDimensionScales(H5header.java:613)
        at ucar.nc2.iosp.hdf5.H5header.makeNetcdfGroup(H5header.java:416)
        at ucar.nc2.iosp.hdf5.H5header.read(H5header.java:216)
        at ucar.nc2.iosp.hdf5.H5iosp.open(H5iosp.java:130)
        at ucar.nc2.NetcdfFile.<init>(NetcdfFile.java:1528)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:820)
        at ucar.nc2.NetcdfFile.open(NetcdfFile.java:427)
        ... 4 more

I traced this back to the documentation for ucar.nc2.Dimension.setLength which 
takes an int argument, which won't work for the 3,282,309,813 size of the 
dimension in TimScienceDataA. Do you know of a workaround for this problem?

Also, possibly related, trying to examine the variables independently through a 
"ncdump -v" is blank for variables/dimensions with a size greater than 2^31-1. 
Is this a normal occurrence?

Thank you,
Ben