[netCDFJava #TIG-448079]: reading netcdf files with lzv compression

Greetings Eric,

Currently netCDF-Java does not support lzf as a filter. As far as I
understand it, we only support the following for reading through Java:

deflate
shuffle
fletcher32

We would need to add an lzf filter to our code. It looks like there
is a java implementation of lzf:

https://github.com/ning/compress/wiki


and the license is nice (apache-2). That makes me think we have a
good chance at adding this without too many roadblocks. The big
roadblock in this case is time on our end. We'd also need some sample
files to verify that we are doing things correctly, which we currently
do not have.

If you would like to get up and running today, you can build a version
of netCDF-C that links to a build of HDF5 that includes the lzf filter.
From there, you can tell netCDF-Java to read the file through the
netCDF-C libary. One way to do this is to use the runtime configuration
infrastructure:

https://www.unidata.ucar.edu/software/thredds/current/netcdf-java/reference/RuntimeLoading.html#XML

specifically by setting the libraryPath attribute to point to your
netCDF-C lib directory, and useForReading as true.

Not super pretty, but it should get the job done.

Cheers,

Sean

> Hi
> 
> I am attempting to use the netCDF java api to read files where the data
> has been compressed using lzf compression. Currently, when I attempt to
> call Variable.read() I get the following error:
> 
> fail = java.lang.RuntimeException: Unknown filter type=32000
> java.lang.RuntimeException: Unknown filter type=32000
> at 
> ucar.nc2.iosp.hdf5.H5tiledLayoutBB$DataChunk.getByteBuffer(H5tiledLayoutBB.java:248)
> at ucar.nc2.iosp.LayoutBBTiled.hasNext(LayoutBBTiled.java:128)
> at ucar.nc2.iosp.hdf5.H5tiledLayoutBB.hasNext(H5tiledLayoutBB.java:152)
> at ucar.nc2.iosp.IospHelper.readData(IospHelper.java:310)
> at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:292)
> at ucar.nc2.iosp.hdf5.H5iosp.readData(H5iosp.java:169)
> at ucar.nc2.iosp.hdf5.H5iosp.readData(H5iosp.java:144)
> at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:2026)
> at ucar.nc2.Variable.reallyRead(Variable.java:874)
> at ucar.nc2.Variable._read(Variable.java:845)
> at ucar.nc2.Variable.read(Variable.java:723)
> at lng.ReaderTest.main(ReaderTest.java:89)
> 
> Thanks
> 
> Eric Kaplan
> 
> Hi
> 
> I am attempting to use the netCDF java api to read files where the data
> has been compressed using lzf compression. Currently, when I attempt to
> call Variable.read() I get the following error:
> 
> fail = java.lang.RuntimeException: Unknown filter type=32000
> java.lang.RuntimeException: Unknown filter type=32000
> at 
> ucar.nc2.iosp.hdf5.H5tiledLayoutBB$DataChunk.getByteBuffer(H5tiledLayoutBB.java:248)
> at ucar.nc2.iosp.LayoutBBTiled.hasNext(LayoutBBTiled.java:128)
> at ucar.nc2.iosp.hdf5.H5tiledLayoutBB.hasNext(H5tiledLayoutBB.java:152)
> at ucar.nc2.iosp.IospHelper.readData(IospHelper.java:310)
> at ucar.nc2.iosp.IospHelper.readDataFill(IospHelper.java:292)
> at ucar.nc2.iosp.hdf5.H5iosp.readData(H5iosp.java:169)
> at ucar.nc2.iosp.hdf5.H5iosp.readData(H5iosp.java:144)
> at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:2026)
> at ucar.nc2.Variable.reallyRead(Variable.java:874)
> at ucar.nc2.Variable._read(Variable.java:845)
> at ucar.nc2.Variable.read(Variable.java:723)
> at lng.ReaderTest.main(ReaderTest.java:89)
> 
> Thanks
> 
> Eric Kaplan

Ticket Details
===================
Ticket ID: TIG-448079
Department: Support netCDF Java
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.