[netCDFJava #PYP-475421]: Bug in NetCDF-Java 4.3 exposed by this GRIB2 file

On 6/14/2012 2:54 PM, Rich Signell wrote:
> New Client Reply: Bug in NetCDF-Java 4.3 exposed by this GRIB2 file
>
> Lansing,
>
> I was expecting you to say that this issue was a bug in the GRIB
> reader, and it would get a ticket to be fixed.   But I think you are
> saying that the data was written wrong, and therefore the issue will
> not be fixed?
>
> This is one of those "big deal" datasets that it would be great to
> read correctly.
> http://rda.ucar.edu/pub/cfsr.html
>
> It's good to know that NcML can fix the issue for a file, but there
> are hundreds of these, so I would be better to fix the reader if you
> can.
>
> Thanks,
> -Rich
>
> On Tue, Jun 12, 2012 at 12:53 PM, Unidata netCDF Java Support
> <address@hidden> wrote:
>>> Lansing,
>>>
>>> It looks like you are using ToolsUI 4.2, not ToolsUI 4.3.   In toolsUI
>>> 4.2, you see one variable, but the times are wrong.   In 4.3, you see
>>> two variables with correct times, but we want one variable with
>>> correct time!
>>>
>>> Try this:
>>> http://www.unidata.ucar.edu/software/netcdf-java/v4.3/webstart/netCDFtools.jnlp
>>>
>>> On Mon, Jun 11, 2012 at 1:10 PM, Unidata netCDF Java Support
>>> <address@hidden> wrote:
>>>>> NetCDF-Java folks:
>>>>>
>>>>> There appears to be a bug in the GRIB2 decoding problem using
>>>>> NetCDF-Java 4.3 on some of the surface pressure data we got from
>>>>> http://rda.ucar.edu/pub/cfsr.html
>>>>>
>>>>> This data is supposed to have a single variable (pressure) with
>>>>> uniform hourly sampling, but when we bring it up in NetCDF 4.3 tools,
>>>>> we see two variables instead of one, with the 1st having a uniform
>>>>> time base of 6 hours (0,6,12), and the 2nd having a non-uniform time
>>>>> base that fills in the rest of the hourly data  (1,2,3,4,5
>>>>> 7,8,9,10,11  13,14...)
>>>>>
>>>>> Screenshot: http://screencast.com/t/DWEgKMfy
>>>>>
>>>>> I put a sample file here:
>>>>> http://geoport.whoi.edu/thredds/catalog/usgs/data1/rsignell/test/catalog.html?dataset=usgs/data1/rsignell/test/pressfc.gdas.199801.grb2
>>>>>
>>>>> Thanks,
>>>>> Rich
>>>>> --
>>>>> Dr. Richard P. Signell   (508) 457-2229
>>>>> USGS, 384 Woods Hole Rd.
>>>>> Woods Hole, MA 02543-1598
>>>>>
>>>>>
>>>> Hello Rich,
>>>>
>>>> I'm not sure what's going on, but I tried your file and saw only one 
>>>> variable reported.  I've attached a screenshot from NetCDF-Java ToolsUI.  
>>>> Maybe I'm missing something obvious here?
>>>>
>>>> Regards,
>>>>   Lansing Madry
>>>>   Unidata
>>>>   Boulder, Colorado
>>>>
>>>> Ticket Details
>>>> ===================
>>>> Ticket ID: PYP-475421
>>>> Department: Support netCDF Java
>>>> Priority: Normal
>>>> Status: Open
>>>
>>>
>>> --
>>> Dr. Richard P. Signell   (508) 457-2229
>>> USGS, 384 Woods Hole Rd.
>>> Woods Hole, MA 02543-1598
>>>
>>>
>> Rich,
>>
>> Thanks for educating me on 4.2 vs. 4.3 -- details, details!
>>
>> I dug a little deeper to see what is going on with the data in the file you 
>> sent.  It looks like, at the byte level, there are some differences that 4.2 
>> conveniently ignored and 4.3 notices.  Specifically, if you look at the file 
>> in the iosp/GRIB2/GRIB2Collection tab, you'll notice that two variables seem 
>> to show up, with separate GDS hash codes.  In that view, if you look down to 
>> the Grib2GridDefinitionSection, highlight a row, right-click on the row, and 
>> select "Show GDS", you'll see the information for Lo2 differs for the two 
>> "different" variables:
>>
>> Lo2 - longitude of last grid point == 359687000
>> Lo2 - longitude of last grid point == 359688000
>>
>> This is really a defect in how the data was written out - perhaps a rounding 
>> error somewhere, or some other loss of precision.
>>
>> In the meantime, you can "correct" the issue using a bit of NcML to change 
>> the GDS hashcode.  Details are here:
>>
>> http://www.unidata.ucar.edu/software/netcdf-java/formats/GribFiles.html
>>
>> and I've tried this bit of NcML and found it to work in this case:
>>
>> <?xml version="1.0" encoding="UTF-8"?>
>> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2"; 
>> location="C:/Users/madry/rsignell/pressfc.gdas.199801.grb2">
>>   <iospParam>
>>    <gdsHash from="1450192070" to="1450218978"/>
>>   </iospParam>
>> </netcdf>
>>
>> However, you will need to delete the associated .ncx file before opening 
>> this NcML file in the toolsUI-4.3 viewer.
>>
>> Let me know if this doesn't work for you, and I'll take another crack at it.
>>
>> Cheers,
>>   Lansing
>>
>> Ticket Details
>> ===================
>> Ticket ID: PYP-475421
>> Department: Support netCDF Java
>> Priority: Normal
>> Status: Open
>>
>
>
Rich,

I spoke with someone who has been here a little longer than I have, and 
he tells me that there is a "fix" that has been previously incorporated 
into the TDS, but appears not to be in our current version.  
Essentially, because different machines (or even the same machine, under 
different circumstances) can write the same data out and not have it 
look identical past a certain precision, we have to set a cut-off on the 
precision for hashing purposes.  I apologize for not passing that onto 
you more clearly, but I've submitted a ticket on it.

Regards,
   Lansing Madry
   Unidata
   Boulder, Colorado



Ticket Details
===================
Ticket ID: PYP-475421
Department: Support netCDF Java
Priority: Normal
Status: Open