[netCDFJava #RGN-330338]: Different grib2 decodings between NCL and netCDF Java library

A fix is now on the 4.5.3 branch in github. We are running our unit tests on it 
overnight to see if any problems show up.


> Not sure--I have to check with my development team. But, I will pass this 
> on--thanks!
> ---
> Brian Griffith, Ph.D.
> Senior Research Scientist & Site Lead
> PEMDAS Technologies & Innovations
> Phone: 828-398-8000
> address@hidden<mailto:address@hidden>
> 
> On Aug 22, 2014, at 10:23 AM, Unidata netCDF Java Support 
> <address@hidden<mailto:address@hidden>> wrote:
> 
> Are you building from github repo? I can have a quick fix later today, but we 
> have to examine the entire decoding to make sure we didnt miss anything, so 
> that will take a week or more.
> 
> Thanks John,
> 
> Can you explain the issue to me? I ask so that I can put a workaround in 
> place (since my automated decoding process uses netCDF-Java) for this case 
> until you release a fix.
> 
> Thanks,
> Brian
> ---
> Brian Griffith, Ph.D.
> Senior Research Scientist & Site Lead
> PEMDAS Technologies & Innovations
> Phone: 828-398-8000
> address@hidden<mailto:address@hidden><mailto:address@hidden>
> 
> [cid:3FE592C7-C041-4CEB-8A81-855478D05A7E@hsd1.co.comcast.net<mailto:address@hidden>.]
> CONFIDENTIALITY NOTICE: This email message and all attachments transmitted 
> with it may contain legally privileged and confidential information intended 
> solely for the use of the addressee. If the reader of this message is not the 
> intended recipient, you are hereby notified that any reading, dissemination, 
> distribution, copying, or other use of this message or its attachments is  
> strictly prohibited. If you have received this message in error, please 
> notify the sender immediately and delete this message from your system. Thank 
> you.
> 
> On Aug 22, 2014, at 9:51 AM, Unidata netCDF Java Support 
> <address@hidden<mailto:address@hidden><mailto:address@hidden>> wrote:
> 
> Hi Brian:
> 
> After examining wgrib and gempak code, i think that my code is handling the 
> nbit=0 case incorrectly. I am working on a fix. The fix will be in 4.5.3, and 
> we may backport to 4.3.23, but im not sure.
> 
> Thanks very much for reporting this problem.
> 
> John
> 
> 
> 
> Hi Brian:
> 
> This is what Ive found so far:
> 
> file: https://aws1.pemdastech.com/tests/1420921000300.grib2
> 
> variable: CICEP_P0_L1_GLC0 aka "Categorical_Ice_Pellets_surface"
> 
> record: only has 1 record, starts at pos 498311562
> 
> a dump of the drs( section 5 ) type 3 has these values (note that type 3 
> inherits type 2 and type 0 values):
> 
> Type0{referenceValue=0.0, binaryScaleFactor=0, decimalScaleFactor=0, 
> numberOfBits=0, originalType=0}
> Type2{splittingMethod=1, missingValueManagement=0, primaryMissingValue=0.0, 
> secondaryMissingValue=0.0, numberOfGroups=0, referenceGroupWidths=4, 
> bitsGroupWidths=1, referenceGroupLength=1905141, lengthIncrement=1, 
> lengthLastGroup=-112001040, bitsScaledGroupLength=0}
> Type3{orderSpatial=2, descriptorSpatial=1}
> 
> my code has the logic that if numberOfGroups=0, then set all data equal to 
> missing value, in this case primaryMissingValue=0.0. This would explain why 
> NCL gives all zeroes.
> 
> however, my code has the additional operation that if the data is missing, 
> set them to NaNs. So, it may be possible that "set to missing value" is the 
> way that GRIB-2 drs=3 sets a constant field. For sure, 0.0 is a valid value, 
> given that the unit description is "Categorical ice pellets (yes=1 no=0)"
> 
> Another possibility is that numberOfGroups=0 should not mean "set to missing 
> values", but rather "this is a constant field, set to the reference value", 
> which is also 0.0. Im leaning towards this at the moment.
> 
> I will investigate more tommorrow.
> 
> John
> 
> 
> Thanks John,
> Here's a link to the HRRR forecast model output grib2 (500MB) file:
> https://aws1.pemdastech.com/tests/1420921000300.grib2
> 
> When I decode the file using NCL or wgrib2 I get "0" for the following fields 
> (NCL names):
> CICEP_P0_L1_GLC0
> CFRZR_P0_L1_GLC0
> 
> I also notice that I get the same issue for some lower sigma levels of cloud 
> cover(NCL names):
> TCDC_P0_L100_GLC0
> 
> When I decode either of those with netCDF-Java, I get NaN.
> 
> I didn't expect to have explicitly missing data for the HRRR model output 
> and, using wgrib2, I looked at the grib2 file code table 5.5 to examine the 
> missing value management and it returned a value indicating there were no 
> explictly encoded missing values in the grib2 file.
> 
> Thanks for your help!
> ---
> Brian Griffith, Ph.D.
> Senior Research Scientist & Site Lead
> PEMDAS Technologies & Innovations
> Phone: 828-398-8000
> address@hidden<mailto:address@hidden>
> 
> [cid:3FE592C7-C041-4CEB-8A81-855478D05A7E@hsd1.co.comcast.net.]
> CONFIDENTIALITY NOTICE: This email message and all attachments transmitted 
> with it may contain legally privileged and confidential information intended 
> solely for the use of the addressee. If the reader of this message is not the 
> intended recipient, you are hereby notified that any reading, dissemination, 
> distribution, copying, or other use of this message or its attachments is  
> strictly prohibited. If you have received this message in error, please 
> notify the sender immediately and delete this message from your system. Thank 
> you.
> 
> On Aug 20, 2014, at 1:56 PM, Unidata netCDF Java Support 
> <address@hidden<mailto:address@hidden>> wrote:
> 
> Hi Brian:
> 
> Netcdf-Java typically sets missing values to NaNs, sounds like NCL does not .
> 
> If you can send me a sample file (and as much detail as needed to reproduce) 
> I can double check if anything more nefarious is going on.
> 
> John
> 
> Dear Sir,
> 
> I am decoding NOAA ESRL HRRR model output in grib2 format and have 
> encountered a conflict.
> 
> I am decoding the categorical precipitation fields:
> CRAIN_P0_L1_GLC0
> CFRZR_P0_L1_GLC0
> CICEP_P0_L1_GLC0
> CSNOW_P0_L1_GLC0
> 
> In particular, I am decoding CFRZR and CICEP and am getting an issue that 
> when I use NCL to view the values of these parameters, I get values of "O".
> 
> When I use netCDF-Java library 4.3.x or 4.5.2 (or NASA's Panoply tool which 
> employs netCDF-Java libraries) to view the values of these parameters, I get 
> values of "NaN".
> 
> Can you explain what is happening here? Are the values in the grib2 file 
> actually missing and being decoded in two different ways? How can I determine?
> 
> Thanks,
> Brian
> ---
> Brian Griffith, Ph.D.
> Senior Research Scientist & Site Lead
> PEMDAS Technologies & Innovations
> Phone: 828-398-8000
> address@hidden<mailto:address@hidden><mailto:address@hidden>
> 
> [cid:3FE592C7-C041-4CEB-8A81-855478D05A7E@hsd1.co.comcast.net<mailto:address@hidden>.]
> CONFIDENTIALITY NOTICE: This email message and all attachments transmitted 
> with it may contain legally privileged and confidential information intended 
> solely for the use of the addressee. If the reader of this message is not the 
> intended recipient, you are hereby notified that any reading, dissemination, 
> distribution, copying, or other use of this message or its attachments is  
> strictly prohibited. If you have received this message in error, please 
> notify the sender immediately and delete this message from your system. Thank 
> you.
> 
> 
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: RGN-330338
> Department: Support netCDF Java
> Priority: Normal
> Status: Open
> 
> 
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: RGN-330338
> Department: Support netCDF Java
> Priority: Normal
> Status: Open
> 
> 
> 
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: RGN-330338
> Department: Support netCDF Java
> Priority: Normal
> Status: Open
> 
> 
> 
> 


Ticket Details
===================
Ticket ID: RGN-330338
Department: Support netCDF Java
Priority: Normal
Status: Open