[netCDFJava #RGN-330338]: Different grib2 decodings between NCL and netCDF Java library

Hi Brian:

After examining wgrib and gempak code, i think that my code is handling the 
nbit=0 case incorrectly. I am working on a fix. The fix will be in 4.5.3, and 
we may backport to 4.3.23, but im not sure.

Thanks very much for reporting this problem.

John



> Hi Brian:
> 
> This is what Ive found so far:
> 
> file: https://aws1.pemdastech.com/tests/1420921000300.grib2
> 
> variable: CICEP_P0_L1_GLC0 aka "Categorical_Ice_Pellets_surface"
> 
> record: only has 1 record, starts at pos 498311562
> 
> a dump of the drs( section 5 ) type 3 has these values (note that type 3 
> inherits type 2 and type 0 values):
> 
> Type0{referenceValue=0.0, binaryScaleFactor=0, decimalScaleFactor=0, 
> numberOfBits=0, originalType=0}
> Type2{splittingMethod=1, missingValueManagement=0, primaryMissingValue=0.0, 
> secondaryMissingValue=0.0, numberOfGroups=0, referenceGroupWidths=4, 
> bitsGroupWidths=1, referenceGroupLength=1905141, lengthIncrement=1, 
> lengthLastGroup=-112001040, bitsScaledGroupLength=0}
> Type3{orderSpatial=2, descriptorSpatial=1}
> 
> my code has the logic that if numberOfGroups=0, then set all data equal to 
> missing value, in this case primaryMissingValue=0.0. This would explain why 
> NCL gives all zeroes.
> 
> however, my code has the additional operation that if the data is missing, 
> set them to NaNs. So, it may be possible that "set to missing value" is the 
> way that GRIB-2 drs=3 sets a constant field. For sure, 0.0 is a valid value, 
> given that the unit description is "Categorical ice pellets (yes=1 no=0)"
> 
> Another possibility is that numberOfGroups=0 should not mean "set to missing 
> values", but rather "this is a constant field, set to the reference value", 
> which is also 0.0. Im leaning towards this at the moment.
> 
> I will investigate more tommorrow.
> 
> John
> 
> 
> > Thanks John,
> > Here's a link to the HRRR forecast model output grib2 (500MB) file:
> > https://aws1.pemdastech.com/tests/1420921000300.grib2
> >
> > When I decode the file using NCL or wgrib2 I get "0" for the following 
> > fields (NCL names):
> > CICEP_P0_L1_GLC0
> > CFRZR_P0_L1_GLC0
> >
> > I also notice that I get the same issue for some lower sigma levels of 
> > cloud cover(NCL names):
> > TCDC_P0_L100_GLC0
> >
> > When I decode either of those with netCDF-Java, I get NaN.
> >
> > I didn't expect to have explicitly missing data for the HRRR model output 
> > and, using wgrib2, I looked at the grib2 file code table 5.5 to examine the 
> > missing value management and it returned a value indicating there were no 
> > explictly encoded missing values in the grib2 file.
> >
> > Thanks for your help!
> > ---
> > Brian Griffith, Ph.D.
> > Senior Research Scientist & Site Lead
> > PEMDAS Technologies & Innovations
> > Phone: 828-398-8000
> > address@hidden<mailto:address@hidden>
> >
> > [cid:3FE592C7-C041-4CEB-8A81-855478D05A7E@hsd1.co.comcast.net.]
> > CONFIDENTIALITY NOTICE: This email message and all attachments transmitted 
> > with it may contain legally privileged and confidential information 
> > intended solely for the use of the addressee. If the reader of this message 
> > is not the intended recipient, you are hereby notified that any reading, 
> > dissemination, distribution, copying, or other use of this message or its 
> > attachments is  strictly prohibited. If you have received this message in 
> > error, please notify the sender immediately and delete this message from 
> > your system. Thank you.
> >
> > On Aug 20, 2014, at 1:56 PM, Unidata netCDF Java Support 
> > <address@hidden<mailto:address@hidden>> wrote:
> >
> > Hi Brian:
> >
> > Netcdf-Java typically sets missing values to NaNs, sounds like NCL does not 
> > .
> >
> > If you can send me a sample file (and as much detail as needed to 
> > reproduce) I can double check if anything more nefarious is going on.
> >
> > John
> >
> > Dear Sir,
> >
> > I am decoding NOAA ESRL HRRR model output in grib2 format and have 
> > encountered a conflict.
> >
> > I am decoding the categorical precipitation fields:
> > CRAIN_P0_L1_GLC0
> > CFRZR_P0_L1_GLC0
> > CICEP_P0_L1_GLC0
> > CSNOW_P0_L1_GLC0
> >
> > In particular, I am decoding CFRZR and CICEP and am getting an issue that 
> > when I use NCL to view the values of these parameters, I get values of "O".
> >
> > When I use netCDF-Java library 4.3.x or 4.5.2 (or NASA's Panoply tool which 
> > employs netCDF-Java libraries) to view the values of these parameters, I 
> > get values of "NaN".
> >
> > Can you explain what is happening here? Are the values in the grib2 file 
> > actually missing and being decoded in two different ways? How can I 
> > determine?
> >
> > Thanks,
> > Brian
> > ---
> > Brian Griffith, Ph.D.
> > Senior Research Scientist & Site Lead
> > PEMDAS Technologies & Innovations
> > Phone: 828-398-8000
> > address@hidden<mailto:address@hidden><mailto:address@hidden>
> >
> > [cid:3FE592C7-C041-4CEB-8A81-855478D05A7E@hsd1.co.comcast.net<mailto:address@hidden>.]
> > CONFIDENTIALITY NOTICE: This email message and all attachments transmitted 
> > with it may contain legally privileged and confidential information 
> > intended solely for the use of the addressee. If the reader of this message 
> > is not the intended recipient, you are hereby notified that any reading, 
> > dissemination, distribution, copying, or other use of this message or its 
> > attachments is  strictly prohibited. If you have received this message in 
> > error, please notify the sender immediately and delete this message from 
> > your system. Thank you.
> >
> >
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: RGN-330338
> > Department: Support netCDF Java
> > Priority: Normal
> > Status: Open
> >
> >
> >
> >

Ticket Details
===================
Ticket ID: RGN-330338
Department: Support netCDF Java
Priority: Normal
Status: Open