[netCDFJava #BUZ-409451]: Question about _FillValue for packed data

> Thanks, Russ....the new language clarifies that.  Might I also suggest
> adding a clarification to the "valid_range" just below?

Done, thanks!

--Russ

> 
> On Mon, Mar 22, 2010 at 3:20 PM, Unidata netCDF Java Support
> <address@hidden> wrote:
> >> New Client Reply: Question about _FillValue for packed data
> >>
> >> Russ --
> >>
> >> Thanks for your note.  I had used those two references in "Best
> >> Practices", but then also found some seemingly contradictory
> >> statements in the CF1.4 (section 2.5.1):
> >>
> >> "The missing values of a variable with scale_factor and/or add_offset
> >> attributes (see section Section 8.1, “Packed Data”) are interpreted
> >> relative to the variable's external values, i.e., the values stored in
> >> the netCDF file. Applications that process variables that have
> >> attributes to indicate both a transformation (via a scale and/or
> >> offset) and missing values should first check that a data value is
> >> valid, and then apply the transformation. Note that values that are
> >> identified as missing should not be transformed. Since the missing
> >> value is outside the valid range it is possible that applying a
> >> transformation to it could result in an invalid operation. For
> >> example, the default _FillValue is very close to the maximum
> >> representable value of IEEE single precision floats, and multiplying
> >> it by 100 produces an "Infinity" (using single precision arithmetic)."
> >>
> >> and 8.1:
> >>
> >> "When data to be packed contains missing values the attributes that
> >> indicate missing values (_FillValue, valid_min, valid_max,
> >> valid_range) must be of the same data type as the packed data. See
> >> Section 2.5.1, “Missing Data” for a discussion of how applications
> >> should treat variables that have attributes indicating both missing
> >> values and transformations defined by a scale and/or offset."
> >>
> >> Which seem to indicate the _FillValue should be in the packed data
> >> form (byte or short for example), which seems contrary to:
> >>
> >> "# The _FillValue attribute should have the same data type as the
> >> variable it describes."
> >>
> >> which I take to mean the target (unpacked) variable (like a float or 
> >> double).
> >>
> >> But perhaps I am misinterpreting this?
> >
> > Yes, it's intended to mean the data type of the variable (if unpacked)
> > or the data type of the packed variable (if packed).  I've tried to
> > clarify it in the Best Practices document:
> >
> >  
> > http://www.unidata.ucar.edu/netcdf/docs/BestPractices.html#Missing%20Data%20Values
> >
> > Thanks for pointing out the part the lack of clarity.
> >
> > --Russ
> >
> >
> >> tom
> >>
> >> On Thu, Mar 18, 2010 at 10:41 PM, Unidata netCDF Java Support
> >> <address@hidden> wrote:
> >> > Russ responded:
> >> >
> >> > There is discussion of reserving a packed value for missing data in
> >> > the Best Practices document, where it says:
> >> >
> >> >  
> >> > http://www.unidata.ucar.edu/netcdf/docs/BestPractices.html#Packed%20Data%20
> >> Values
> >> >
> >> >  # In either the signed or unsigned case, an alternate formula may be
> >> >    used for the add_offset and scale_factor packing parameters that
> >> >    reserves a packed value for a special value, such as an indicator of
> >> >    missing data. For example, to reserve the minimum packed value
> >> >    (-2^(n - 1)) for use as a special value in the case of signed packed
> >> >    values:
> >> >
> >> >    scale_factor =(dataMax - dataMin) / (2^n - 2)
> >> >
> >> >    add_offset = (dataMax + dataMin) / 2
> >> >
> >> >  # If the packed values are unsigned, then the analogous formula that
> >> >    reserves 0 as the packed form of a special value would be:
> >> >
> >> >    scale_factor =(dataMax - dataMin) / (2^n - 2)
> >> >
> >> >    add_offset = dataMin - scale_factor
> >> >
> >> > Later in the same document, under the "Missing Data Values" secttion, it
> >> > says
> >> >
> >> >  
> >> > http://www.unidata.ucar.edu/netcdf/docs/BestPractices.html#Missing%20Data%2
> >> 0Values
> >> >
> >> >  # The _FillValue attribute should have the same data type as the
> >> >    variable it describes.
> >> >
> >> >> One of our scientists is trying to do the "right thing" - use CF
> >> >> conventions for creating some files.  He asked me about how to specify
> >> >> the _FillValue when the data values are packed, with scale_factor and
> >> >> add_offset attribute set.
> >> >>
> >> >> I told him I believed that the _FillValue (and valid_range) were
> >> >> specified for the target variable; however, I noticed a comment in the
> >> >> documentation that indicated if the _FillValue was specified in the
> >> >> "type" of the packed data (shorts in this case), then it was applied
> >> >> before the un-packing.  Is that true?  And in this case, is NaN the
> >> >> value used?
> >> >
> >> > yes. a NaN cant be used for a short.
> >> >
> >> >>
> >> >> In addition, he asked whether there was a way to specify the value
> >> >> used for "missing" for the unpacked data.  He'd apparently like to put
> >> >> -999.0f into the arrays, instead of whatever the value would otherwise
> >> >> be.  I told him I would ask, since I didn't know...
> >> >
> >> > the CDM stack allows that, but its non-standard and i wouldnt recommend 
> >> > it.
> >> >
> >> >>
> >> >> Finally, since he's creating these files for use in IDV and Matlab, he
> >> >> asked whether there were differences between the Java and C/Fortran
> >> >> environments.  The only one I thought of is that the Java API will
> >> >> unpack the values, but the C/Fortran leave that up to the application .
> >> >> Is that still true?
> >> >
> >> > yes, the Java library will optionally convert them.
> >> >
> >> >>
> >> >> I must also ask about the "missing_value" attribute.  A couple of
> >> >> years ago, we talked about this and the last note I saw was that you
> >> >> were going to try to convince the CF folks it was a mistake to
> >> >> deprecate this and you would continue to support it in the libraries
> >> >> ad infinitum.  Has that changed?
> >> >
> >> > i often recommend missing_value, its much clearer that _FillValue, which 
> >> > re
> >> ally means "never written to". We have removed the missing_value 
> >> deprecation
> >> in the Users Guide. I keep forgetting to ask CF to remove that, I think I 
> >> wil
> >> l do it right now....
> >> >
> >> >
> >> >>
> >> >> Thanks ahead....
> >> >>
> >> >> tom
> >> >>
> >> >> --
> >> >> Tom Whittaker
> >> >> University of Wisconsin-Madison
> >> >> Space Science & Engineering Center (SSEC)
> >> >> Cooperative Institute for Meteorological Satellite Studies (CIMSS)
> >> >> 1225 W. Dayton Street
> >> >> Madison, WI  53706  USA
> >> >> ph: +1 608 262 2759
> >> >>
> >> >>
> >> >
> >> >
> >> > Ticket Details
> >> > ===================
> >> > Ticket ID: BUZ-409451
> >> > Department: Support netCDF Java
> >> > Priority: Normal
> >> > Status: Closed
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Tom Whittaker
> >> University of Wisconsin-Madison
> >> Space Science & Engineering Center (SSEC)
> >> Cooperative Institute for Meteorological Satellite Studies (CIMSS)
> >> 1225 W. Dayton Street
> >> Madison, WI  53706  USA
> >> ph: +1 608 262 2759
> >>
> >>
> >>
> >> Ticket Details
> >> ===================
> >> Ticket ID: BUZ-409451
> >> Department: Support netCDF Java
> >> Priority: Normal
> >> Status: Open
> >> Link:  
> >> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=vi
> >> ewticket&ticketid=10850
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: BUZ-409451
> > Department: Support netCDF Java
> > Priority: Normal
> > Status: Open
> >
> >
> 
> 
> 
> --
> Tom Whittaker
> University of Wisconsin-Madison
> Space Science & Engineering Center (SSEC)
> Cooperative Institute for Meteorological Satellite Studies (CIMSS)
> 1225 W. Dayton Street
> Madison, WI  53706  USA
> ph: +1 608 262 2759
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: BUZ-409451
Department: Support netCDF Java
Priority: Normal
Status: Closed