Re: [netCDFJava #KJR-472449]: missing/fill values and EnhanceScaleMissingImpl

> Thanks for the response, John.  Perhaps you or Russ could talk to the 
> NCL people and convince them to use missing_value instead of _FillValue 
> as their preferred attribute for missing data since NCL can read 
> multiple formats (including GrADS binary) and might run into the same 
> issues.
> 
> Don
> 
> On 7/29/11 2:55 PM, Unidata netCDF Java Support wrote:
> > Hi Don:
> >
> > I think Russ and I have talked about this and decided that _FillValue shoul
> d match the bit pattern exactly (for portability?). valid_range, valid_min, v
> alid_max, and missing value are intended to match data values and so should b
> e allowed some slop. Im cc'ing Russ in case he remembers different.

The way ncdump handles it (which seems to differ from the documented
requirement for exact bit-for-bit equality), is to treat any floating
point value that's within one machine epsilon of the _FillValue (that is
one bit in the least significant bit) as equal to _FillValue, and
display it as "_".  The code for this is in ncdump/dumplib.c, in the
ncfloat_val_equals() and corresponding double precision functions:

  /* 
   * Return ( *(float* )v1p == *(float* )v2p);
   * except use floating epsilon to compare very close vals as equal
   * and handle IEEE NaNs and infinities.
   */
  boolean 
  ncfloat_val_equals(const nctype_t *this, 
                     const void *v1p, const void *v2p) {
      float v1 = *(float* )v1p;
      float v2 = *(float* )v2p;
      if((v1 > 0.0f) != (v2 > 0.0f))    /* avoid overflow */
          return false;
      if(isfinite(v1) && isfinite(v2))
          return (absval(v1 - v2) <= absval(float_eps * v2)) ;
      if(isnan(v1) && isnan(v2))
          return true;
      if(isinf(v1) && isinf(v2))
          return true;
      return false;
  }

It looks like I should fix the documentation to match the behavior.  I
think it was necessary to add the 1-bit fuzz to make the ncdump tests
pass on every tested platform, but it may have been an artifact of bad
compiler code that's fixed now.  I may try taking it out to see if all
the ncdump and ncgen tests pass without it.

> >
> > My own opinion is that the trouble stems from overloading _FillValue with m
> ultiple meanings, plus the lack of clarity to tell people what they should do
> . We should probably make another pass at
> >
> > http://www.unidata.ucar.edu/software/netcdf/docs/BestPractices.html

Yup.

--Russ

> > John
> >
> >> John-
> >>
> >> In the GrADS IOSP, I added a _FillValue attribute for the GrADS Data
> >> Descriptor File "undef" value:
> >>
> >> http://www.iges.org/grads/gadoc/descriptorfile.html#UNDEF
> >>
> >> I used _FillValue instead of missing_value because it seemed more
> >> accepted in the CF conventions until the recent change to undeprecate
> >> missing_value.  Also, NCL prefers _FillValue over missing_value and I
> >> was working with NCL at the time I wrote the IOSP.
> >>
> >> I came across a file today where the undef was set to 9.99E8, but the
> >> "missing" values in the file ranged from 9.9899994E8 to 9.99E8 (We're
> >> looking into why it is inconsistent - the file is written by GrADS).
> >> Talking to one of the developers, they always use 32-bit values (floats)
> >> for the data when working with GrADS and I'm using float for the
> >> variable type.
> >>
> >> The logic in EnhanceScaleMissingImpl is different for _FillValue vs.
> >> missing_value.  isMissingValue uses ucar.nc2.util.Misc.closeEnough
> >> whereas isFillValue uses straight equality.
> >>
> >> I can change the IOSP to have both _FillValue and missing_value
> >> attributes, or change _FillValue to missing_value and I get the results
> >> I expect.  However, it seems like there may be some slop even in fill
> >> values, so perhaps isFillValue should use Misc.closeEnough instead of
> >> straight equality which also solves this problem.  That seems more
> >> prudent for the typically large values that are used as _FillValues.
> >>
> >> Thoughts?
> >>
> >> Don
> >> --
> >> Don Murray
> >> NOAA/ESRL/PSD and CIRES
> >> 303-497-3596
> >> http://www.esrl.noaa.gov/psd/people/don.murray/
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: KJR-472449
> > Department: Support netCDF Java
> > Priority: Critical
> > Status: Open
> >
> 
> -- 
> Don Murray
> NOAA/ESRL/PSD and CIRES
> 303-497-3596
> http://www.esrl.noaa.gov/psd/people/don.murray/