This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
> Thanks, Russ....the new language clarifies that. Might I also suggest > adding a clarification to the "valid_range" just below? Done, thanks! --Russ > > On Mon, Mar 22, 2010 at 3:20 PM, Unidata netCDF Java Support > <address@hidden> wrote: > >> New Client Reply: Question about _FillValue for packed data > >> > >> Russ -- > >> > >> Thanks for your note. I had used those two references in "Best > >> Practices", but then also found some seemingly contradictory > >> statements in the CF1.4 (section 2.5.1): > >> > >> "The missing values of a variable with scale_factor and/or add_offset > >> attributes (see section Section 8.1, “Packed Data”) are interpreted > >> relative to the variable's external values, i.e., the values stored in > >> the netCDF file. Applications that process variables that have > >> attributes to indicate both a transformation (via a scale and/or > >> offset) and missing values should first check that a data value is > >> valid, and then apply the transformation. Note that values that are > >> identified as missing should not be transformed. Since the missing > >> value is outside the valid range it is possible that applying a > >> transformation to it could result in an invalid operation. For > >> example, the default _FillValue is very close to the maximum > >> representable value of IEEE single precision floats, and multiplying > >> it by 100 produces an "Infinity" (using single precision arithmetic)." > >> > >> and 8.1: > >> > >> "When data to be packed contains missing values the attributes that > >> indicate missing values (_FillValue, valid_min, valid_max, > >> valid_range) must be of the same data type as the packed data. See > >> Section 2.5.1, “Missing Data” for a discussion of how applications > >> should treat variables that have attributes indicating both missing > >> values and transformations defined by a scale and/or offset." > >> > >> Which seem to indicate the _FillValue should be in the packed data > >> form (byte or short for example), which seems contrary to: > >> > >> "# The _FillValue attribute should have the same data type as the > >> variable it describes." > >> > >> which I take to mean the target (unpacked) variable (like a float or > >> double). > >> > >> But perhaps I am misinterpreting this? > > > > Yes, it's intended to mean the data type of the variable (if unpacked) > > or the data type of the packed variable (if packed). I've tried to > > clarify it in the Best Practices document: > > > > > > http://www.unidata.ucar.edu/netcdf/docs/BestPractices.html#Missing%20Data%20Values > > > > Thanks for pointing out the part the lack of clarity. > > > > --Russ > > > > > >> tom > >> > >> On Thu, Mar 18, 2010 at 10:41 PM, Unidata netCDF Java Support > >> <address@hidden> wrote: > >> > Russ responded: > >> > > >> > There is discussion of reserving a packed value for missing data in > >> > the Best Practices document, where it says: > >> > > >> > > >> > http://www.unidata.ucar.edu/netcdf/docs/BestPractices.html#Packed%20Data%20 > >> Values > >> > > >> > # In either the signed or unsigned case, an alternate formula may be > >> > used for the add_offset and scale_factor packing parameters that > >> > reserves a packed value for a special value, such as an indicator of > >> > missing data. For example, to reserve the minimum packed value > >> > (-2^(n - 1)) for use as a special value in the case of signed packed > >> > values: > >> > > >> > scale_factor =(dataMax - dataMin) / (2^n - 2) > >> > > >> > add_offset = (dataMax + dataMin) / 2 > >> > > >> > # If the packed values are unsigned, then the analogous formula that > >> > reserves 0 as the packed form of a special value would be: > >> > > >> > scale_factor =(dataMax - dataMin) / (2^n - 2) > >> > > >> > add_offset = dataMin - scale_factor > >> > > >> > Later in the same document, under the "Missing Data Values" secttion, it > >> > says > >> > > >> > > >> > http://www.unidata.ucar.edu/netcdf/docs/BestPractices.html#Missing%20Data%2 > >> 0Values > >> > > >> > # The _FillValue attribute should have the same data type as the > >> > variable it describes. > >> > > >> >> One of our scientists is trying to do the "right thing" - use CF > >> >> conventions for creating some files. He asked me about how to specify > >> >> the _FillValue when the data values are packed, with scale_factor and > >> >> add_offset attribute set. > >> >> > >> >> I told him I believed that the _FillValue (and valid_range) were > >> >> specified for the target variable; however, I noticed a comment in the > >> >> documentation that indicated if the _FillValue was specified in the > >> >> "type" of the packed data (shorts in this case), then it was applied > >> >> before the un-packing. Is that true? And in this case, is NaN the > >> >> value used? > >> > > >> > yes. a NaN cant be used for a short. > >> > > >> >> > >> >> In addition, he asked whether there was a way to specify the value > >> >> used for "missing" for the unpacked data. He'd apparently like to put > >> >> -999.0f into the arrays, instead of whatever the value would otherwise > >> >> be. I told him I would ask, since I didn't know... > >> > > >> > the CDM stack allows that, but its non-standard and i wouldnt recommend > >> > it. > >> > > >> >> > >> >> Finally, since he's creating these files for use in IDV and Matlab, he > >> >> asked whether there were differences between the Java and C/Fortran > >> >> environments. The only one I thought of is that the Java API will > >> >> unpack the values, but the C/Fortran leave that up to the application . > >> >> Is that still true? > >> > > >> > yes, the Java library will optionally convert them. > >> > > >> >> > >> >> I must also ask about the "missing_value" attribute. A couple of > >> >> years ago, we talked about this and the last note I saw was that you > >> >> were going to try to convince the CF folks it was a mistake to > >> >> deprecate this and you would continue to support it in the libraries > >> >> ad infinitum. Has that changed? > >> > > >> > i often recommend missing_value, its much clearer that _FillValue, which > >> > re > >> ally means "never written to". We have removed the missing_value > >> deprecation > >> in the Users Guide. I keep forgetting to ask CF to remove that, I think I > >> wil > >> l do it right now.... > >> > > >> > > >> >> > >> >> Thanks ahead.... > >> >> > >> >> tom > >> >> > >> >> -- > >> >> Tom Whittaker > >> >> University of Wisconsin-Madison > >> >> Space Science & Engineering Center (SSEC) > >> >> Cooperative Institute for Meteorological Satellite Studies (CIMSS) > >> >> 1225 W. Dayton Street > >> >> Madison, WI 53706 USA > >> >> ph: +1 608 262 2759 > >> >> > >> >> > >> > > >> > > >> > Ticket Details > >> > =================== > >> > Ticket ID: BUZ-409451 > >> > Department: Support netCDF Java > >> > Priority: Normal > >> > Status: Closed > >> > > >> > > >> > >> > >> > >> -- > >> Tom Whittaker > >> University of Wisconsin-Madison > >> Space Science & Engineering Center (SSEC) > >> Cooperative Institute for Meteorological Satellite Studies (CIMSS) > >> 1225 W. Dayton Street > >> Madison, WI 53706 USA > >> ph: +1 608 262 2759 > >> > >> > >> > >> Ticket Details > >> =================== > >> Ticket ID: BUZ-409451 > >> Department: Support netCDF Java > >> Priority: Normal > >> Status: Open > >> Link: > >> https://www.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=vi > >> ewticket&ticketid=10850 > > > > > > > > Ticket Details > > =================== > > Ticket ID: BUZ-409451 > > Department: Support netCDF Java > > Priority: Normal > > Status: Open > > > > > > > > -- > Tom Whittaker > University of Wisconsin-Madison > Space Science & Engineering Center (SSEC) > Cooperative Institute for Meteorological Satellite Studies (CIMSS) > 1225 W. Dayton Street > Madison, WI 53706 USA > ph: +1 608 262 2759 > > Russ Rew UCAR Unidata Program address@hidden http://www.unidata.ucar.edu Ticket Details =================== Ticket ID: BUZ-409451 Department: Support netCDF Java Priority: Normal Status: Closed