Re: NaN, _FillValue and the FMRC

Hi all:

FMRC is different from other aggregations in that it operates on 
GridDatasets rather than NetcdfFiles. The default enhance mode of 
GridDataset is to add scale/offset and convert missing to NaNs. 
Currently that setting can only be changed globally. Also, theres a bug 
in the code allowing non-default enhancements in 4.1, that i may not be 
able to fix until 4.2.

Non-FMRC aggregations dont do any enhancements unless you ask for it in 
the NcML.

Whenever scale/offset is applied, the attributes scale_factor and 
add_offset are removed, so there shouldnt be the danger of applying 
twice. Since other (non-CDM) client libraries dont apply scale/offset, 
it seems like the best default is to convert on the server. Performance 
is typically (eg IDV) limited by latency, not bandwidth, so the 2X 
increase in size hasnt been a problem. You may have use cases where its 
more important.

Theres a lot of complexity around the enhancement code, and I will have 
to look carefully at what I can support. Id like to hear your "gotta 
have" needs, plus your "would be nice if it doesnt make the code 
unstable" wishes.

John

On 4/27/2010 10:33 AM, Rich Signell wrote:
> Steve,
>
>    
> > Will wait to hear from John.  My  inclination would be that the aggregation
> > process ought not to cause _FillValue, scale_factor, add_offset and data
> > type to be presented differently than they would have been in the original
> > unaggregated files.
> >      
> And I would agree.  I've been burned when people applied the wrong
> scale_factor and add_offsets, and all I knew was that the values
> looked funny (I didn't know they even had scale_factor and
> add_offsets, but it eventually came out on investigation).   And
> doesn't it take twice as long to deliver data over opendap if the data
> is float instead of short?
>
> -Rich
>
>    
> >      - Steve
> >
> > If you look at these original files (for example)
> >
> > http://rocky.umeoce.maine.edu:8080/thredds/dodsC/gompom/operational/201004/gomoos.20100427.cdf.html
> >
> > you can see that the variable "temp" is a "short integer" with
> > "scale_factor","add_offset" and "missing_value", while the FMRC "best
> > time series"
> >
> > http://rocky.umeoce.maine.edu:8080/thredds/dodsC/gomoos/operational_model/UMaine_GoMOOS_cirulation_model_best.ncd.html
> >
> > now the variable "temp" is a "float" with none of those attributes,
> > only NaN values.
> >
> > I'm CC'ing John Caron, just to make sure I've got this right.
> >
> >
> > -Rich
> >
> > On Tue, Apr 6, 2010 at 7:22 PM, Kevin O'Brien<Kevin.M.O'address@hidden>
> > wrote:
> >
> >
> > Hi Rich -
> >
> > I added some of your USGS best time series data to the UAF clean catalog at:
> >
> >    http://ferret.pmel.noaa.gov/geoide/geoIDECleanCatalog.html
> >
> > I think we'll find that this is also a case where "NaN" is used for the
> > missing value, but not specified in the variable attributes from the best
> > time series. Maybe we should ask John Caron why the missing value of NaN
> > doesn't get set by default as an attribute in those datasets.   At any rate,
> > I'll be interested to hear what you have to report as far as performance
> > goes on those COAWST data...
> >
> > Bob - I'm cc'ing you because you wanted to know when the catalog was
> > changed.   I've also added some NOAA coastwatch aggregations to the
> > catalog...
> >
> > Let me know if there are any questions..
> >
> > Kevin
> >
> > --
> > Kevin O'Brien                   UW/JISAO
> > Research Scientist              NOAA/PMEL/TMAP
> > 206-526-6751                    http://www.pmel.noaa.gov
> >
> > "The contents of this message are mine personally and do
> > not necessarily reflect any position of the Government
> > or the  National Oceanic and Atmospheric Administration."
> >
> >
> >
> >
> >
> >      
>
>
>