[IDV #BER-255195]: IDV - IDV and subsetting Grids from OpenDAP/THREDDS

Hi Valentijn-

> Institution: ITC
> Package Version: 2.1b1
> Operating System: os.name:Windows XP; os.arch:x86; os.version:5.1;
> Hardware Information: java.vendor:Sun Microsystems Inc.; 
> java.version:1.6.0-beta2; java.home:D:\\Program Files\\Java\\jre1.6.0; 
> j3d.version:1.3.2 fcs (build12); j3d.vendor:Sun Microsystems, Inc.; 
> j3d.renderer:OpenGL;
> Inquiry: 
> Dear support,
> 
> I'm running the "nightly" buid of the IDV and If you select "I'm still 
> feeling lucky" in Data Chooser / Catalog, when you are about to load the 
> data, you will have the "Omni Control" as the only available option for 
> displaying the data.
> However, if you select "Grids from an OPeNDAP server" instead, you will get 
> the advanced from, where you are able to select (among others) the field 
> needed, the spatial subset and the aggregation, which is exactly what you 
> want.

I've been playing with this with some of the recent inquiries you sent
in.  There are some bugs in the netCDF Java library and IDV that we've been
working on to make this work better.  However, as you are finding, not
all OPeNDAP datasets are created equal and some work and some don't.
Also, for those that use the netCDF layer, you don't need the .dods extension.

> I found that this works for the Pathfinder data. An example URL is:
> http://data.nodc.noaa.gov/cgi-bin/nph-dods/pathfinder/Version5.0/8day/2001/2001001-2001008.s0481pfv50-sst-16b.hdf

This is one of the datasets that has uncovered some problems.  First,
as you found, the navigation is off because the netCDF layer was not
giving the correct units for latitude and longitude.  Secondly,
there is a scaling bug in the library which causes the values to
be scaled twice.  This is fixed in the next version of the netCDF
library.  Unfortunately, we can't use that yet. Lastly, there is 
an "add_off" attribute for these values which does not get used 
because the standard attribute name for netCDF is "add_offset".

> It does not work for the other OPeNDAP datasets e.g. TOMS/EP:
> http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/EP/TOMS-EP_L3-TOMSEPL3_2005m0409_v8.HDF

I'll pass this along to the netCDF folks.  At the very least, it should
fail gracefully instead of throwing a NullPointerException.  I think
the problem is that the x and y dimension variables cannot be forced
to be latitude and longitude, i.e., how is one to know that 
XDim%3aTOMS%20Level%203 is Longitude and  YDim%3aTOMS%20Level%203
is latitude based on the metadata in the dataset?

> So the last case i have to "hardcode" the geographic/parameter subsetting and 
> choose the lucky load option. The url the looks as follows:
> http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/TOMS/EP/TOMS-EP_L3-TOMSEPL3_2005m0409_v8.HDF.dods?Erythemal[0:1:179][0:1:287]

The netCDF layer cannot handle the constraint expressions.
 
> My main conclusion from this experiment has been that the so-called 
> right-part of the URL (everything after the ?), should be generated by IDV. 
> IDV has such provision, as long as you tell it that it is dealing with an 
> OPeNDAP (DODS) server. The problem sometimes is that we have encountered an 
> OPeNDAP server that is not compatible with IDV (like TOMS/EP). IDV seems to 
> expect some parameters into the .das or .dds responses of DODS servers, which 
> can not be found.

Correct.  Is there some documentation somewhere on the conventions used
for these datasets that would allow a mapping of variables like the
x and y to latitude and longitude?

> 
> Question 1:
> In such a case, can ncml "enhance" the way our THREDDS server interpretes the 
> HDF/netCDF DODS server (e.g by reinterpreting the variables/units or by 
> adding metadata) necessary for IDV to show the spatial subsetting GUI?

I'll defer that to the netCDF group.

> Question 2:
> Hopefully this is a bug, but spatial subsetting of the Pathfinder Gridded SST 
> seems to confuse the IDV  (see the attached bundle). Maybe a 
> projection/navigation issue?

This will be fixed in tomorrow's build (9/7), however the scaling issue
still remains.  I hope to have that resolved next week, but upgrading
to the next version of netCDF is problematic because some of the
API's changed.

Don

Ticket Details
===================
Ticket ID: BER-255195
Department: Support IDV
Priority: Normal
Status: Open