20040817: MODIS DODS dataset problems with IDV version 1.1

>From: "Valentijn Venus" <address@hidden>
>Organization: ITC
>Keywords: 200408171235.i7HCZIaW012100  IDV MODIS HDF

Hi Valentijn-

>i remember asking the same question before, but about a year ago you added sup
> port for the MODIS DODS dataset to IDV. 

Actually, we fixed a bug that was causing problems getting at the data. ;-)

> Now with the new IDV release when i e
> nter the following url as a data source i get the requisted parameter (i.e. "
> surface_temperature"), but can only display the data source in an omni contro
> l (and not any longer as a map):
> 
>valid_range: 0, 20000
>_FillValue: -32768
>long_name: "Surface Temperature"
>units: "K"
>scale_factor: 0.01
>add_offset: -15000

Was this working before with just a opening as a URL?  From our support
archives, when you asked before, I wrote:

> Basically, I wrote a Jython script that would take the VisAD
> data object returned by the DODS request and refactor it into
> something useful.

We didn't add in conventions to read the data directly.  But now the
Jython script works.  Here's what you do:

1) Use the Edit->Formulas->Jython Library menu to open up the 
Jython library editor.

2) paste the following Jython routine into you editor.  You'll need
to fix any line breaks:

import sys;
sys.add_package('visad');

def makeHDFImage(url):
   from visad import *
   lat = a[0][0]
   xyLen = lat.getDomainSet().getLengths()
   lon = a[0][1]
   values = a[1]
   latVals = lat.getFloats(0)[0]
   lonVals = lon.getFloats(0)[0]
   domain = Gridded2DSet(RealTupleType.SpatialEarth2DTuple,
                        (lonVals, latVals), xyLen[0], xyLen[1],
                        None, None, None, 0, 0 )
   image = 
FlatField(FunctionType(domain.getType().getDomain(),makeRealType("values")), 
domain)
   image.setSamples(values.getFloats(0),0)
   # dumpTypes(image)
   return image

3) From the Data Selector, right click on the "Formula" Data Source and
choose the "Create Formula" menu item to bring up the formula editor.

4) In the formula editor, enter HDFImage for the name, and make the
formula:

makeHDFImage(a)

Click the Advanced button to open the advanced options (if they are not
already open).  Under Displays, click the Use Selected button, then
click the All off button.  Expand the Plan Views category and check
"Color Shaded Plan View" checkbox.

5) Now, load in the URL of the DODS dataset through the URL chooser.
I found that the URL you gave:

> 
>http://g0dug03u.ecs.nasa.gov/dods-bin/nph-dods/datapool/MOAT/MOD07_L2.004/2004
> .08.15/MOD07_L2.A2004228.1120.004.2004228223236.hdf.dods?Latitude[400:405][26
> 0:269],Longitude[400:405][260:269],Surface_Temperature[400:405][260:269]

Had all missing values.  I had success with the Water Vapor variable:

http://g0dug03u.ecs.nasa.gov/dods-bin/nph-dods/datapool/MOAT/MOD07_L2.004/2004.08.15/MOD07_L2.A2004228.1120.004.2004228223236.hdf?Latitude[0:1:405][0:1:269],Longitude[0:1:405][0:1:269],Water_Vapor[0:1:405][0:1:269]


>In ASCII the data looks like this:
>Dataset: MOD07_L2.A2004228.1120.004.2004228223236.hdf
>mod07.Geolocation Fields.Latitude[0], 37.9526252746582, 37.9059715270996, 37.8
> 576240539551, 37.8074760437012, 37.7554054260254, 37.7012786865234, 37.644958
> 4960938, 37.5862770080566, 37.5250625610352, 37.4611167907715
>mod07.Geolocation Fields.Latitude[1], 37.8776969909668, 37.8301811218262, 37.7
> 809448242188, 37.7298736572266, 37.6768493652344, 37.6217384338379, 37.564388
> 2751465, 37.5046424865723, 37.4423217773438, 37.3772277832031
>mod07.Geolocation Fields.Latitude[2], 37.8681716918945, 37.8216133117676, 37.7
> 733688354492, 37.7233276367188, 37.6713676452637, 37.6173629760742, 37.561164
> 855957, 37.502613067627, 37.441535949707, 37.3777389526367
>mod07.Geolocation Fields.Latitude[3], 37.7932357788086, 37.7458152770996, 37.6
> 966819763184, 37.6457176208496, 37.5928039550781, 37.5378074645996, 37.480583
> 190918, 37.420970916748, 37.3587875366211, 37.2938385009766
>mod07.Geolocation Fields.Latitude[4], 37.7831153869629, 37.7366333007812, 37.6
> 884689331055, 37.6385116577148, 37.5866432189941, 37.5327301025391, 37.476631
> 1645508, 37.4181823730469, 37.3572158813477, 37.293529510498
>mod07.Geolocation Fields.Latitude[5], 37.7081565856934, 37.660816192627, 37.61
> 1759185791, 37.5608787536621, 37.508056640625, 37.4531517028809, 37.396026611
> 3281, 37.3365135192871, 37.2744369506836, 37.2095985412598
>mod07.Geolocation Fields.Longitude     [0], 3.70441126823425, 3.89282870292664
> , 4.08659362792969, 4.28601360321045, 4.49142265319824, 4.70318460464478, 4.9
> 2169857025146, 5.14740133285522, 5.38076782226562, 5.62231779098511
>mod07.Geolocation Fields.Longitude[1], 3.67430353164673, 3.86217284202576, 4.0
> 5536794662476, 4.2541937828064, 4.45898389816284, 4.67010068893433, 4.8879404
> 0679932, 5.11293792724609, 5.3455662727356, 5.58634185791016
>mod07.Geolocation Fields.Longitude[2], 3.66348171234131, 3.85169744491577, 4.0
> 4525184631348, 4.24445199966431, 4.44963073730469, 4.66115236282349, 4.879413
> 60473633, 5.10485124588013, 5.33794116973877, 5.5792064666748
>mod07.Geolocation Fields.Longitude[3], 3.6334662437439, 3.82113575935364, 4.01
> 412200927734, 4.21272993087769, 4.41729211807251, 4.62817001342773, 4.8457593
> 9178467, 5.07049417495728, 5.30284833908081, 5.54334211349487
>mod07.Geolocation Fields.Longitude[4], 3.62472176551819, 3.81279706954956, 4.0
> 0620603561401, 4.20525550842285, 4.41027879714966, 4.62163829803467, 4.839732
> 17010498, 5.06499528884888, 5.29790592193604, 5.53899002075195
>mod07.Geolocation Fields.Longitude[5], 3.59479737281799, 3.78232789039612, 3.9
> 7517037391663, 4.17362976074219, 4.37803745269775, 4.58875560760498, 4.806179
> 04663086, 5.03074169158936, 5.2629189491272, 5.50323390960693
>
>mod07.Data Fields.Surface_Temperature[0], -32768, -32768, -32768, -32768, -327
> 68, -32768, 14449, 14699, 14654, -32768
>mod07.Data Fields.Surface_Temperature[1], -32768, -32768, -32768, -32768, -327
> 68, -32768, -32768, -32768, 14716, -32768
>mod07.Data Fields.Surface_Temperature[2], -32768, -32768, -32768, -32768, -327
> 68, -32768, -32768, 14622, 14723, -32768
>mod07.Data Fields.Surface_Temperature[3], -32768, -32768, -32768, -32768, -327
> 68, -32768, -32768, 14613, 14673, -32768
>mod07.Data Fields.Surface_Temperature[4], -32768, -32768, -32768, -32768, -327
> 68, -32768, -32768, 14596, 14637, -32768
>mod07.Data Fields.Surface_Temperature[5], -32768, -32768, -32768, -32768, -327
> 68, -32768, -32768, -32768, 14650, -32768

Since these are scaled values, it doesn't look like the VisAD DODS adapter
is using that information.  It may be that it doesn't get that info
when the request is made The output above indicates that they are not
all missing.  I'll have to look at that further.

>Also, i have problems with the mappings. What ever i try, i'm not getting the 
> parameter in RGB (or contour interval), latitude as x, and longitude as y. Is
>  there any way i can solve this?

What you get back is a tuple of lat and lon values indexed by x and y
and the surface temperature indexed by x and y.  Since there is no
dependence of temp on lat and lon in this structure (i.e., they are all
dependent on x and y, but independent of each other), all you'll get
is points.  The Jython script rearranges the data into:

(lon, lat) -> temp

so now temp is a function of (lon, lat).

Note that the Jython code above only works for the 2D data structured as
((((x, y) -> Latitude), ((x, y) -> Longitude)), ((x,y) -> variable))

As I said, we need to have an adapter that can restructure the data
as we do with gridded data.  Unfortunately, we don't have the resources
to put on that right now.  I put in a request again to get this added.

Don
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.