Re: Catalog generator crawling remote OPenDAP/DODS servers

Hi Valentijn,

Valentijn Venus wrote:

> Hi Ethan,
>
> First a fundamental questions related to THREDDS: the idea for the
> catalog generator is to crawl whatever source of data you have (local or
> remote) and generate a catalog, correct?
That is the idea but unfortunately crawling a local file system is all 
that is currently implemented.

> Now I'm looking for some
> directions on how to explore all sub-directories on a remote DODS
> server. The following servers I have in mind:
>
> MODIS AQUA/TERRA:
> http://g0dup05u.ecs.nasa.gov/OPENDAP_DP/long_term/MOAA/MYD04_L2.004/2002
> .07.16/
> TOMS Ozone:
> http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/T
> OMS/EP/ 
>
> Recently they moved these servers, and since then only the DODS Dataset
> Access Form for TOMS seems to work. When accessing one of these HDF
> files (i.e.
> http://reason.gsfc.nasa.gov/opendap-bin/nph-dods/FTP_DATA/Giovanni/OPS/T
> OMS/EP/TOMS-EP_L3-TOMSEPL3_1996m0725_v8.HDF.html) and once the form is
> setup, I can see the resulting "get"/"post" parameters trailing the "?"
> in a DODS url (...?Ozone[0:1:179][0:1:287]) How does this work once the
> catalog is generated, and a client (IDV, Matlab, etc.) accesses it in
> real-time? I guess THREDDS supports geographic/paramter subsetting in a
> similair manner, but how does it work technically?
>  
Generally, the THREDDS catalog provides the base DODS URL and the client 
needs to understand the OPeNDAP/DODS protocol (the ".dds", ".das", 
".dods" extensions and all the stuff after the "?"). THREDDS allows for 
metadata to indicate the geographic coverage of a dataset but it doesn't 
support subsetting a particular dataset. The actual subsetting all 
dependent on the protocol being used to access the dataset, in this case 
OPeNDAP.


Back to the catalog generation. At one point I did have OPeNDAP server 
crawling capabilities in the catalog generator but no one was using it 
and I don't think it is still working after a few major modifications. 
Are you using the stand-alone CatalogGen application, the catalog 
generation servlet interface, or the newer TDS configuration which also 
generates catalogs? The TDS config is a completely different framework 
than that used by the app and servlet. The code (non-TDS framework) for 
crawling OPeNDAP servers is still around if you want to look into 
getting it running again. However, I'm planning on moving the CatGen app 
and servlet over to the new TDS framework at some point but that may be 
awhile.

Hope this helps.

Ethan

> Cheers, Valentijn
>  

--
Ethan R. Davis                                Telephone: (303) 497-8155
Software Engineer                             Fax:       (303) 497-8690
UCAR Unidata Program Center                   E-mail:    address@hidden
P.O. Box 3000
Boulder, CO  80307-3000                       http://www.unidata.ucar.edu/
---------------------------------------------------------------------------