Re: latest Catalog XML

Daniel Holloway wrote:
> 
> Ethan Davis wrote:
> 
> > Daniel Holloway wrote:
> > >
> > > ....
> > >
> > >     I've used the DTD listed above to create two examples of what
> > > I believe I could return in an XML-based dods-dir response.  The
> > > catalogs represent simple directory representations of dods-accessible
> > > data; level1.xml is a higher-level directory containing only subdirs
> > > and level2.xml is one of the subdirs containing the dods-accessible
> > > datafiles.
> > >
> > >     Both of these files validate, I get some warnings but mostly
> > > due to style issues, there are no errors.   These examples represent
> > > the minimum information that I could provide, there are ways I
> > > could augment this service to provide some of the additional fields
> > > represented in the DTD but this example is meant to illustrate a
> > > minimum set.
> > >
> > >     Points to note:
> > >
> > > 1: I'd need to provide 2 service elements, one for
> > > type = DODS, and one for type = Other, where Other represents
> > > the dods-dir service.   In many cases there will be both datafiles
> > > and subdirectories in a file system.   Do I need to have two
> > > service elements?   If so, in the 'Level2.xml' example how do
> > > I indicate which service to use without adding an explicit
> > > access element for each dataset element?
> >
> > In each dataset element you can indicate the service it uses with the
> > serviceName attribute; its value would be the same as the value of the
> > service
> > elements name attribute.
> >
> 
> OK
> 
> >
> > > 2:  I use 'collection' as a simple filesystem collection, there
> > > may not be any meaningful relation between the files contained
> > > in the directory other than they're accessible via the DODS DAP.
> > >
> > > 3:  Am I using 'catalogRef' correctly?  The intention is to
> > > indicate another collection level that should only be traversed
> > > at the user's request.
> >
> > catalogRef should reference a THREDDS catalog. Yours is referencing a
> > dods-dir
> > page. I've reworked and attached your examples (with fewer years in
> > level1.xml
> > and fewer example datasets in level2.1985.xml).
> 
> Actually I'm trying to reference a THREDDS catalog, that is, my intention
> is that the response from a dods-dir service request would be a valid THREDDS
> catalog and it may look as indicated in the original example.   Currently,
> the dods-dir service returns an html-encoded page, that service would be
> augmented to return a THREDDS catalog.  Granted a THREDDS catalog
> is meant to offer more than a simple filesystem view of the accessible files
> but the catalog DTD supports seems to support this use.

Sorry, my mistake. I didn't realize you meant for the dods-dir to return a
THREDDS catalog. That sounds good. Your original example should work fine. The
only change needed would be making the datasets reference a service.

> I may be misreading the examples which have been provided in earlier
> messages, but it seems that all of the 'hrefs' in the catalogRef examples
> use explict filenames, but as long as the response to the 'href' returns
> a valid THREDDS catalog representation, why can't the 'href' reference
> a service which returns such a catalog.   The use of 'service' in this
> case is somewhat different than the 'access' service that is encoded
> in the service element, though arguably there is the 'other' and 'catalog'
> types.
>
> If you allow this, then the catalog becomes much simpler for the dods-dir
> case, basically following my initial example.   Actually, I'm not sure how
> the THREDDS API or any other parsing application would know the
> difference in the href so long as it returned a valid xml representation.
> 

Yup. The 'href' can be any URL that returns a THREDDS catalog. In your top level
catalog you could even get rid of all the service elements because nothing in
that catalog references them.

> I think the example you provided below using both dataset and
> catalogRef elements to depict the filesystem layout is somewhat
> ambiguous since the relation between these two elements is
> implicit in this use, and not explicit in the DTD, such that any
> API and or parsing application would need to recognize this
> particular relationship.

Good point. Not a very clean representation.

> > 1) I changed the catalogRefs in level1.xml to point to the level2.*.xml
> > files.
> 
> A minor point but I'd prefer not to point to individual 'level#.xml' files in
> the references and just point to the service instance for invoking dods-dir
> on the underlying subdirs.  Not sure how to do that now.
> 
> >
> 
> >
> > 2) I added datasets in level1.xml that give access to the dods-dir pages.
> > 3) I added a serviceName attribute in the dataset elements in the
> > level2.*.xml
> > files.
> >
> > A few alternatives:
> >
> > 1) You could have just one level. Each year could be a collection that
> > contains
> > the individual datasets.
> 
> The catalog I'm representing with the dods-dir response is a view
> of the filesystem tree for an arbitrary branch or leaf of the
> filesystem, and for only dods-accessible datafiles at that branch
> or leaf.   I definitely don't want to populate the complete filesystem
> catalog at a high level but want the client app or UI to traverse
> the references to finally get the complete list of datafiles.  Our site
> has approx. 100K datafiles accessible via DODS, so I need to
> limit the amount of information I transmit to a reasonable amount
> unless explictly requested by the client.
> 
> >
> >
> > 2) Again, just one level. Each year is a dataset with the dods-dir access
> > and
> > contains sub-datasets for each individual dataset.
> >
> > I'm working on the THREDDS catalog generator tool. Currently I'm working on
> > expanding DODS file servers into THREDDS catalogs. After I get that working
> > I'd
> > like to work on crawling dods-dir pages. So, I'll be wanting to pick your
> > brains
> > on this stuff.
> >
> 
> My plan is to augment dods-dir to return an xml-encoded response, hopefully
> a THREDDS catalog representation, then you won't have to crawl dods-dir pages
> just traverse those catalog representations.   We could then discuss how to
> extend the dods-dir service, possibly to read some config.xml or other
> ancillary
> info sources on the server to provide the additional information needed to
> extend these simple filesystem catalog views into more functional catalog
> representations.

Sounds great. Sorry for the confusion. Your one step (at least) ahead of me.

Ethan


> >   ------------------------------------------------------------------------
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE catalog SYSTEM
> > "http://www.unidata.ucar.edu/projects/THREDDS/xml/InvCatalog.0.6d.dtd";>
> > <catalog name="htn_sst_decloud" version="0.6d">
> >    <collection name="htn_sst_decloud/">
> >       <service name="URI-DODS" serviceType="DODS"
> >                base="http://dods.gso.uri.edu/dods-3.2/nph-dods/"/>
> >       <service name="DODSdirectory" serviceType="Other"
> >                      base="http://dods.gso.uri.edu/dods-3.2/nph-dods/"/>
> >
> >       <dataset name="DODS_dir of htn_sst_decloud/1985/"
> >                serviceName="DODSdirectory"/>
> >       <catalogRef xlink:title="htn_sst_decloud/1985/"
> >                   xlink:href="./level2.1985.xml"/>
> >
> 
> >
> >       <dataset name="DODS_dir of htn_sst_decloud/1986/"
> >                serviceName="DODSdirectory"/>
> >       <catalogRef xlink:title="htn_sst_decloud/1986/"
> >                   xlink:href="./level2.1986.xml"/>
> >
> >    </collection>
> > </catalog>
> >
> >   ------------------------------------------------------------------------
> > <?xml version="1.0" encoding="UTF-8"?>
> > <!DOCTYPE catalog SYSTEM
> > "http://www.unidata.ucar.edu/projects/THREDDS/xml/InvCatalog.0.6d.dtd";>
> > <catalog name="htn_sst_decloud/1985" version="0.6d">
> >   <collection name="htn_sst_decloud/1985">
> >      <service name="URI-DODS" serviceType="DODS"
> >               base="http://dods.gso.uri.edu/dods-3.2/nph-dods/"/>
> >      <service name="DODSDirectory" serviceType="Other"
> >               base="http://dods.gso.uri.edu/dods-3.2/nph-dods/"/>
> >      <dataset name="k85001182828.htn_d" serviceName="URI-DODS"
> >               urlPath="htn_sst_decloud/1985/k85001182828.htn_d.Z"/>
> >      <dataset name="k85001182828.htn_d" serviceName="URI-DODS"
> >               urlPath="htn_sst_decloud/1985/k85002181750.htn_d.Z"/>
> >   </collection>
> > </catalog>

-- 
Ethan R. Davis                       Telephone: (303) 497-8155
Software Engineer                    Fax:       (303) 497-8690
UCAR Unidata Program Center          E-mail:    address@hidden
P.O. Box 3000
Boulder, CO  80307-3000              http://www.unidata.ucar.edu/