[THREDDS #JRP-282661]: Fwd: WMS in THREDDS 5

Oops...here is the attachment.

Sean

> Greetings Alan,
> 
> The reason I had you try with the vanilla TDS war and and simple example
> was to see if we could even access WMS on your server, or if there was
> a potential issue outside of the TDS (maybe Tomcat, Apache, etc). Given
> that the simple test works, it looks like the issue is related to a difference
> in the esgf bundled TDS.
> 
> There are two things I'd like to try now.
> 
> 1) Try the aggregation with the vanilla TDS by adding the following
> to the main catalog.xml (the one with the simple datasetScan that works):
> 
> <dataset ID="Aggtest"
> name="AggTest"
> urlPath="AggTest">
> 
> <metadata>
> <dataType>GRID</dataType>
> </metadata>
> 
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> <aggregation dimName="time" type="joinExisting">
> <variableAgg name="cee"/>
> <variableAgg name="cee_corr_unc"/>
> <scan location="/neodc/esacci/cloud/data/phase-2/L3C/AVHRR-PM/v2.0" 
> suffix=".nc"/>
> </aggregation>
> </netcdf>
> </dataset>
> 
> between the </datasetScan> and </catalog> entries at the end of the 
> catalog.xml file.
> Note I've attached an updated copy of catalog.xml with the changes in case 
> the xml
> snippets above do not show up in this message. Check to see if things still 
> work in the first
> dataset, and then see if they work with your aggregation. Try hitting the 
> opendap link
> of your aggregation dataset first to make sure it does not return a 404 or 
> 500.
> 
> 2) Can you send me the thredds/WEB-INF/web.xml file from the webapps directory
> included with the ESGF distribution?
> 
> Thanks!
> 
> Sean
> 
> > Changing the address to the support address as Sean requested.]
> >
> > I have tried with the a top-level catalogue containing both the "test
> > all files in a directory" <datasetScan> element and also the "Earth
> > System Root" <catalogRef> element containing files published by the
> > ESGF publisher.  I have tried this with both the vanilla thredds.war
> > file you supplied and also with the thredds directory from ESGF.
> >
> > The situation now seems to be:
> >
> > 1) if I try with the vanilla thredds.war file you supplied, then the
> > "test all files in a directory" link in your test top-level catalogue
> > seem to work, including the WMS links, but I have two problems:
> >
> > (a) it is a very different example from the one which I was trying to
> > get working, and I do not know how to adapt it -- namely, it has a
> > <datasetScan> which finds a bunch of datasets consisting of one file
> > each, whereas I want to serve a single dataset but that dataset is to
> > consist of an aggregation of netCDF files
> >
> > (b) with the vanilla thredds.war file, I do not have the changes made
> > by ESGF, and if I try to use a catalogue produced by the ESGF
> > publisher then it does not work, for example with the HTTPServer links
> > it redirects to /thredds/restrictedAccess/esg-user and then Chrome
> > reports "ERR_INVALID_HTTP_RESPONSE", and if I insert WMS links then I
> > get a NullPointerException.
> >
> > 2) If I reinstate the thredds/ directory supplied by ESGF (removing
> > the vanilla thredds directory and thredds.war), then we are back where
> > we were before, namely that the datasets published by the ESGF
> > publisher work as regards access methods other than wms (e.g.
> > HTTPServer), but /thredds/wms and any URLs below it get a 404 error.
> >
> > 3) I also tried to merge the two (use the later versions of the .jar
> > files but import the ESGF xml changes), but this did not work (I think
> > 500 errors), and to be honest, I don't really know which differences
> > to import.
> >
> > I forgot to attach to my last email the diffs between the two
> > directories.  They are now reattached.
> >
> > Any help appreciated.
> >
> > Thanks,
> > Alan
> >
> > On 12 October 2017 at 16:35, Sean Arms <address@hidden> wrote:
> > > Greetings Alan,
> > >
> > > The ESGF group makes some changes to WEB-INF/web.xml for security related
> > > things (authentication / authorization, I believe), and so they ship the
> > > "exploded war file", which is just the directory. If Tomcat determines the
> > > actual war file in the webapps directory is newer than what is in the
> > > exploded directory, it will overwrite it and all the custom security stuff
> > > will be removed. So, the ESGF group simply ships the directory rather than
> > > the war so that they can assure the changes they make will stay in place
> > > when tomcat is restarted.
> > >
> > > The files under altContent are the default TDS config files. If the
> > > directory that is specified using the tds.content.root.path variable in
> > > setenv.sh is empty or does not exist, the TDS will create it and populate 
> > > it
> > > with these files. As you have found, they are never actually read 
> > > directly.
> > > Everything you need to edit is going to be in the tds.content.root.path
> > > directory.
> > >
> > > The diff of the two thredds webapps directory didn't make it though - 
> > > would
> > > you mind sending it again? Given that you were able to get wms to respond
> > > with the vanilla TDS, I'm thinking ESGF has tweaked something, probably in
> > > WEB-INF/, that is preventing the ESGF distribution from being able to 
> > > "see"
> > > the WMS server properly.
> > >
> > > We'll get this figured out - it just might take a little more time :-)
> > >
> > > Cheers,
> > >
> > > Sean
> > >
> > >
> > >
> > > On Wed, Oct 11, 2017 at 7:54 PM, Alan Iwi (STFC) <address@hidden>
> > > wrote:
> > >>
> > >> Sean,
> > >>
> > >> I also attach a recursive diff between the original thredds webapp
> > >> directory (renamed webapps.tmp/thredds) and the new one as
> > >> automatically extracted by tomcat from your new thredds.war file.
> > >>
> > >> By the way, I think the lines I put into
> > >> webapps/thredds/WEB-INF/altContent/startup/threddsConfig.xml are not
> > >> significant.  I subsequently discovered (with strace) that it is not
> > >> reading that file, and that the active threddsConfig.xml lives in the
> > >> directory with the catalog.xml (outside the webapps directory).  The
> > >> active one is the threddsConfig.xml that I already attached to an
> > >> earlier email, and I never bothered to edit back the one under
> > >> altContent.
> > >>
> > >> Thanks,
> > >> Alan
> > >>
> > >> On 12 October 2017 at 02:46, Alan Iwi (STFC) <address@hidden> wrote:
> > >> > Sean,
> > >> >
> > >> > Okay well, it's the middle of the night for me and maybe you've
> > >> > finished now anyway, but I thought I'd try to turn this around quickly
> > >> > just in case...
> > >> >
> > >> > I tried with your vanilla .war file and it seemed to work.
> > >> >
> > >> > Specifically, the webapps directory previously contained an extracted
> > >> > thredds directory (and no .war file).  I moved this subdirectory out
> > >> > of the way (out of the webapps directory), and in its place I put the
> > >> > new thredds.war without extracting it.  I then started the server
> > >> > using the usual top-level script provided by ESGF, and did not modify
> > >> > the JAVA_OPTS to point to the config dir because it's already in
> > >> > principle set up.  For now, I am still using your root catalogue.
> > >> >
> > >> > Now when I click on the WMS link, I get the XML file which I have
> > >> > attached (the attachment filename was my choice rather than anything
> > >> > provided by the server).  It seems that tomcat has extracted the .war
> > >> > file automatically so there is now both thredds/ and thredds.war.
> > >> >
> > >> > Tomorrow I can try investigating further -- in particular, trying it
> > >> > with the catalogues that I was trying to use originally.
> > >> >
> > >> > Regards,
> > >> > Alan
> > >> >
> > >> >
> > >> > On 12 October 2017 at 02:22, Alan Iwi (STFC) <address@hidden>
> > >> > wrote:
> > >> >> Sean,
> > >> >>
> > >> >> Thanks very much.  I'll have a look tomorrow.
> > >> >>
> > >> >> In case I get stuck further, what time tomorrow will you have time to
> > >> >> look at it?  I'm just wondering whether it is worth me planning to try
> > >> >> to overlap with you again if it's not too late.
> > >> >>
> > >> >> Many thanks,
> > >> >> Alan
> > >> >>
> > >> >>
> > >> >> On 11 October 2017 at 23:26, Sean Arms <address@hidden> wrote:
> > >> >>> Oops, you also likely need to add the following after the JAVA_OPTS
> > >> >>> line in
> > >> >>> setenv.sh:
> > >> >>>
> > >> >>> export JAVA_OPTS
> > >> >>>
> > >> >>> Also, if your tomcat users shell isn't bash, you'll need to add the
> > >> >>> following to the top of setenv.sh:
> > >> >>>
> > >> >>> #!/bin/sh
> > >> >>>
> > >> >>> Sorry if you already know these things - just covering the bases :-)
> > >> >>>
> > >> >>> Cheers,
> > >> >>>
> > >> >>> Sean
> > >> >>>
> > >> >>>
> > >> >>> On Wed, Oct 11, 2017 at 4:23 PM, Sean Arms <address@hidden> wrote:
> > >> >>>>
> > >> >>>> This is very interesting. DAP4 isn't working yet, so that's 
> > >> >>>> expected,
> > >> >>>> but
> > >> >>>> it is not expected for WMS. I'm wondering if something gets changed
> > >> >>>> during
> > >> >>>> the process in which the ESGF group bundles in the TDS into their
> > >> >>>> installation process. Would it be possible for you to test with a
> > >> >>>> vanilla
> > >> >>>> TDS install? You can grab the war file here:
> > >> >>>>
> > >> >>>>
> > >> >>>>
> > >> >>>> https://artifacts.unidata.ucar.edu/content/repositories/unidata-releases/edu/ucar/tds/ESGF-5.0.1/tds-ESGF-5.0.1.war
> > >> >>>>
> > >> >>>> You'll need to rename the war file to thredds.war, or
> > >> >>>> thredds##esgf-5.0.1.war before putting it in the tomcat webapps
> > >> >>>> directory.
> > >> >>>> You'll also need to edit the tomcat bin/setenv.sh to add the
> > >> >>>> following:
> > >> >>>>
> > >> >>>> JAVA_OPTS=-Dtds.content.root.path=</some/path/for/tds/configs>
> > >> >>>>
> > >> >>>> Sorry for the craziness - I don't see anything funky in the config
> > >> >>>> files
> > >> >>>> you sent, so I am wondering if we can even get a plain TDS install
> > >> >>>> working.
> > >> >>>>
> > >> >>>> Cheers,
> > >> >>>>
> > >> >>>> Sean
> > >> >>>>
> > >> >>>>
> > >> >>>> On Wed, Oct 11, 2017 at 1:35 PM, Alan Iwi (STFC)
> > >> >>>> <address@hidden>
> > >> >>>> wrote:
> > >> >>>>>
> > >> >>>>> Sean,
> > >> >>>>>
> > >> >>>>> I also attach here the threddsConfig.xml file in case it helps.
> > >> >>>>>
> > >> >>>>> Thanks,
> > >> >>>>> Alan
> > >> >>>>>
> > >> >>>>> On 11 October 2017 at 20:31, Alan Iwi (STFC) <address@hidden>
> > >> >>>>> wrote:
> > >> >>>>> > Sean,
> > >> >>>>> >
> > >> >>>>> > Thanks for your help.  I installed your top-level catalogue, and
> > >> >>>>> > if I
> > >> >>>>> > click on one of the test files I get a bunch of links.
> > >> >>>>> >
> > >> >>>>> > OPENDAP: /thredds/dodsC/testAll/2004050400_eta_211.nc.html
> > >> >>>>> > CdmRemote: /thredds/cdmremote/testAll/2004050400_eta_211.nc
> > >> >>>>> > CdmrFeature:
> > >> >>>>> > /thredds/cdmrfeature/grid/testAll/2004050400_eta_211.nc
> > >> >>>>> > DAP4: /thredds/dap4/testAll/2004050400_eta_211.nc.dmr.xml
> > >> >>>>> > HTTPServer: /thredds/fileServer/testAll/2004050400_eta_211.nc
> > >> >>>>> > NetcdfSubset:
> > >> >>>>> > /thredds/ncss/grid/testAll/2004050400_eta_211.nc/dataset.html
> > >> >>>>> > WMS: /thredds/wms/testAll/2004050400_eta_211.nc
> > >> >>>>> > WCS: /thredds/wcs/testAll/2004050400_eta_211.nc
> > >> >>>>> > ISO: /thredds/iso/testAll/2004050400_eta_211.nc
> > >> >>>>> > NCML: /thredds/ncml/testAll/2004050400_eta_211.nc
> > >> >>>>> > UDDC: /thredds/uddc/testAll/2004050400_eta_211.nc
> > >> >>>>> >
> > >> >>>>> > Of these, all the following links all do something:
> > >> >>>>> >
> > >> >>>>> > OPENDAP
> > >> >>>>> > CdmRemote
> > >> >>>>> > CdmrFeature
> > >> >>>>> > HTTPServer
> > >> >>>>> > NetcdfSubset
> > >> >>>>> > WCS
> > >> >>>>> > ISO
> > >> >>>>> > NCML
> > >> >>>>> > UDDC
> > >> >>>>> >
> > >> >>>>> > and just the following two links give 404s:
> > >> >>>>> >
> > >> >>>>> > DAP4
> > >> >>>>> > WMS
> > >> >>>>> >
> > >> >>>>> > I attach the catalogInit.log obtained after installing your
> > >> >>>>> > catalog.xml.
> > >> >>>>> >
> > >> >>>>> > Thanks,
> > >> >>>>> > Alan
> > >> >>>>> >
> > >> >>>>> >
> > >> >>>>> >
> > >> >>>>> > On 11 October 2017 at 20:19, Sean Arms <address@hidden> wrote:
> > >> >>>>> >> Greetings Alan,
> > >> >>>>> >>
> > >> >>>>> >> Can you give this root catalog a try? Make sure to keep it's 
> > >> >>>>> >> name
> > >> >>>>> >> as
> > >> >>>>> >> catalog.xml and put it in the top level content directory of the
> > >> >>>>> >> TDS
> > >> >>>>> >> (<tds.content.root.path>/thredds/). Also, can you send me the 
> > >> >>>>> >> TDS
> > >> >>>>> >> catalogInit.log log file?
> > >> >>>>> >> (<tds.content.root.path>/thredds/logs/catalogInit.log?)
> > >> >>>>> >>
> > >> >>>>> >> Cheers,
> > >> >>>>> >>
> > >> >>>>> >> Sean
> > >> >>>>> >>
> > >> >>>>> >>
> > >> >>>>> >>
> > >> >>>>> >> On Wed, Oct 11, 2017 at 1:05 PM, Alan Iwi (STFC)
> > >> >>>>> >> <address@hidden>
> > >> >>>>> >> wrote:
> > >> >>>>> >>>
> > >> >>>>> >>> Sean,
> > >> >>>>> >>>
> > >> >>>>> >>> I tried your revised client catalogue, and I am still getting a
> > >> >>>>> >>> 404.
> > >> >>>>> >>> And my test with the plain /thredds/wms/ URL (where I get a 404
> > >> >>>>> >>> instead of the ServiceExceptionReport document) rather suggests
> > >> >>>>> >>> to me
> > >> >>>>> >>> that I have a configuration problem which is going to mean that
> > >> >>>>> >>> I
> > >> >>>>> >>> continue to get these 404 errors *regardless* of the contents 
> > >> >>>>> >>> of
> > >> >>>>> >>> the
> > >> >>>>> >>> client catalogue.  It seems to me that - at a high level
> > >> >>>>> >>> independent
> > >> >>>>> >>> of any particular dataset - THREDDS does not know what to do
> > >> >>>>> >>> with
> > >> >>>>> >>> these wms URLs.  How can I debug this please?
> > >> >>>>> >>>
> > >> >>>>> >>> (I have also looked at your configuration catalogue, but I am
> > >> >>>>> >>> struggling to see what it corresponds to on my system.)
> > >> >>>>> >>>
> > >> >>>>> >>> Thanks,
> > >> >>>>> >>> Alan
> > >> >>>>> >>>
> > >> >>>>> >>>
> > >> >>>>> >>> On 11 October 2017 at 19:54, Alan Iwi (STFC)
> > >> >>>>> >>> <address@hidden>
> > >> >>>>> >>> wrote:
> > >> >>>>> >>> > Sean,
> > >> >>>>> >>> >
> > >> >>>>> >>> > Thanks.  I'll give it a go...
> > >> >>>>> >>> >
> > >> >>>>> >>> > Regards,
> > >> >>>>> >>> > Alan
> > >> >>>>> >>> >
> > >> >>>>> >>> > On 11 October 2017 at 19:32, Sean Arms <address@hidden> 
> > >> >>>>> >>> > wrote:
> > >> >>>>> >>> >> Greetings Alan,
> > >> >>>>> >>> >>
> > >> >>>>> >>> >> You won't be able to use the xml from our server directly, 
> > >> >>>>> >>> >> as
> > >> >>>>> >>> >> that
> > >> >>>>> >>> >> is a
> > >> >>>>> >>> >> client catalog and not a configuration catalog. The
> > >> >>>>> >>> >> configuration
> > >> >>>>> >>> >> catalog
> > >> >>>>> >>> >> looks like this:
> > >> >>>>> >>> >>
> > >> >>>>> >>> >>
> > >> >>>>> >>> >>
> > >> >>>>> >>> >>
> > >> >>>>> >>> >> https://github.com/Unidata/TdsConfig/blob/master/threddsDev/idd/forecastModels.xml
> > >> >>>>> >>> >>
> > >> >>>>> >>> >> Now, of course, this catalog has many GRIB featureCollection
> > >> >>>>> >>> >> elements,
> > >> >>>>> >>> >> which
> > >> >>>>> >>> >> are not helpful in your case. But, notice how there isn't 
> > >> >>>>> >>> >> any
> > >> >>>>> >>> >> <service>
> > >> >>>>> >>> >> or
> > >> >>>>> >>> >> <serviceName> elements anywhere in the catalog.
> > >> >>>>> >>> >>
> > >> >>>>> >>> >> For your example catalog, mytest.xml, I've updated a few
> > >> >>>>> >>> >> things
> > >> >>>>> >>> >> (see
> > >> >>>>> >>> >> attached). Give this catalog a try and see how it does.
> > >> >>>>> >>> >>
> > >> >>>>> >>> >> Thanks!
> > >> >>>>> >>> >>
> > >> >>>>> >>> >> Sean
> > >> >>>>> >>> >>
> > >> >>>>> >>> >>
> > >> >>>>> >>> >>
> > >> >>>>> >>> >>
> > >> >>>>> >>> >> On Wed, Oct 11, 2017 at 11:50 AM, Alan Iwi (STFC)
> > >> >>>>> >>> >> <address@hidden>
> > >> >>>>> >>> >> wrote:
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> Sean,
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> I've now done the following tests, which convince me that
> > >> >>>>> >>> >>> the
> > >> >>>>> >>> >>> problem
> > >> >>>>> >>> >>> is somehow with our server setup rather than with the
> > >> >>>>> >>> >>> dataset
> > >> >>>>> >>> >>> catalogue.
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> One is a very simple one, that if I just go to:
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> https://esgf-test-install.ceda.ac.uk/thredds/wms/
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> then I get a 404, whereas if I go to:
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> http://thredds-dev.unidata.ucar.edu/thredds/wms/
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> then I get an XML file with a ServiceExceptionReport.
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> (NB, you won't see our test server as it is firewalled 
> > >> >>>>> >>> >>> off.)
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> The other test involves fetching one of your catalogues and
> > >> >>>>> >>> >>> trying to
> > >> >>>>> >>> >>> serve it from our server -- described in more detail below,
> > >> >>>>> >>> >>> but
> > >> >>>>> >>> >>> the
> > >> >>>>> >>> >>> summary is that the HTTPServer works and the WMS doesn't.
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> Where would I go from here to work out why the server does
> > >> >>>>> >>> >>> not
> > >> >>>>> >>> >>> know
> > >> >>>>> >>> >>> what to do with /thredds/wms URLs?
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> Thanks,
> > >> >>>>> >>> >>> Alan
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> [details of second test follow]
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> ==================================================================
> > >> >>>>> >>> >>> * wget the following XML from your server to mine, saving 
> > >> >>>>> >>> >>> as
> > >> >>>>> >>> >>> a
> > >> >>>>> >>> >>> local
> > >> >>>>> >>> >>> file in the thredds directory
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> http://thredds-dev.unidata.ucar.edu/thredds/catalog/grib/NCEP/NBM/Alaska/National_Blend_Alaska_20170911_0000.grib2/catalog.xml
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> * edit it, making the following changes (only):
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>  - in VirtualServices, comment out all except wms
> > >> >>>>> >>> >>>  - in fullServices, comment out all except HTTPServer and
> > >> >>>>> >>> >>> wms
> > >> >>>>> >>> >>>  - search and replace the relative path of the data file to
> > >> >>>>> >>> >>> point
> > >> >>>>> >>> >>> to
> > >> >>>>> >>> >>> esg_dataroot/National_Blend_Alaska_20170911_0000.grib2
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>    where esg_dataroot is one of the data roots set up, and 
> > >> >>>>> >>> >>> I
> > >> >>>>> >>> >>> drop
> > >> >>>>> >>> >>> into
> > >> >>>>> >>> >>> there a world-readable copy of the data file which I
> > >> >>>>> >>> >>> downloaded
> > >> >>>>> >>> >>> from
> > >> >>>>> >>> >>> your server using the HTTPServer URL
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> * set a global allow policy in the esg-orp security layer
> > >> >>>>> >>> >>> for any
> > >> >>>>> >>> >>> URL
> > >> >>>>> >>> >>> matching .*Alaska.*
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> * link this XML file from top level catalog and do a reinit
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> * try accessing over http
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> The result is a catalogue that has a working HTTPServer 
> > >> >>>>> >>> >>> link
> > >> >>>>> >>> >>> and
> > >> >>>>> >>> >>> a
> > >> >>>>> >>> >>> broken WMS link.  See attached:
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> 1) the XML file from the filesystem  (the XML as served by
> > >> >>>>> >>> >>> THREDDS is
> > >> >>>>> >>> >>> the same as this except for the omission of the comments 
> > >> >>>>> >>> >>> and
> > >> >>>>> >>> >>> adding
> > >> >>>>> >>> >>> 'zpositive="up"' in the geospatialCoverage element).
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> 2) HTML as served by THREDDS
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> 3) for convenience, screenshot of the HTML in my browser
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> The HTTPServer link works (file is downloaded), and the WMS
> > >> >>>>> >>> >>> link
> > >> >>>>> >>> >>> gives
> > >> >>>>> >>> >>> a
> > >> >>>>> >>> >>> 404.
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> ==================================================================
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>>
> > >> >>>>> >>> >>> On 11 October 2017 at 17:11, Alan Iwi (STFC)
> > >> >>>>> >>> >>> <address@hidden>
> > >> >>>>> >>> >>> wrote:
> > >> >>>>> >>> >>> > Sean,
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > In case any use, I am sending here what I am getting in a
> > >> >>>>> >>> >>> > simplified
> > >> >>>>> >>> >>> > example.
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > Attached is the dataset catalogue, and also screenshots 
> > >> >>>>> >>> >>> > as
> > >> >>>>> >>> >>> > follows
> > >> >>>>> >>> >>> > (clicking through from the top-level catalogue):
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > 1.png
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > https://esgf-test-install.ceda.ac.uk/thredds/catalog/catalog.html
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > 2.png
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > https://esgf-test-install.ceda.ac.uk/thredds/catalog/mytest.html
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > 3.png
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > https://esgf-test-install.ceda.ac.uk/thredds/catalog/mytest.html?dataset=mytest
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > 4.png
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > https://esgf-test-install.ceda.ac.uk/thredds/wms/mytest?service=WMS&version=1.3.0&request=GetCapabilities
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > The top-level XML file has:
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> >   <WMS>
> > >> >>>>> >>> >>> >     <allow>true</allow>
> > >> >>>>> >>> >>> >     <allowRemote>false</allowRemote>
> > >> >>>>> >>> >>> >     <maxImageWidth>2048</maxImageWidth>
> > >> >>>>> >>> >>> >     <maxImageHeight>2048</maxImageHeight>
> > >> >>>>> >>> >>> >   </WMS>
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > directly inside the <threddsConfig> element.
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > Please say if I should send anything from the log file to
> > >> >>>>> >>> >>> > accompany
> > >> >>>>> >>> >>> > it.
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > But also, maybe if I see the XML code and any config file
> > >> >>>>> >>> >>> > stuff
> > >> >>>>> >>> >>> > behind
> > >> >>>>> >>> >>> > your working example, I could try to work forward from
> > >> >>>>> >>> >>> > there?
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > Thanks again,
> > >> >>>>> >>> >>> > Alan
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> >
> > >> >>>>> >>> >>> > On 11 October 2017 at 14:55, Alan Iwi (STFC)
> > >> >>>>> >>> >>> > <address@hidden>
> > >> >>>>> >>> >>> > wrote:
> > >> >>>>> >>> >>> >> Sean,
> > >> >>>>> >>> >>> >>
> > >> >>>>> >>> >>> >> Thanks very much.  I think there is something I'm not
> > >> >>>>> >>> >>> >> understanding,
> > >> >>>>> >>> >>> >> though.  If I have the <service name=...> and
> > >> >>>>> >>> >>> >> <serviceName>
> > >> >>>>> >>> >>> >> elements
> > >> >>>>> >>> >>> >> both included, then I get a WMS link but it gives a 404
> > >> >>>>> >>> >>> >> if I
> > >> >>>>> >>> >>> >> click
> > >> >>>>> >>> >>> >> on
> > >> >>>>> >>> >>> >> it.  If I comment either of them out, then there is no
> > >> >>>>> >>> >>> >> link
> > >> >>>>> >>> >>> >> visible
> > >> >>>>> >>> >>> >> at
> > >> >>>>> >>> >>> >> all.
> > >> >>>>> >>> >>> >>
> > >> >>>>> >>> >>> >> Please could I have a look at what the relevant catalog
> > >> >>>>> >>> >>> >> XML
> > >> >>>>> >>> >>> >> file
> > >> >>>>> >>> >>> >> looks
> > >> >>>>> >>> >>> >> like in your working example?  (Sorry if this is
> > >> >>>>> >>> >>> >> something I
> > >> >>>>> >>> >>> >> can
> > >> >>>>> >>> >>> >> see
> > >> >>>>> >>> >>> >> for myself over http, but I'm not sure what the URL 
> > >> >>>>> >>> >>> >> would
> > >> >>>>> >>> >>> >> be.)
> > >> >>>>> >>> >>> >>
> > >> >>>>> >>> >>> >> Thanks,
> > >> >>>>> >>> >>> >> Alan
> > >> >>>>> >>> >>> >>
> > >> >>>>> >>> >>> >> On 10 October 2017 at 16:02, Sean Arms <address@hidden>
> > >> >>>>> >>> >>> >> wrote:
> > >> >>>>> >>> >>> >>> Greetings Ag, Alan,
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>> Sorry for the delay - I've been out and am playing a 
> > >> >>>>> >>> >>> >>> bit
> > >> >>>>> >>> >>> >>> of
> > >> >>>>> >>> >>> >>> catch
> > >> >>>>> >>> >>> >>> up
> > >> >>>>> >>> >>> >>> still.
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>> The WMS service should be working in TDS 5 - here is an
> > >> >>>>> >>> >>> >>> example on
> > >> >>>>> >>> >>> >>> our
> > >> >>>>> >>> >>> >>> TDS
> > >> >>>>> >>> >>> >>> dev server:
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>> http://thredds-dev.unidata.ucar.edu/thredds/wms/grib/NCEP/GFS/Global_0p25deg/GFS_Global_0p25deg_20171010_0600.grib2?service=WMS&version=1.3.0&request=GetCapabilities
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>> This is running a 5.0 snapshot version from this past
> > >> >>>>> >>> >>> >>> March;
> > >> >>>>> >>> >>> >>> I'm
> > >> >>>>> >>> >>> >>> pretty sure
> > >> >>>>> >>> >>> >>> the latest ESGF release is using a more up-to-date
> > >> >>>>> >>> >>> >>> version.
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>> One of the changes made to the TDS in version 5.0 is 
> > >> >>>>> >>> >>> >>> the
> > >> >>>>> >>> >>> >>> concept
> > >> >>>>> >>> >>> >>> of
> > >> >>>>> >>> >>> >>> default
> > >> >>>>> >>> >>> >>> services. I've encountered a few places where 
> > >> >>>>> >>> >>> >>> explicitly
> > >> >>>>> >>> >>> >>> defining
> > >> >>>>> >>> >>> >>> a
> > >> >>>>> >>> >>> >>> service
> > >> >>>>> >>> >>> >>> in a catalog causes errors, while not defining any
> > >> >>>>> >>> >>> >>> services
> > >> >>>>> >>> >>> >>> for a
> > >> >>>>> >>> >>> >>> dataset
> > >> >>>>> >>> >>> >>> works (as long as the dataset you are serving includes
> > >> >>>>> >>> >>> >>> "<dataType>GRID</dataType>" in the datasets metadata. I
> > >> >>>>> >>> >>> >>> am
> > >> >>>>> >>> >>> >>> wondering
> > >> >>>>> >>> >>> >>> if WMS
> > >> >>>>> >>> >>> >>> is one of those services. Could you try removing all
> > >> >>>>> >>> >>> >>> "<serviceName>"
> > >> >>>>> >>> >>> >>> and
> > >> >>>>> >>> >>> >>> <service name=...>" elements from the catalog and see 
> > >> >>>>> >>> >>> >>> if
> > >> >>>>> >>> >>> >>> that
> > >> >>>>> >>> >>> >>> works?
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>> If that does not work, would it be possible for me to
> > >> >>>>> >>> >>> >>> get a
> > >> >>>>> >>> >>> >>> sample
> > >> >>>>> >>> >>> >>> of
> > >> >>>>> >>> >>> >>> the
> > >> >>>>> >>> >>> >>> dataset you are working with?
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>> Cheers,
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>> Sean
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>> On Tue, Oct 10, 2017 at 6:11 AM,
> > >> >>>>> >>> >>> >>> <address@hidden>
> > >> >>>>> >>> >>> >>> wrote:
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> Hi Sean,
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> We are trying to look into this issue this week. It
> > >> >>>>> >>> >>> >>>> would be
> > >> >>>>> >>> >>> >>>> great if
> > >> >>>>> >>> >>> >>>> you
> > >> >>>>> >>> >>> >>>> could get back to Alan regarding the WMS on TDS5.0
> > >> >>>>> >>> >>> >>>> question.
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> It would save us a heap of time if it is now working 
> > >> >>>>> >>> >>> >>>> in
> > >> >>>>> >>> >>> >>>> TDS5.0.
> > >> >>>>> >>> >>> >>>> :-)
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> Many thanks,
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> Ag
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> -----Original Message-----
> > >> >>>>> >>> >>> >>>> From: address@hidden
> > >> >>>>> >>> >>> >>>> [mailto:address@hidden] On
> > >> >>>>> >>> >>> >>>> Behalf
> > >> >>>>> >>> >>> >>>> Of Alan Iwi (STFC)
> > >> >>>>> >>> >>> >>>> Sent: 04 October 2017 10:12
> > >> >>>>> >>> >>> >>>> To: address@hidden
> > >> >>>>> >>> >>> >>>> Cc: Stephens, Ag (STFC,RAL,RALSP); Kershaw, Philip
> > >> >>>>> >>> >>> >>>> (STFC,RAL,RALSP)
> > >> >>>>> >>> >>> >>>> Subject: WMS in THREDDS 5
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> Dear Sean,
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> Ag Stephens gave me your name as the current contact
> > >> >>>>> >>> >>> >>>> for TDS
> > >> >>>>> >>> >>> >>>> development.
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> Please can you tell me whether WMS is believed to be
> > >> >>>>> >>> >>> >>>> working
> > >> >>>>> >>> >>> >>>> in
> > >> >>>>> >>> >>> >>>> TDS
> > >> >>>>> >>> >>> >>>> 5.
> > >> >>>>> >>> >>> >>>> We are already using WMS in TDS 4, but it would be
> > >> >>>>> >>> >>> >>>> convenient if
> > >> >>>>> >>> >>> >>>> it
> > >> >>>>> >>> >>> >>>> is
> > >> >>>>> >>> >>> >>>> also working in 5, because then we could serve it from
> > >> >>>>> >>> >>> >>>> the
> > >> >>>>> >>> >>> >>>> same
> > >> >>>>> >>> >>> >>>> THREDDS
> > >> >>>>> >>> >>> >>>> deployment as we are using for the file download and
> > >> >>>>> >>> >>> >>>> the
> > >> >>>>> >>> >>> >>>> opendap
> > >> >>>>> >>> >>> >>>> service --
> > >> >>>>> >>> >>> >>>> currently we have to manage content on two different
> > >> >>>>> >>> >>> >>>> servers
> > >> >>>>> >>> >>> >>>> in
> > >> >>>>> >>> >>> >>>> order
> > >> >>>>> >>> >>> >>>> to
> > >> >>>>> >>> >>> >>>> serve different access methods for the same datasets.
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> Our TDS 4 deployment is on a standalone Tomcat
> > >> >>>>> >>> >>> >>>> installation
> > >> >>>>> >>> >>> >>>> (installed
> > >> >>>>> >>> >>> >>>> from RPM), but I am now trying to enable WMS in TDS
> > >> >>>>> >>> >>> >>>> 5.0.0
> > >> >>>>> >>> >>> >>>> which
> > >> >>>>> >>> >>> >>>> is
> > >> >>>>> >>> >>> >>>> part of a
> > >> >>>>> >>> >>> >>>> deployment of the latest ESGF software stack on a test
> > >> >>>>> >>> >>> >>>> node.
> > >> >>>>> >>> >>> >>>> When I try it, I get a 404 response for all URLs under
> > >> >>>>> >>> >>> >>>> /thredds/wms/,
> > >> >>>>> >>> >>> >>>> and
> > >> >>>>> >>> >>> >>>> nothing in the catalina.{out,err} files or THREDDS log
> > >> >>>>> >>> >>> >>>> files.
> > >> >>>>> >>> >>> >>>> There is also no mention of wms (or WMS) in the
> > >> >>>>> >>> >>> >>>> serverStartup.log:
> > >> >>>>> >>> >>> >>>> the
> > >> >>>>> >>> >>> >>>> serverStartup log for TDS 4 has messages about
> > >> >>>>> >>> >>> >>>> FrameworkServlet
> > >> >>>>> >>> >>> >>>> initialization for both 'root' and 'wms', whereas the
> > >> >>>>> >>> >>> >>>> serverStartup
> > >> >>>>> >>> >>> >>>> log for
> > >> >>>>> >>> >>> >>>> TDS 5 only has one for 'spring', so it is handling
> > >> >>>>> >>> >>> >>>> things
> > >> >>>>> >>> >>> >>>> differently
> > >> >>>>> >>> >>> >>>> in a
> > >> >>>>> >>> >>> >>>> way I don't know the details of.
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> I am enabling the service everywhere that I am aware
> > >> >>>>> >>> >>> >>>> of.  In
> > >> >>>>> >>> >>> >>>> particular, I
> > >> >>>>> >>> >>> >>>> am putting the relevant <service> elements and
> > >> >>>>> >>> >>> >>>> corresponding
> > >> >>>>> >>> >>> >>>> <access
> > >> >>>>> >>> >>> >>>> serviceName="wms"...> in the dataset catalog.
> > >> >>>>> >>> >>> >>>> Also the threddsConfig.xml for TDS 5 already has WMS
> > >> >>>>> >>> >>> >>>> enabled
> > >> >>>>> >>> >>> >>>> (with an
> > >> >>>>> >>> >>> >>>> "allow" element containing text "true").  I know that
> > >> >>>>> >>> >>> >>>> the
> > >> >>>>> >>> >>> >>>> request
> > >> >>>>> >>> >>> >>>> is
> > >> >>>>> >>> >>> >>>> being
> > >> >>>>> >>> >>> >>>> forwarded through to tomcat, as it appears in the
> > >> >>>>> >>> >>> >>>> Tomcat
> > >> >>>>> >>> >>> >>>> access
> > >> >>>>> >>> >>> >>>> log.
> > >> >>>>> >>> >>> >>>> It
> > >> >>>>> >>> >>> >>>> also seems that the esg-orp is filtering the request
> > >> >>>>> >>> >>> >>>> per
> > >> >>>>> >>> >>> >>>> normal,
> > >> >>>>> >>> >>> >>>> because if
> > >> >>>>> >>> >>> >>>> I change the part of the URL after /thredds/wms so 
> > >> >>>>> >>> >>> >>>> that
> > >> >>>>> >>> >>> >>>> the
> > >> >>>>> >>> >>> >>>> full
> > >> >>>>> >>> >>> >>>> URL
> > >> >>>>> >>> >>> >>>> no
> > >> >>>>> >>> >>> >>>> longer matches the pattern for open access, then this
> > >> >>>>> >>> >>> >>>> changes the
> > >> >>>>> >>> >>> >>>> response.
> > >> >>>>> >>> >>> >>>> (In fact this generates an error, because our current
> > >> >>>>> >>> >>> >>>> esg-orp
> > >> >>>>> >>> >>> >>>> deployment
> > >> >>>>> >>> >>> >>>> seems to have some bug relating to protected 
> > >> >>>>> >>> >>> >>>> resources,
> > >> >>>>> >>> >>> >>>> but
> > >> >>>>> >>> >>> >>>> the
> > >> >>>>> >>> >>> >>>> details are
> > >> >>>>> >>> >>> >>>> probably not important here because is not affecting
> > >> >>>>> >>> >>> >>>> openly
> > >> >>>>> >>> >>> >>>> accessible
> > >> >>>>> >>> >>> >>>> resources.)
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> Is there anything else I need to do to make it work, 
> > >> >>>>> >>> >>> >>>> or
> > >> >>>>> >>> >>> >>>> do I
> > >> >>>>> >>> >>> >>>> just
> > >> >>>>> >>> >>> >>>> have to
> > >> >>>>> >>> >>> >>>> continue to use the THREDDS 4 deployment instead?
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> Any advice appreciated.
> > >> >>>>> >>> >>> >>>>
> > >> >>>>> >>> >>> >>>> Many thanks,
> > >> >>>>> >>> >>> >>>> Alan
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>> >>>
> > >> >>>>> >>> >>
> > >> >>>>> >>> >>
> > >> >>>>> >>
> > >> >>>>> >>
> > >> >>>>
> > >> >>>>
> > >> >>>
> > >
> > >
> >
> >


Ticket Details
===================
Ticket ID: JRP-282661
Department: Support THREDDS
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.

<?xml version="1.0" encoding="UTF-8"?>
<catalog name="THREDDS Server Default Catalog : You must change this to fit 
your server!"
         xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
         xmlns:xlink="http://www.w3.org/1999/xlink";
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
         
xsi:schemaLocation="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0
           http://www.unidata.ucar.edu/schemas/thredds/InvCatalog.1.0.6.xsd";>

  <datasetRoot path="test" location="content/testdata/" />

  <datasetScan name="Test all files in a directory" ID="testDatasetScan"
               path="testAll" location="content/testdata">
    <metadata inherited="true">
      <dataType>Grid</dataType>
    </metadata>

    <filter>
      <include wildcard="*eta_211.nc"/>
      <include wildcard="testgrid*.nc"/>
    </filter>
  </datasetScan>

  <dataset ID="AggTest"
           name="AggTest"
           urlPath="AggTest">

    <metadata>
      <dataType>GRID</dataType>
    </metadata>

    <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
      <aggregation dimName="time" type="joinExisting">
        <variableAgg name="cee"/>
        <variableAgg name="cee_corr_unc"/>
        <scan location="/neodc/esacci/cloud/data/phase-2/L3C/AVHRR-PM/v2.0" 
suffix=".nc"/>
      </aggregation>
    </netcdf>
  </dataset>

</catalog>