[THREDDS #MAS-184300]: metadata duplication and datasets/catalogs disappearance when requesting xml versions of datasets

Hi Jonathan,

It looks like this only affects catalogs that the TDS caches in memory. If you 
do the same type of process with a catalog that is generated from a datasetScan 
you don't have this problem. Looks like our subset method modifies the catalog 
object rather than making a copy. 

Also, the data access services to any "lost" datasets still work as do the 
catalogs the "lost" catalogRef-s reference. It appears to only affect the 
catalog that gets subset as most of the other stuff is cached separately.

I haven't dug into the duplicate metadata issue yet. I'm guessing it is related 
to the same "changing the underlying catalog" problem.

Anyway, I've moved these bugs up in our priority list a bit. I'll notify you 
when we get them fixed.

Thanks,

Ethan

> Further tests demonstrate that it does not affect other
> catalogs/datasets that reside
> at differents levels, only all catalogRef and dataset contained in the
> one who initially
> caused the 'bug'
> 
> In fact, I realized that when making a request to a catalog's dataset
> like this:
> 
> https://localhost:8443/thredds/catalog.xml?dataset=test.nc,
> 
> the server then seems like 'blocked' on that xml (or whatever object is
> beside this xml)
> 
> Then, any further request like to the containing catalog like this:
> 
> https://localhost:8443/thredds/catalog.xml
> 
> or to the same dataset, or any other at the same level:
> 
> https://localhost:8443/thredds/catalog.xml?dataset=test.nc
> https://localhost:8443/thredds/catalog.xml?dataset=testnotime.nc
> ...
> results in the same xml/html as
> https://localhost:8443/thredds/catalog.xml?dataset=test.nc
> 
> 
> This is also reproductible with any other catalogs that are in lower
> levels in the hierarchy.
> 
> 
> Another thing I noticed, the <catalog name="Test catalog"...> is the
> same in
> https://localhost:8443/thredds/catalog.xml
> and in
> https://localhost:8443/thredds/catalog.xml?dataset=test.nc
> which is normal as test.nc is a dataset nested into the root catalog (so
> in the same catalog)
> but perhaps it my be related ...
> 
> 
> Regards,
> 
> Jonathan
> 
> 


And the earlier email:

> I'm experiencing a strange thing in Thredds 4.2.20100607.1833 (at least 
> this version, I think also with previous 4.x series)
> 
> I'm currently developping a web client (js/php) to access and display 
> data from opendap/wms services.
> It use the xml versions of catalogs and dataset for browsing thredds 
> (plus ddx for opendap, and xml from wms services access)
> 
> For this test case, I'm only accessing the dataset's xml
> 
> In the first access, all is ok but when making a new request, the 
> dataset metadata is duplicated as in the
> returned xml below (documentation and property elements of the metadata):
> 
> 
> <catalog name="Test catalog" version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/"/>
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/"/>
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/"/>
> <service name="wms" serviceType="WMS" base="/thredds/wms/"/>
> </service>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar"/>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar"/>
> <property name="Producer" value="Actimar"/>
> <property name="Producer" value="Actimar"/>
> </metadata>
> </dataset>
> </catalog>
> 
> If I make a third request, I then get four documentation/property 
> elements for each instead of one.
> If i make another request, eight elements, and so on.
> 
> This is a bit annoying but there is a more serious problem: all other 
> datasets are made unavailable,
> at least at the same level of the one requested above (which in fact is 
> in the root catalog) !
> I mean they are then not displayed either in the html or xml form of the 
> catalog.
> Try to access the other datasets give me a 404 error
> 
> However, requesting in a different catalogRoot hierarchy still works fine
> 
> Note that without having previously done xml requests and if I only 
> access the server using the traditionnal way (html pages delivered by 
> thredds), all work fine.
> 
> =====================================
> 
> The steps to reproduce this are:
> 
> # wget -O - http://localhost:8080/thredds/catalog.xml
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
> xmlns:xlink="http://www.w3.org/1999/xlink"; name="Test catalog" 
> version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> </service>
> <dataset name="Actimar" ID="actimarTest">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <property name="Producer" value="Actimar" />
> </metadata>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc" />
> <dataset name="testonetime.nc" ID="testonetime.nc" 
> urlPath="test/testonetime.nc" />
> <dataset name="testnotime.nc" ID="testnotime.nc" 
> urlPath="test/testnotime.nc" />
> <dataset name="Basic Tests" ID="basicTests">
> <catalogRef name="" xlink:href="/thredds/catalog/wrfBRE_d01/catalog.xml" 
> xlink:title="wrfBRE_d01">
> <property name="DatasetScan" value="true" />
> </catalogRef>
> <dataset name="wrfBRE_d01_wind_2010-01-01_00.nc" 
> ID="wrfBRE_d01_wind_2010-01-01_00.nc" 
> urlPath="data/wrfBRE_d01_wind_2010-01-01_00.nc" />
> <dataset name="wrfBRE_d01_temp_2010-01-01_00.nc" 
> ID="wrfBRE_d01_temp_2010-01-01_00.nc" 
> urlPath="data/wrfBRE_d01_temp_2010-01-01_00.nc" />
> </dataset>
> <dataset name="Restricted Tests" ID="restrictedTests" 
> restrictAccess="restrictedDatasetUser">
> <catalogRef name="" restrictAccess="restrictedDatasetUser" 
> xlink:href="/thredds/catalog/wrfBRE_d01_restricted/catalog.xml" 
> xlink:title="wrfBRE_d01_restricted">
> <property name="DatasetScan" value="true" />
> </catalogRef>
> </dataset>
> <dataset name="Compressed Tests" ID="compressedTest">
> <dataset name="Gzipped: wrfBRE_d01_wind_2010-01-01_00.nc.gz" 
> ID="wrfBRE_d01_wind_2010-01-01_00.nc.gz" 
> urlPath="data/wrfBRE_d01_wind_2010-01-01_00.nc.gz" />
> <dataset name="Bzipped: wrfBRE_d01_wind_2010-01-01_00.nc.bz2" 
> ID="wrfBRE_d01_wind_2010-01-01_00.nc.bz2" 
> urlPath="data/wrfBRE_d01_wind_2010-01-01_00.nc.bz2" />
> <dataset name="Zipped: wrfBRE_d01_wind_2010-01-01_00.nc.zip" 
> ID="wrfBRE_d01_wind_2010-01-01_00.nc.zip" 
> urlPath="data/wrfBRE_d01_wind_2010-01-01_00.nc.zip" />
> </dataset>
> <dataset name="NCML Tests" ID="ncmlTests">
> <dataset name="wrfBRE_d01_temp Modified using ncml" 
> ID="wrfBRE_d01_temp_Ncml" urlPath="wrfBRE_d01_temp_Ncml" />
> <dataset name="wrfBRE_d01_temp Forecast Model Run Collection 
> Aggregation" ID="wrfBRE_d01_temp_Fmrc" urlPath="wrfBRE_d01_temp_Fmrc" />
> <dataset name="wrfBRE_d01_temp Aggregation JoinExisting (join along time 
> dimension)" ID="wrfBRE_d01_temp_JoinExisting" 
> urlPath="wrfBRE_d01_temp_JoinExisting">
> <timeCoverage>
> <start>2009-01-01T01:00:00</start>
> <end>2009-01-07T01:00:00</end>
> </timeCoverage>
> </dataset>
> <dataset name="wrfBRE_d01_temp Aggregation JoinNew (+ runtime 
> dimension)" ID="wrfBRE_d01_temp_JoinNew" urlPath="wrfBRE_d01_temp_JoinNew">
> <timeCoverage>
> <start>2009-01-01T01:00:00</start>
> <end>2009-01-07T01:00:00</end>
> </timeCoverage>
> </dataset>
> </dataset>
> </dataset>
> </catalog>
> 
> =====================================
> 
> # wget -O - http://localhost:8080/thredds/catalog.xml?dataset=test.nc
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
> xmlns:xlink="http://www.w3.org/1999/xlink"; name="Test catalog" 
> version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> </service>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <property name="Producer" value="Actimar" />
> </metadata>
> </dataset>
> </catalog>
> 
> Past this first dataset xml request, only test.nc dataset is available.
> 
> Each following dataset xml request on test.nc duplicate the metadata 
> elements.
> 
> =====================================
> 
> # wget -O - http://localhost:8080/thredds/catalog.xml?dataset=test.nc
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
> xmlns:xlink="http://www.w3.org/1999/xlink"; name="Test catalog" 
> version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> </service>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> </metadata>
> </dataset>
> </catalog>
> 
> =====================================
> 
> # wget -O - http://localhost:8080/thredds/catalog.xml?dataset=test.nc
> 
> ?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
> xmlns:xlink="http://www.w3.org/1999/xlink"; name="Test catalog" 
> version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> </service>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> </metadata>
> </dataset>
> </catalog>
> 
> =====================================
> 
> # wget -O - http://localhost:8080/thredds/catalog.xml
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
> xmlns:xlink="http://www.w3.org/1999/xlink"; name="Test catalog" 
> version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> </service>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> </metadata>
> </dataset>
> </catalog>
> 
> 
> Did you already met this ?
> 
> 
> Best regards
> 
> Jonathan

Ticket Details
===================
Ticket ID: MAS-184300
Department: Support THREDDS
Priority: Normal
Status: Open