[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[THREDDS #MAS-184300]: metadata duplication and datasets/catalogs disappearance when requesting xml versions of datasets

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.


  • Subject: [THREDDS #MAS-184300]: metadata duplication and datasets/catalogs disappearance when requesting xml versions of datasets
  • Date: Thu, 01 Jul 2010 16:47:55 -0600

Hi Jonathan,

It looks like this only affects catalogs that the TDS caches in memory. If you 
do the same type of process with a catalog that is generated from a datasetScan 
you don't have this problem. Looks like our subset method modifies the catalog 
object rather than making a copy. 

Also, the data access services to any "lost" datasets still work as do the 
catalogs the "lost" catalogRef-s reference. It appears to only affect the 
catalog that gets subset as most of the other stuff is cached separately.

I haven't dug into the duplicate metadata issue yet. I'm guessing it is related 
to the same "changing the underlying catalog" problem.

Anyway, I've moved these bugs up in our priority list a bit. I'll notify you 
when we get them fixed.

Thanks,

Ethan

> Further tests demonstrate that it does not affect other
> catalogs/datasets that reside
> at differents levels, only all catalogRef and dataset contained in the
> one who initially
> caused the 'bug'
> 
> In fact, I realized that when making a request to a catalog's dataset
> like this:
> 
> https://localhost:8443/thredds/catalog.xml?dataset=test.nc,
> 
> the server then seems like 'blocked' on that xml (or whatever object is
> beside this xml)
> 
> Then, any further request like to the containing catalog like this:
> 
> https://localhost:8443/thredds/catalog.xml
> 
> or to the same dataset, or any other at the same level:
> 
> https://localhost:8443/thredds/catalog.xml?dataset=test.nc
> https://localhost:8443/thredds/catalog.xml?dataset=testnotime.nc
> ...
> results in the same xml/html as
> https://localhost:8443/thredds/catalog.xml?dataset=test.nc
> 
> 
> This is also reproductible with any other catalogs that are in lower
> levels in the hierarchy.
> 
> 
> Another thing I noticed, the <catalog name="Test catalog"...> is the
> same in
> https://localhost:8443/thredds/catalog.xml
> and in
> https://localhost:8443/thredds/catalog.xml?dataset=test.nc
> which is normal as test.nc is a dataset nested into the root catalog (so
> in the same catalog)
> but perhaps it my be related ...
> 
> 
> Regards,
> 
> Jonathan
> 
> 


And the earlier email:

> I'm experiencing a strange thing in Thredds 4.2.20100607.1833 (at least 
> this version, I think also with previous 4.x series)
> 
> I'm currently developping a web client (js/php) to access and display 
> data from opendap/wms services.
> It use the xml versions of catalogs and dataset for browsing thredds 
> (plus ddx for opendap, and xml from wms services access)
> 
> For this test case, I'm only accessing the dataset's xml
> 
> In the first access, all is ok but when making a new request, the 
> dataset metadata is duplicated as in the
> returned xml below (documentation and property elements of the metadata):
> 
> 
> <catalog name="Test catalog" version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/"/>
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/"/>
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/"/>
> <service name="wms" serviceType="WMS" base="/thredds/wms/"/>
> </service>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar"/>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar"/>
> <property name="Producer" value="Actimar"/>
> <property name="Producer" value="Actimar"/>
> </metadata>
> </dataset>
> </catalog>
> 
> If I make a third request, I then get four documentation/property 
> elements for each instead of one.
> If i make another request, eight elements, and so on.
> 
> This is a bit annoying but there is a more serious problem: all other 
> datasets are made unavailable,
> at least at the same level of the one requested above (which in fact is 
> in the root catalog) !
> I mean they are then not displayed either in the html or xml form of the 
> catalog.
> Try to access the other datasets give me a 404 error
> 
> However, requesting in a different catalogRoot hierarchy still works fine
> 
> Note that without having previously done xml requests and if I only 
> access the server using the traditionnal way (html pages delivered by 
> thredds), all work fine.
> 
> =====================================
> 
> The steps to reproduce this are:
> 
> # wget -O - http://localhost:8080/thredds/catalog.xml
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
> xmlns:xlink="http://www.w3.org/1999/xlink"; name="Test catalog" 
> version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> </service>
> <dataset name="Actimar" ID="actimarTest">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <property name="Producer" value="Actimar" />
> </metadata>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc" />
> <dataset name="testonetime.nc" ID="testonetime.nc" 
> urlPath="test/testonetime.nc" />
> <dataset name="testnotime.nc" ID="testnotime.nc" 
> urlPath="test/testnotime.nc" />
> <dataset name="Basic Tests" ID="basicTests">
> <catalogRef name="" xlink:href="/thredds/catalog/wrfBRE_d01/catalog.xml" 
> xlink:title="wrfBRE_d01">
> <property name="DatasetScan" value="true" />
> </catalogRef>
> <dataset name="wrfBRE_d01_wind_2010-01-01_00.nc" 
> ID="wrfBRE_d01_wind_2010-01-01_00.nc" 
> urlPath="data/wrfBRE_d01_wind_2010-01-01_00.nc" />
> <dataset name="wrfBRE_d01_temp_2010-01-01_00.nc" 
> ID="wrfBRE_d01_temp_2010-01-01_00.nc" 
> urlPath="data/wrfBRE_d01_temp_2010-01-01_00.nc" />
> </dataset>
> <dataset name="Restricted Tests" ID="restrictedTests" 
> restrictAccess="restrictedDatasetUser">
> <catalogRef name="" restrictAccess="restrictedDatasetUser" 
> xlink:href="/thredds/catalog/wrfBRE_d01_restricted/catalog.xml" 
> xlink:title="wrfBRE_d01_restricted">
> <property name="DatasetScan" value="true" />
> </catalogRef>
> </dataset>
> <dataset name="Compressed Tests" ID="compressedTest">
> <dataset name="Gzipped: wrfBRE_d01_wind_2010-01-01_00.nc.gz" 
> ID="wrfBRE_d01_wind_2010-01-01_00.nc.gz" 
> urlPath="data/wrfBRE_d01_wind_2010-01-01_00.nc.gz" />
> <dataset name="Bzipped: wrfBRE_d01_wind_2010-01-01_00.nc.bz2" 
> ID="wrfBRE_d01_wind_2010-01-01_00.nc.bz2" 
> urlPath="data/wrfBRE_d01_wind_2010-01-01_00.nc.bz2" />
> <dataset name="Zipped: wrfBRE_d01_wind_2010-01-01_00.nc.zip" 
> ID="wrfBRE_d01_wind_2010-01-01_00.nc.zip" 
> urlPath="data/wrfBRE_d01_wind_2010-01-01_00.nc.zip" />
> </dataset>
> <dataset name="NCML Tests" ID="ncmlTests">
> <dataset name="wrfBRE_d01_temp Modified using ncml" 
> ID="wrfBRE_d01_temp_Ncml" urlPath="wrfBRE_d01_temp_Ncml" />
> <dataset name="wrfBRE_d01_temp Forecast Model Run Collection 
> Aggregation" ID="wrfBRE_d01_temp_Fmrc" urlPath="wrfBRE_d01_temp_Fmrc" />
> <dataset name="wrfBRE_d01_temp Aggregation JoinExisting (join along time 
> dimension)" ID="wrfBRE_d01_temp_JoinExisting" 
> urlPath="wrfBRE_d01_temp_JoinExisting">
> <timeCoverage>
> <start>2009-01-01T01:00:00</start>
> <end>2009-01-07T01:00:00</end>
> </timeCoverage>
> </dataset>
> <dataset name="wrfBRE_d01_temp Aggregation JoinNew (+ runtime 
> dimension)" ID="wrfBRE_d01_temp_JoinNew" urlPath="wrfBRE_d01_temp_JoinNew">
> <timeCoverage>
> <start>2009-01-01T01:00:00</start>
> <end>2009-01-07T01:00:00</end>
> </timeCoverage>
> </dataset>
> </dataset>
> </dataset>
> </catalog>
> 
> =====================================
> 
> # wget -O - http://localhost:8080/thredds/catalog.xml?dataset=test.nc
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
> xmlns:xlink="http://www.w3.org/1999/xlink"; name="Test catalog" 
> version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> </service>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <property name="Producer" value="Actimar" />
> </metadata>
> </dataset>
> </catalog>
> 
> Past this first dataset xml request, only test.nc dataset is available.
> 
> Each following dataset xml request on test.nc duplicate the metadata 
> elements.
> 
> =====================================
> 
> # wget -O - http://localhost:8080/thredds/catalog.xml?dataset=test.nc
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
> xmlns:xlink="http://www.w3.org/1999/xlink"; name="Test catalog" 
> version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> </service>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> </metadata>
> </dataset>
> </catalog>
> 
> =====================================
> 
> # wget -O - http://localhost:8080/thredds/catalog.xml?dataset=test.nc
> 
> ?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
> xmlns:xlink="http://www.w3.org/1999/xlink"; name="Test catalog" 
> version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> </service>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> </metadata>
> </dataset>
> </catalog>
> 
> =====================================
> 
> # wget -O - http://localhost:8080/thredds/catalog.xml
> 
> <?xml version="1.0" encoding="UTF-8"?>
> <catalog 
> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0"; 
> xmlns:xlink="http://www.w3.org/1999/xlink"; name="Test catalog" 
> version="1.0.1" expires="2100-01-01T00:00:00">
> <service name="multiServices" serviceType="Compound" base="">
> <service name="opendap" serviceType="OPENDAP" base="/thredds/dodsC/" />
> <service name="http" serviceType="HTTPServer" base="/thredds/fileServer/" />
> <service name="wcs" serviceType="WCS" base="/thredds/wcs/" />
> <service name="wms" serviceType="WMS" base="/thredds/wms/" />
> </service>
> <dataset name="test.nc" ID="test.nc" urlPath="test/test.nc">
> <metadata inherited="true">
> <serviceName>multiServices</serviceName>
> <dataType>GRID</dataType>
> <dataFormat>NetCDF</dataFormat>
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <documentation xlink:href="http://www.actimar.fr"; xlink:title="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> <property name="Producer" value="Actimar" />
> </metadata>
> </dataset>
> </catalog>
> 
> 
> Did you already met this ?
> 
> 
> Best regards
> 
> Jonathan

Ticket Details
===================
Ticket ID: MAS-184300
Department: Support THREDDS
Priority: Normal
Status: Open