[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: phone message -- sample XML to follow (here it is)
- Subject: Re: phone message -- sample XML to follow (here it is)
- Date: Thu, 08 Jan 2009 17:56:26 -0700
get this version:
ftp://ftp.unidata.ucar.edu/pub/thredds/4.0/thredds.war
its not quite beta - we are still trying to finish a few things. but feedback
would be very useful.
BTW, note the ids on the joinExisting netcdf elements. these are needed to make
the caching work.
thanks
Kevin O'Brien wrote:
>
> Hi John -
>
> This sounds great - and good timing too! I'd be very happy to try a
> beta version and test it out on our server at GFDL. Steve and I are
> actually at GFDL right now, and heading back to Seattle tonight. Can
> I grab the beta from the usual place on the THREDDS pages?
>
> thanks!
> Kevin
>
> John Caron wrote:
>> Hi guys:
>>
>> The good news is that Ive found the problem with the caching.
>> Performance now is a lot better, though i dont have a measurement, and
>> a lot may depend on your server.
>>
>> The bad news (maybe) is that I am only going to fix this in the 4.0
>> version of NcML/TDS. We are pushing hard to get this out to beta this
>> month. Id love to have you start to use it, to get feedback on other
>> issues that may be lurking.
>>
>> The main problem was the "anonymous" inner aggregations. To get the
>> caching right, we need to give them ids, eg:
>>
>> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>> <aggregation dimName="time" type="joinExisting">
>> <netcdf ncoords="36500" id="first100">
>> <aggregation type="union">
>> <netcdf location="pr_A2.00010101-01001231.nc"/>
>> <netcdf location="tasmax_A2.00010101-01001231.nc"/>
>> <netcdf location="tasmin_A2.00010101-01001231.nc"/>
>> </aggregation>
>> </netcdf>
>> <netcdf ncoords="36500" id="sec100">
>> <aggregation type="union">
>> <netcdf location="pr_A2.01010101-02001231.nc"/>
>> <netcdf location="tasmax_A2.01010101-02001231.nc"/>
>> <netcdf location="tasmin_A2.01010101-02001231.nc"/>
>> </aggregation>
>> </netcdf>
>> <netcdf ncoords="36500" id="third100">
>> <aggregation type="union">
>> <netcdf location="pr_A2.02010101-03001231.nc"/>
>> <netcdf location="tasmax_A2.02010101-03001231.nc"/>
>> <netcdf location="tasmin_A2.02010101-03001231.nc"/>
>> </aggregation>
>> </netcdf>
>> </aggregation>
>> </netcdf>
>>
>> I might be able to generate auto ids, but for now they have to be
>> added by hand. As I said, this will only be useful in the 4.0 version.
>> Ill get a release out later today in case you want to try it.
>>
>> John
>>
>> Steve Hankin wrote:
>>
>>> Hi John,
>>>
>>> Thanks for looking into this. At this moment Kevin is modifying the
>>> code that creates the ncML aggregation configuration from the contents
>>> of our database. It looks like we will be "down to the wire" in seeing
>>> how much faster TDS becomes when we start using the improved ncML (the
>>> changes are bigger than just moving the ncoords attribute).
>>> Can we ask you to "stand by" and maybe be willing to set your peepers on
>>> it later today? Kevin's preliminary tests indicated that we will still
>>> getting the cache hit failures (that for unknown reasons TDS rebuilds
>>> the aggregation in cache instead of reusing what it saved
>>> previously). But we don't have an up-to-date TDS site to show you yet.
>>>
>>> - Steve
>>>
>>> John Caron wrote:
>>>
>>>> hi kevin:
>>>>
>>>> your ftp site is pretty slow (600 KB/sec) - is it throttled, or just
>>>> overwhelmed? should i wait until tonight to try to download these
>>>> files?
>>>>
>>>> Kevin OBrien wrote:
>>>>
>>>>
>>>>> Hi John -
>>>>>
>>>>> I did as you suggested and moved the ncoords attribute to the outer
>>>>> aggregation and I was able to get to the aggregation in around 28
>>>>> seconds. Just to confirm that it wasn't something system-related, I
>>>>> changed the xml configuration back, and verified that when the ncoords
>>>>> attribute was in the inner aggregation, it took around 2 minutes to
>>>>> open. So that's a big speed improvement! I think I will expand
>>>>> the
>>>>> aggregation xml to include full experiments and see how the
>>>>> performance
>>>>> changes.
>>>>>
>>>>> By the way, you can get all of these files at
>>>>>
>>>>>
ftp://nomads.gfdl.noaa.gov/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> and you'll see there are many more that would actually be configured
>>>>> into the aggregation..
>>>>>
>>>>> One thing I did notice and have a question about - after I moved the
>>>>> ncoords attribute to the outer aggregation, and I restarted the
>>>>> server -
>>>>> a cache file showed up in the cacheAged directory. When I then just
>>>>> restarted the server to test the use of cache, after I had opened the
>>>>> aggregation (which again took around 30 seconds), I noticed that the
>>>>> cache file in the cacheAged directory had apparently been updated (at
>>>>> least the time stamp of the file was new). If nothing in the
>>>>> aggregation has changed, should it be updating the cache file? Or
>>>>> should it use the cache file already there?
>>>>>
>>>>> thanks -
>>>>> kevin
>>>>>
>>>>> John Caron wrote:
>>>>>
>>>>>> Hi Kevin, Steve:
>>>>>>
>>>>>> You should try putting the ncoords attribute on the outer
>>>>>> aggregation:
>>>>>>
>>>>>> <netcdf
>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";
>>>>>> ncoords="36500">
>>>>>> <aggregation type="union">
>>>>>> <netcdf
>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/pr_A2.00010101-01001231.nc"
>>>>>> />
>>>>>> <netcdf
>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmax_A2.00010101-01001231.nc"
>>>>>>
>>>>>> />
>>>>>> <netcdf
>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmin_A2.00010101-01001231.nc"
>>>>>>
>>>>>> />
>>>>>> </aggregation>
>>>>>> </netcdf>
>>>>>>
>>>>>> let me know if that helps.
>>>>>>
>>>>>> Id like to test this nested aggregation as a use case . Can I get
>>>>>> those 9 files? thanks.
>>>>>>
>>>>>>
>>>>>> Steve Hankin wrote:
>>>>>>
>>>>>>> (This is a continuation of the conversation that Kevin O'Brien
>>>>>>> started with you.)
>>>>>>>
>>>>>>> Hi John,
>>>>>>>
>>>>>>> Below is the ncML and TDS configuration information. It all "works"
>>>>>>> ... except the caching. Any clues?
>>>>>>>
>>>>>>> - Steve
>>>>>>>
>>>>>>> ===
>>>>>>>
>>>>>>> This from threddsConfig.xml
>>>>>>>
>>>>>>> <AggregationCache>
>>>>>>>
>>>>>>>
<dir>/home/pmel/DataPortal/apache-tomcat-5.5.25/content/thredds/cacheAged/</dir>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> <scour>24 hours</scour>
>>>>>>> <maxAge>90 days</maxAge>
>>>>>>> </AggregationCache> ===
>>>>>>>
>>>>>>> And this is the latest ncML that Kevin tested: "It took nearly two
>>>>>>> minutes to open the aggregation the first time. After that,
>>>>>>> accesses
>>>>>>> were quick -- evidently caching was working. Then I restarted the
>>>>>>> tomcat server, and again it took nearly two minutes to open the
>>>>>>> aggregation. I could see that the cache file in the caching
>>>>>>> directory was again updated after the second tomcat restart (ie, the
>>>>>>> cache was rewritten rather than used)..."
>>>>>>>
>>>>>>> <catalog name="test IPCC Datasets"
>>>>>>>
>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/thredds/InvCatalog/v1.0";
>>>>>>>
>>>>>>> xmlns:xlink="http://www.w3.org/1999/xlink";>
>>>>>>>
>>>>>>> <service name="thisDODS3" serviceType="OpenDAP"
>>>>>>> base="/thredds/dodsC/" />
>>>>>>>
>>>>>>>
>>>>>>> <dataset ID="CM2Q-d2_1PctTo4x_j1 atmos daily all vars
>>>>>>> 00010101-03001231 test" name="CM2Q-d2_1PctTo4x_j1 atmos daily all
>>>>>>> vars 00010101-03001231 test"
>>>>>>> urlPath="ipcc_ar4_CM2.0_R1_1to4x-0_daily_atmos_00010101-03001231_test">
>>>>>>>
>>>>>>>
>>>>>>> <serviceName>thisDODS3</serviceName>
>>>>>>> <netcdf
>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>>>>>>> <aggregation dimName="time" type="joinExisting">
>>>>>>> <netcdf
>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>>>>>>> <aggregation type="union">
>>>>>>> <netcdf
>>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/pr_A2.00010101-01001231.nc"
>>>>>>>
>>>>>>>
>>>>>>> ncoords="36500" />
>>>>>>> <netcdf
>>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmax_A2.00010101-01001231.nc"
>>>>>>>
>>>>>>>
>>>>>>> ncoords="36500" />
>>>>>>> <netcdf
>>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmin_A2.00010101-01001231.nc"
>>>>>>>
>>>>>>>
>>>>>>> ncoords="36500" />
>>>>>>> </aggregation>
>>>>>>> </netcdf>
>>>>>>> <netcdf
>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>>>>>>> <aggregation type="union">
>>>>>>> <netcdf
>>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/pr_A2.01010101-02001231.nc"
>>>>>>>
>>>>>>>
>>>>>>> ncoords="36500" />
>>>>>>> <netcdf
>>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmax_A2.01010101-02001231.nc"
>>>>>>>
>>>>>>>
>>>>>>> ncoords="36500" />
>>>>>>> <netcdf
>>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmin_A2.01010101-02001231.nc"
>>>>>>>
>>>>>>>
>>>>>>> ncoords="36500" />
>>>>>>> </aggregation>
>>>>>>> </netcdf>
>>>>>>> <netcdf
>>>>>>> xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
>>>>>>> <aggregation type="union">
>>>>>>> <netcdf
>>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/pr_A2.02010101-03001231.nc"
>>>>>>>
>>>>>>>
>>>>>>> ncoords="36500"/>
>>>>>>> <netcdf
>>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmax_A2.02010101-03001231.nc"
>>>>>>>
>>>>>>>
>>>>>>> ncoords="36500"/>
>>>>>>> <netcdf
>>>>>>>
location="file:/data/gfdl_cm2_0/CM2Q-d2_1PctTo4x_j1/pp/atmos/ts/daily/tasmin_A2.02010101-03001231.nc"
>>>>>>>
>>>>>>>
>>>>>>> ncoords="36500" />
>>>>>>> </aggregation>
>>>>>>> </netcdf>
>>>>>>> </aggregation>
>>>>>>> </netcdf>
>>>>>>> </dataset>
>>>>>>>
>>>>>>> </catalog>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Steve Hankin wrote:
>>>>>>>
>>>>>>>> Hi John,
>>>>>>>>
>>>>>>>> We're at the phone number in the signature line below. Will follow
>>>>>>>> this email shortly with some XML fragments ... hoping maybe you
>>>>>>>> have
>>>>>>>> a suggestion.
>>>>>>>>
>>>>>>>> - Steve
>>>>>>>>
>>>>>>>>
>>>>>>> --
>>>>>>> Steve Hankin, NOAA/PMEL -- address@hidden
>>>>>>> 7600 Sand Point Way NE, Seattle, WA 98115-0070
>>>>>>> ph. (206) 526-6080, FAX (206) 526-6744
>>>>>>>
>>>>>>> "The only thing necessary for the triumph of evil is for good men
>>>>>>> to do nothing." -- Edmund Burke
>>>>>>>
>>>>>>>