[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: THREDDS performance [was Re: THREDDS and grib]

Hi John -
I realize this email thread goes back a ways, but I was wondering if there'd been any recent work done regarding the performance of Union aggregations in TDS. We (LAS group) are still working with the GFDL data collection and finding that it is still really slow to initialize the aggregations (sometimes as long as 7 minutes!). I've actually moved all of the data to a public TDS at 
   http://data1.gfdl.noaa.gov:8380/thredds/ipcc/all_ipcc.html

So that they can be accessed by anyone.
Like I said, I realize this discussion was started a while ago, but just thought I'd touch base and see if there were any new thoughts that I could try in order to speed up performance. I've been working a lot with this data and installing our new LAS on top of the data, which is of course why it's on my mind... 

Thanks -
Kevin

John Caron wrote:



Kevin O'Brien wrote:

Hi John

John Caron wrote:

- can you send me the cache file for the test dataset?
Would I find this in the $TOMCAT_HOME/content/thredds/cacheAged directory? Because what I see in there are the cache files for the JoinExisting aggregations, which then get put into a Union aggregation, and it is the Union aggregation URL that is being accessed and is slow. There doesn't seem to be any cache entries, in that directory anyway, representing the Union aggregation. Are they somewhere else? 

only the joinExisting aggs get cached.
- can yuo give me approximate times of what you see vs what you would expect ?
On example took 3 minutes and 40 seconds to open up the dataset html page through the OPeNDAP html interface. That particular example was a Union aggregation of 47 JoinExisting aggregations. Maybe that's just the time it should take? And if that is the case, then I think we can live with it on the first access, but then the question is why not use the cached info after a tomcat restart?
that seems ok for the first time, but then it should use the cache and be much faster. 
im guessing theres some other problem i dont see yet.
can you send me the ncml for the union, and also one or more of the aggExisting cached files. 


kev


Kevin O'Brien wrote:

Hi John -
I installed the version 3.16.37 of the server and unfortunately, it doesn't seem to make the problem go away. It did kind of seem like things were a bit quicker at times, but it was hard to accurately assess. Of course, the initial access of these large aggregations is still pretty slow, and then subsequent accesses are faster. However, it does seem like a restart of the tomcat server somehow erases the cache information and so every initial access after a tomcat reboot is slow. 
Is there anything else I can do to help further debug the problem?

thanks -
Kevin

John Caron wrote:
Hi Kevin: I made a small fix that looks like it would affect your case, but im not convinced it really would cause a huge slowdown. anyway, i wonder if you would give it a try and let me know? 

Its TDS release 3.16.37.

thanks for your patience

Kevin O'Brien wrote:


Hi John -
Not to be a pest - but I was wondering if you'd had a chance to look at these performance issues, or even been able to recreate them? 

Thanks -
kevin

John Caron wrote:
these are all good questions - there have been similar reports of the agg cache not working like it should. i will have to reproduce to see whats happening. 

Kevin O'Brien wrote:


Hi John -
I tried what you suggested and it didn't seem to have a significant effect in making the initial access of the aggregated dataset quicker. It still took over a minute and a half to open the dataset. I've pasted the xml config that I used to define the new aggregation below. To be honest, I'm actually kind of glad because I wasn't looking forward to modifying the guts of the application which generates the xml config automatically.... :-)
I guess I can understand and probably even accept the fact that for the first time the dataset is accessed, things will be a little slow. After that, I presume the dataset is available in the cache, and of course subsequent accesses prove that it is because the response is quite quick. However, if the tomcat server is restarted, it seems like whatever is in the cache is ignored and the cache entries have to be rebuilt. I have my aggregation cache set like so: 

 <AggregationCache>
<dir>/home/pmel/DataPortal/apache-tomcat-5.5.25/content/thredds/cacheAged/</dir> 
   <scour>24 hours</scour>
   <maxAge>90 days</maxAge>
</AggregationCache> Does that seem correct? Also, as an aside, you mention that you thought this would be quicker because it avoids the OPeNDAP URL's....Shouldn't there be some client side caching done w/ the OPeNDAP datasets? For example, if I access a remote dataset with ncdump (or Ferret), and my OPeNDAP caching is turned on my ~/.dodsrc file, it will cache the response in the ~/.dods_cache directory. Does any of that happen when OPeNDAP URL's are accessed through TDS??? 

Anyway - here's the xml config I used as per your suggestion:
<dataset ID="CM2.1U-D4_1PctTo2X_I1 atmos daily all vars 00010101-02201231_2" name="CM2.1U-D4_1PctTo2X_I1 atmos daily all vars 00010101-02201231_2" urlPath="ipcc_ar4_CM2.1_R1_1to2x-1_daily_atmos_00010101-02201231_2"> 
       <serviceName>thisDODS3</serviceName>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> 
         <aggregation type="union">
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> 
              <aggregation dimName="time" type="joinExisting">
<netcdf location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/pr_A2.00010101-01001231.nc" ncoords="36500" /> <netcdf location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/pr_A2.01010101-02001231.nc" ncoords="36500" /> <netcdf location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/pr_A2.02010101-02201231.nc" ncoords="7300" /> 
              </aggregation>
            </netcdf>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> 
              <aggregation dimName="time" type="joinExisting">
<netcdf location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmax_A2.00010101-01001231.nc" ncoords="36500" /> <netcdf location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmax_A2.01010101-02001231.nc" ncoords="36500" /> <netcdf location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmax_A2.02010101-02201231.nc" ncoords="7300" /> 
              </aggregation>
            </netcdf>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";> 
              <aggregation dimName="time" type="joinExisting">
<netcdf location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmin_A2.00010101-01001231.nc" ncoords="36500" /> <netcdf location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmin_A2.01010101-02001231.nc" ncoords="36500" /> <netcdf location="file:/data/gfdl_cm2_1/CM2.1U-D4_1PctTo2X_I1/pp/atmos/ts/daily/tasmin_A2.02010101-02201231.nc" ncoords="7300" /> 
              </aggregation>
            </netcdf>
         </aggregation>
       </netcdf>
   </dataset>


I'm open to any suggestions or ideas!

thanks -
kevin


John Caron wrote:

Hi Kevin:
I havent had time to reproduce this yet, but im guessing one source of the slowdown is using opendap URLS in the compound aggregation. It would be interesting to time 1) the single aggregations, 2) the compound agg as it exists, and 3) the compound agg, but replace the opendap URLs with direct netcdf files, 

see attached file










--
Kevin O'Brien                   UW/JISAO        
Research Scientist              NOAA/PMEL/TMAP
206-526-6751                    http://www.pmel.noaa.gov
"The contents of this message are mine personally and do not necessarily reflect any position of the Government or the National Oceanic and Atmospheric Administration."