Re: IOOS test data set goes away?

Hi Rich,

Rich Signell wrote:
> Steve,
>
> We had a problem where periodically we would show up in the morning to
> find that our bathy_catalog.xml would only have 1 entry instead of the
> 30 or so that it should have.   And restarting tomcat fixed the
> problem.   Eventually we discovered that by blocking web crawlers our
> problem went away, and the reaso speculated by our sys admin was that
> the crawlers were gobbling up all the tomcat memory.
>   

If so, it sounds like a tomcat memory leak. That's one thing that I've 
always disliked about Java. Even though it is object-oriented and it 
should be easy for deconstructors to free up the memory it never seems 
to work that way.

I remember you sent an email to the IOOS model data interoperability 
group about this. I'll give it a try and see if that helps. Also, I 
uncommented the logging section of the tomcat conf/server.xml to start 
logging requests.

> So yes, we've seen problems where only one THREDDS catalog seems to be
> affected, perhaps by memory issues, and restarting Tomcat helps.
>
> Perhaps your attempt to aggregate 3 years of forecast data into the
> FMRC aggregation is blowing things out?
>   

Is there a rule of thumb for knowing how many elements is reasonable for 
aggregation? Is 1000 too much? 10000?

> I think we need a tutorial (webinar?) on caching, aggregation and memory...
>   

That would be very helpful now that I have a better idea about what is 
happening.

Steve

> -Rich
>
>
> On Thu, May 28, 2009 at 9:49 AM, Steve Cousins <address@hidden> wrote:
>   
> > Hi Rich,
> >
> > Do you have problems with THREDDS data sets going away? I came in this
> > morning and the Monthly Average data set was available but the IOOS Test one
> > was not available. I didn't see anything to speak of in the logs about it.
> > Restarting Tomcat brought it back. I remember tomcat being a problem in LAS
> > a number of years ago but usually the whole thing would be hosed not just a
> > single data set. Is this to be expected?
> >
> > Thanks,
> >
> > Steve
> >
> >     
>
>
>
>