Re: Thredds out of memory

John Caron wrote:

> Tennessee Leeuwenburg wrote:
>
> > For anyone who's interested, I am now quite happily serving large 
> > data sets via HTTP to thredds. The key was to re-write my java 
> > servlet in python. I think object creation overhead was killing me 
> > because of the sheer number of independant requests made by thredds. 
> > I used a default window size of 65,536 bytes, and was able to get 
> > 1.1Mb/s throughput downloading from thredds. My test file was 590Mb.
> >
> > I like this approach, because my custom data source is neatly 
> > de-coupled from thredds code. All I have to do is deal with the HTTP 
> > protocol, and everything is handled for me.
> >
> > If anyone else is having difficulty integrating a new data source 
> > into thredds, feel free to contact me for my code.
> >
> > Thanks for everyone's help...
> >
> > Cheers,
> > -Tennessee
>
>
> Are you sure the key was moving to python, and not moving to a 65K 
> buffer?


That could well be the case, as I did both at once. Java is a lovely 
language, however I can also see how it could suffer in the face of 
repeated object creation when handling many small requests. In this 
case, however, the baby and the bathwater can't be easily differentiated :)

I'm going to stick with the Python code on the grounds that it works, 
and I've got more urgent things to do right now, but it could be that 
Java could do the same job.

Cheers,
-T