Re: buffer size and performance

Hi Jeff:

BTW, the default buffer size in Structure.Iterator is 500K, which gives better 
performance (I think) when the structure is actually a "psuedo-structure", that 
is, stored by column rather than row.

I could try modifying the default buffer size of RAF, that might improve 
performance on the server.

The trick is to have a wide range of queries to optimize over, and not just get 
one case right at the expense of the whole.

Do you want to put together an interesting "load" on the server, maybe based on 
a bunch of IDV tests or something?



Jeff McWhirter wrote:
> What experience do you all have with the performance characteristics of 
> data access
> with varying bufferesizes. I have found with the point obs that it 
> varies dramatically for remote
> data sets. e.g, getting a data iterator with BUFFERSIZE using:
>
> PointObsDataset.getDataIterator(BUFFERSIZE)
>
> and reading this data set (~90000 obs,32 values)  
> http://lead4.unidata.ucar.edu:8080/thredds/dodsC/station/metar/20060716_metar.nc 
>
> I get:
> buffer size:4096 Total time:41882
> buffer size:8192 Total time:30348
> buffer size:16384 Total time:25124
> buffer size:32768 Total time:23919
> buffer size:65536 Total time:26789
> buffer size:131072 Total time:27540
>
> It seems as though 32768 is a sweet spot. Any ideas why the higher 
> buffer sizes give worse performance?
> Are you allocating the buffers repeatedly? (thus triggering worse GC 
> behavior?)
>
> On the other hand reading a local file:
> /upc/share/testdata/station/madis/20060615_1200
> which has 23000 obs and 180 vars I get the exact opposite:
> buffer size:4096 Total time:8557
> buffer size:8192 Total time:8534
> buffer size:16384 Total time:8307
> buffer size:32768 Total time:11004
> buffer size:65536 Total time:9247
> buffer size:131072 Total time:9010
> buffer size:262144 Total time:9032
>
>
>
> -Jeff
>
>
> =============================================================================== 
>
> To unsubscribe thredds-dev, visit:
> http://www.unidata.ucar.edu/mailing-list-delete-form.html
> ===============================================================================