Re: TDS Questions

Hi Estanislao :

On 5/11/2011 2:25 PM, Estanislao Gonzalez wrote:
> Hi John,
>
> as we've already spoken I'll try to summarize here the questions:
> 1) To minimize the memory footprint of the TDS, is it possible to 
> limit the cache amount used for storing catalogs in memory? Is that 
> configurable somehow?
not currently but i will add asap.

> 2) Is it possible to move the catalog to a DB? directly (read from DB) 
> or indirectly (read from Catalog but saved into DB for intermediate 
> caching)?

we dont want the overhead of administering a DB, and it wouldnt add any 
functionality in this case i think, do you?.

> 3) I'll have to describe the problem here a little as I can't think 
> about the proper question:
>
> I've developed a stager for staging files served by the TDS that are 
> on tape, or actually in any remote repo 
> (http://esgf.org/esgf-stager-site/ - this is part of esgf, BSD; I 
> promise to write a proper documentation but the code is pretty much 
> well documented, go to the repo link).
> The stager sits on a tomcat filter that is configured to catch some 
> urls and map them into the expected location of the file in the file 
> system. Using the user-agent attribute in the http head its decided if 
> the stager goes into interactive (user is redirected to a waiting 
> page) or non-interactive mode (call blocks, in the worst case client 
> will time out, but a new call doesn't trigger the retrieval again).
> For this to work the stager recreates the complete directory structure 
> as stored on the remote server (e.g. tape library), in this way, for 
> example, the OpenDAP will find the file where it should be.
> The problem happens when the file is buried into a directory structure 
> that wasn't there before. If only the file is retrieved, i.e. the 
> directory structure is already in the cache area, the application 
> works perfectly. However if a directory structure is being generated 
> the TDS answers with a 404. Restarting the TDS shows the file is there 
> and can be accessed properly.
>
> You were right and the modification timestamp of the directory is 
> being updated *iff* something in it gets changed. *However* this is 
> not the case for sub-directories.
>
> So, if I have:
> root/dir1
>
> and the stager downloads dir2/file1 to root/dir1 
> (root/dir1/dir2/file1) the file stamp of the root doesn't change, only 
> that of dir1 does. So this might be the problem.
>
> .. I'm missing the question so: is that so? :-)

if the directory.lastModified() date doesnt change when a subdirectory 
changes, then my caching assumptions are wrong and i need to fix it. 
what OS have you tested this behavior on?


> Thanks,
> Estani