20030714: ldmadmin scour ignore option? (cont.)

>From: Jared P Bostic <address@hidden>
>Organization:  Oklahoma Climatological Survey Operations Center
>Keywords:  200307142148.h6ELm7Ld016870 LDM scour

Jared,

re:
>> Yes.  You configure scouring by setting directories and file name
>> masks in the ~ldm/etc/scour.conf file.  To not scour a directory,
>> simply don't include it in your scour.conf.

>Thanks - that's what we're doing now.  Guess I wasn't really clear
>(sorry!).  My situation is the following...
>
>Say for example one was saving incoming files in the following way:
>
>/mnt/ldm/data/YYYY/mm/dd/YYYYmmdd.txt
>
>where YYYY = 4 digit year, mm = 2 digit month, dd = 2 digit day
>
>Ok, say I only want to keep /mnt/ldm/data from filling up, yet
>don't want to delete files from certain directories,
>eg. 1999/05/03 (wouldn't have any idea why I picked that date :-P
>Obviously, if I just put /mnt/local/data in the scour.conf, the
>pattern would get applied to everything in there, and the case
>study days would get nuked.

I don't think that the case study subdirectory would get nuked.
'ldmadmin scour' runs ~ldm/bin/scour.  The operative part of the
scouring logic is to get a list of files in a directory and delete
those.  It does not do recursive deletes into subdirectories.

>I suppose one could write a cron
>script to create a new scour.conf file each day with that day's
>YYYY/mm/dd in it, to add that to the end of the root data dir
>on the appropriate line in the file.

OK, now I see that you want to be able to scour current directories
that don't necessarily exist at any particular point in time (like when
you setup a scour.conf file).  'scour' will not work the way you want
it to, so you will have to take a different approach (scour is pretty
simple in design).

>Alternatively, one could
>have a script that added certain (protected) days' patterns to
>a "NOT" condition in the regular expression (maybe?).

Yes.  You could take ~ldm/bin/scour and modify it to add the notion of
directories to not scour.

>My basic
>question is: does anyone have experience with this sort of issue
>and is there a solution (other than what I've mentioned) for this?

For specialized scouring, we wrote new scouring scripts which we
run from cron.  Although none of the ones we wrote for things like
image scouring quite does what you are after, you are more than
willing to take a look to see if there is something you can adapt
to your purposes:

Anonymous FTP::

machine:    ftp.unidata.ucar.edu
user:       anonymous
pass:       address@hidden
directory:  pub/ldm/scour
files:      prune_nexrcomp.csh, prune_nexrcomp.csh, scour_bynumber.csh,
            prune_gini.csh

>Thanks Again!

Sorry we don't have a ready made solution for you...

>Jared Bostic
>OCS/Oklahoma Mesonet

Tom

>From address@hidden Mon Jul 14 19:25:41 2003

Thanks!  I'll have a look at those specialty versions of
scour.  As you said, those could probably be adapted to
do what I need done.

Thanks Again,

Jared

>From address@hidden Tue Jul 15 06:42:58 2003

Jared,

I've run into similar problems here too and have usually written a custom
script to handle it.  As far as I know, I don't think it deletes
directories either which is a problem with dated directories which
accumulate over time and then bog down the "find" command which it uses
for deletion.  I've had cases on a busy system where so many nested
directories have accumulated (because I forgot to remove them manually)
that one scour doesn't finish before the next one begins and it eventually
brings the system down.  I've had fantasies of making some improvements to
the scour script but (like all of us) haven't yet found the time.

                                 Art.