20020923: data scouring at STC

>From: "Anderson, Alan C. " <address@hidden>
>Organization: St. Cloud State
>Keywords: 200209232129.g8NLTo103996 McIDAS-X mcscour

Hi Alan,

Long time no hear!

>Hi, to all but most likely it will be Tom

Ready.

>Our system runs very well here, so well that I have forgotten where to go
>to adjust the amount of data that our system saves.

McIDAS data files are scoured by a cron entry for either the user 'ldm'
or 'mcidas'.  The cron entry runs the Bourne shell script 'mcscour.sh'.
What you should do to figure out which user is running the McIDAS scour
script is login as each user in turn, and list out the crontab entries.
The one that has an invocation for mcscour.sh is the one you need to
consider.

In the crontab listing, you should see the location of the mcscour script.
The likely locations for the file are:

~ldm/decoders
~ldm/util
~mcidas/workdata
~mcidas/bin

I don't recally which directory is located in on your machine; sorry.

After you find the copy of mcscour.sh that is being run, all you need
to do is edit it and change the number of days that the various products
will be kept.  The block of code in mcscour.sh will look something like:

MCPATH=$MCPATH PATH=$PATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH mcenv << EOF

qrtmdg.k GRID 5001 6300 1
doqtl.k  1  70 2
doqtl.k 71  80 2
doqtl.k 81  90 2
doqtl.k 91 100 2
delwxt.k 1 10
igu.k DEL 132
lwu.k DEL VIRT9001
lwu.k DEL VIRT9002
lwu.k DEL ROUTEPP.LOG
exit

EOF

In this example listing, the routine 'qrtmdg.k' is being used for scouring
XCD-produced GRID files, and 'doqtl.k' is used for scouring MD files.
The last number on each invocation line is the number of days to keep
on disk at any time.

If you are decoding GRID files, you should change the range of GRID file
numbers that qrtmdg.k acts over.  The starting file number, 5001, is OK,
but the ending one is not.  Change whatever you have to 6400.

>Found the scour.conf file in the ldm, which is run by cron, but this
>does not seem to point
>at the right files as the data directory names are outdated.

Right.  Do not use the LDM scour mechanism for scouring McIDAS data
files.  Keep using a cron initiated mcscour.sh instead.

>Looked at your web archive for LDM and did a search on scouring data
>files, nothing came up.

You would have found information on mcscour.sh in the McIDAS Support
area.

>What I want to change is the no. of days of  MD  and AREA  files that
>are saved on our ingest machine which runs the ldm.

The number of MD files that are saved is adjusted by editing mcscour.sh
as I discuss above.  Adjusting the number of AREA files you keep, however,
can be quite a bit more difficult depending on how the images are being
saved to disk.  Is the imagery to be used for McIDAS or is it to be
used by McIDAS and/or GEMPAK?  How you save the files to disk and
how those files are scoured will depend on the intended use.  Before
plowing into explanations for this, I need to get an idea of what your
intended use is first.

>We do not run MCIDAS
>on it, although mcidas is a user, and an older version of MCIDAS is
>installed.

OK.  But is McIDAS being run from other machines which read data files
from this machine?

>Since MCIDAS is not running, I assume its Scheduler is not running.

That is correct.

>So, how about a nudge (or kick) in the right direction?

The MD (and GRID) scouring is a snap.  Changing the number of AREA files
will require that you either muck around with your McIDAS routing table
or adjust a separate script that scours images saved into a directory
hierarchy needed by GEMPAK.  Once I understand your use, I can help
you to do the correct thing.

Tom