19990127: file REDIRECTions disappeared

>From: Clint Rowe <address@hidden>
>Organization: University of Nebraska
>Keywords: 199901272102.OAA24483 McIDAS

Clint,

>I must have screwed up something, but I'll be damned if I can figure out what.
>
>The LDM keeps putting McIDAS data into ~mcidas/workdata instead of /data/XCD
>where I want it.  I can get things started out correctly (I think), but at
>some point it seems to switch over to ~mcidas/workdata.  Any ideas what's
>going wrong?
>
>Thanks for any help you can give me.

For some reason, your file REDIRECTion definitions in LWPATH.NAM are
getting hammered.  I just logged onto your machine and see that the some
total of REDIRECTions is:

MDXX00* /data/XCD

It should be substantially more than that.  It looks like the set of
REDIRECTions that you really want are contained in XCDSER.NAM since
that has redirections that point to /data/XCD.  Guessing that this is
correct, I did the following:

cd ~mcidas/workdata
cp LWPATH.NAM BAD.NAM
cp XCDSER.NAM LWPATH.NAM

I then edited LWPATH.NAM to remove commented out lines (to be tidy) and
made a backup copy of it:

cp LWPATH.NAM LWPATH.BAK

At the same time, it looks like scouring has not taken place in
/data/XCD for awhile (there are files there from 12 days ago!).  So,
it looks like the problem is related to data scouring.

I see that you are running ~mcidas/scripts/mcscour.sh from cron in
the user 'mcidas' account:

# Scour McIDAS files created by ldm-mcidas and XCD
#  at 00:01 UTC (18:01 CDT)
1 18 * * * scripts/mcscour.sh

I decided to take a look at your copy of ~mcidas/scripts/mcscour.sh.
Since your copy of mcscour dates back to 7.10, I decided to update
it (maybe not necessary, but maybe so):

# Scour MD, GRID, and TEXT data files, and the ROUTE PostProcess Log file
MCPATH=$MCPATH PATH=$PATH LD_LIBRARY_PATH=$LD_LIBRARY_PATH mcenv << 'EOF'

qrtmdg.k MD 1 70 3
qrtmdg.k GRID 5001 5400 2
doqtl.k 71  80 3
doqtl.k 81  90 3
doqtl.k 91 100 3
delwxt.k 1 14
igu.k DEL 132
lwu.k DEL VIRT9001
lwu.k DEL VIRT9002
lwu.k DEL ROUTEPP.LOG
batch.k \"BROADCAST.BAT
exit

EOF

Note that:

o one invocation of mcenv creates an environment for each '.k' executable.
  This is more resource efficient.

o I increased the number of days that delwxt.k will go back to delete
  files from 3 to 14.

First, I logged on as the user running your LDM and stopped the LDM.  I
had to do this since the McIDAS environment in which the XCD routines
are running was one in which the proper REDIRECTions had been munged.

I then moved the files in ~mcidas/workdata that should be in /data/XCD
to /data/XCD:

cd ~mcidas/workdata
mv HRS.SPL *.IDT *.IDX *.XCD GRID5* /data/XCD

I then moved the AREA files being created in ~mcidas/workdata to /data/mcidasd:

mv AREA00[6-9]* AREA02* /data/mcidasd

I note that there were some files that belonged to previous McIDAS sessions
in ~mcidas/workdata (ALLOC.WWW, *.001, and SYSIMAGE.SAV) and a core file:

cd ~mcidas/workdata
rm *.001 SYSIMAGE.SAV core

Finally, back as the user running the LDM, I edited
/usr/local/ldm/decoders/bin/xcd_run.  The mod was minor: I added 'exec'
to the starting of the XCD ingestor routines ingebin.k and ingetext.k
so that there would be two less processes running:

case $1 in
        DDS)     exec ingetext.k DDS;;
        IDS)     exec ingetext.k DDS;;
        PPS)     exec ingetext.k DDS;;
        DDPLUS)  exec ingetext.k DDS;;
        HRS)     exec ingebin.k  HRS;;
        HDS)     exec ingebin.k  HRS;;
        GRID)    exec ingebin.k  HRS;;
        MONITOR) rm -f $MCLOG
                 exec startxcd.k ;;
        *)       echo "xcd_run action $1 incorrectly specified, failing..." ;;
esac


I then restarted the LDM:

ldmadmin start

ps -u ldm
   PID TTY      TIME CMD
 10686 ?        0:00 startxcd
 10709 ?        0:28 dchrly
 10685 ?        0:09 pqbinsta
 10734 ?        0:02 ingetext
 10724 ?        0:00 dmraob.k
 10720 ?        0:00 startxcd
 10690 ?        0:25 pqsurf
 10727 ?        0:00 dmmisc.k
 10756          0:00 <defunct>
 10696 ?        0:00 rpc.ldmd
  9586 pts/8    0:01 csh
 10700 ?        0:00 ingetext
 10683 ?        0:00 rpc.ldmd
 10733 ?        0:01 ingebin.
 10725 ?        0:01 dmsyn.k
 10701 ?        0:00 ingebin.
 10721 ?        0:00 dmsfc.k
 10688 ?        0:06 pqact
 10691 ?        0:22 rpc.ldmd
 10684 ?        0:14 pqexpire
 10744 ?        0:01 dcuair
 10755 ?        0:01 dcsynop
 10695 ?        0:07 pqact

While poking around further, I note that you have not gotten any new
NLDN lightning or FSL profiler data in awhile:

mdu.k LIST 1 100
  MD#  CREATED SCHM PROJ  NR   NC     ID   DESCRIPTION
 ----- ------- ---- ---- ---- ---- ------- -----------
     7   99026 ISFC    0   72 4500   99027 SAO/METAR data for   27 JAN 1999
     8   99027 ISFC    0   72 4500   99028 SAO/METAR data for   28 JAN 1999
    17   99026 IRAB    0    8 1300   99027 Mand. Level RAOB for 27 JAN 1999
    18   99027 IRAB    0    8 1300   99028 Mand. Level RAOB for 28 JAN 1999
    19   99028 IRAB    0    8 1300   99029 Mand. Level RAOB for 29 JAN 1999
    27   99026 IRSG    0   16 6000   99027 Sig.  Level RAOB for 27 JAN 1999
    28   99027 IRSG    0   16 6000   99028 Sig.  Level RAOB for 28 JAN 1999
    29   99028 IRSG    0   16 6000   99029 Sig.  Level RAOB for 29 JAN 1999
    37   99027 ISHP    0   24 2000   99027 SHIP/BUOY data for   27 JAN 1999
    38   99028 ISHP    0   24 2000   99028 SHIP/BUOY data for   28 JAN 1999
    47   99027 FO14    0   38  600   99027 NGM MOS for day      27 JAN 1999
    56   99028 SYN     0    8 6000   99026 SYNOPTIC data for    26 JAN 1999
    57   99026 SYN     0    8 6000   99027 SYNOPTIC data for    27 JAN 1999
    58   99027 SYN     0    8 6000   99028 SYNOPTIC data for    28 JAN 1999
    67   99026 PIRP    0   24 1500   99027 PIREP/AIREP data for 27 JAN 1999
    68   99027 PIRP    0   24 1500   99028 PIREP/AIREP data for 28 JAN 1999
    76   98326 NLDN    0 1000 1000   99026 NLDN data for        22 Nov 1998
    77   98327 NLDN    0 1000 1000   99017 NLDN data for        23 Nov 1998
    78   98328 NLDN    0 1000 1000   99018 NLDN data for        24 Nov 1998
    86   98346 WPRO    0   48   35   98346 "WIND PROFILER UPPER AIR DATA
    87   98347 WPRO    0   48   35   98347 "WIND PROFILER UPPER AIR DATA
    88   98348 WPRO    0   48   35   98348 "WIND PROFILER UPPER AIR DATA
 -- END OF LISTING

The dates for NLDN and WPRO go back to December.  Also, it looks like
you are not using the current decoder for the profiler data nor have you
switch to getting the data from the FSL2 feed.  These will be necessary
if you want to look at the profiler data (and I would think that you
would since coverage over NE is good) since the PROFILER.CDF product
will be dropped from the Unidata-Wisconsin datastream by summer.

The question remains as to why your REDIRECTion file was getting munged.
I _think_ that it is related to the scouring of the non-XCD MD files
in /data/mcidasd.  The reason for this is that the sole REDIRECTion
in effect when I logged on was the last thing in BROADCAST.BAT
(which is run from mcscour.sh):

batch.k \"BROADCAST.BAT

If this is the problem (I can't explain it), then you could add the
restoration of the defines in LWPATH.NAM to the end of mcscour.sh:

lwu.k COPY LWPATH.BAK LWPATH.NAM

This is kinda of a sledge hammer, but it might give you peace of mind ;-)

Let me know if things need more fine tuning.

Tom