20000202: McIDAS problems (cont.)

>From: Michael Voss <address@hidden>
>Organization: SJSU
>Keywords: 200001181849.LAA08980 McIDAS-X ROUTE PP BATCH syslogd crontab LDM

Mike,

I was looking in the LDM setup on vortex for the cause of two things that
had me perplexed:

o /data3/xcd/STRTABLE kept getting recreated by the user running the LDM

o the file /data3/xcd/-l kept getting created (although zero length)

I looked through the McIDAS shell scripts in ~ldm/decoders (batch.k,
mcscour.k, and xcd_run) and think I see the reason for STRTABLE getting
recreated.  ~ldm/decoders/batch.k did not have a 'cd $MCDATA'.  What
is probably going on is the following:

o the NLDN ldm-mcidas decoder, nldn2md, has a -d flag that points to
  /data3/xcd:

NLDN    ^([12][0-9][0-9][0-9]|[0-9][0-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9
])([0-5][0-9])  PIPE
        -close /usr/local/ldm/ldm-mcidas/ldm-mcidas-7.6.1/bin/nldn2md -v
        -d /data3/xcd 70 NLDN DIALPROD=LD \1\2 \3\400 DEV=CCN
        -l /usr/local/ldm/logs/mcidas.log -xv

  this will cause the current working directory to become /data3/xcd when
  the decoder runs.  (There are problems with this entry that I address below.)

  BTW, the current release of ldm-mcidas decoders is 7.6.2.

o the MCPATH in effect when nldn2md runs has as its component diretories
  ~mcidas/workdata, ~mcidas/data, and ~mcidas/help

o the REDIRECTion defined in ~mcidas/workdata/LWPATH for STRTABLE says
  to find/read/write STRTABLE in the current directory (LWPATH.NAM .);
  since the current directory is probably /data3/xcd, this is where
  the ROUTE PostProcess BATCH file, NLDN.BAT, would expect to find/read/write
  STRTABLE.  The lines from NLDN.BAT that do this are:

REM Get the lightning time
RUN "PTABLE MID$("%4",1,2),"NTIME"

  and

REM Clean up string used
:CLEAN
TD NTIME

  Since the McIDAS process is expecting STRTABLE to exist in the CWD,
  and since newer versions of McIDAS can create STRTABLE on-the-fly
  if it doesn't already exist, one is created by the PTABLE invocation
  and updated by the TD invocation.

o adding the 'cd $MCDATA' to ~ldm/decoders/batch.k should make the CWD
  ~mcidas/workdata.  The copy of STRTABLE that lives there will now
  be the one that will get updated by the NLDN.BAT string table manipulation
  lines

After a number of NLDN lightning reports have come in, no STRTABLE file
has been created in /data3/xcd.

I think that the /data3/xcd/-l file is being created by the nldn2md
action in your LDM's pqact.conf file.  The -close, -d,  and -l lines
for this entry (listed above) have spaces at the beginning of the line;
these need to be a tabs.  In addition, the -l line comes _after_ the
McIDAS positional parameters '70 NLDN DIALPROD=LD \1\2 \3\400
DEV=CCN'.  This causes the -l line to be totally ignored in the
invocation.  It _is_ the cause of the '-l' file being written into
/data3/xcd!  I fixed the entry and removed the redundant -v flag since
it is already specified with the -xv setting.

The only reason that these problems were not causing worse problems on
your machine is that this entry is essentially the last one in
pqact.conf.  I corrected this problem and then did the following:

ldmadmin stop
ldmadmin pqactcheck
/usr/local/ldm/etc/pqact.conf is syntactically correct
rm -- /data3/xcd/-l
ldmadmin start

I think that the '-l' problem is now history.

On a different note, while logged in as the user running your LDM, we had
a chance to look at your crontab entries.  What we saw there, we did not
like.  In particular, the technique of:

o getting the last 2000 lines of a log file being written to under by
  syslogd processes and then 'mv'ing the shortened file over the original
  will most likely no do what you think it should/hoping it will

I changed the crontab entries from things like:

1 1 1 * * tail -2000 /usr/local/ldm/logs/dchrly.log > 
/usr/local/ldm/logs/dctemp.log
3 1 1 * * mv /usr/local/ldm/logs/dctemp.log /usr/local/ldm/logs/dchrly.log

to:

1 1 1 * * bin/newlog logs/dchrly.log 2

The LDM process newlog uses the hupsyslog process to tell syslogd to create
a new output log file.  Here is a snippit from the man page for syslogd:

     whenever  it receives a HUP signal (see signal(5), at
     which time it also closes all files it  has  open,  re-reads
     its  configuration  file,  and then opens only the log files
     that are listed in that file.

We think that your system will be more standard with the changes I made.

The last thing I did was to add LDM scouring of the ASUS and FSUS frontal
files that are being written into /data3/mcidas.  As you move to ADDE,
these files will be completely obsolete since the same frontal information
is available through the ADDE TEXT server (which now works on vortex).
Until your transition is complete, however, it is wise to scour these
files (which are used by the non-ADDE FRONT command).  I added the scouring
by adding the line:

/data3/mcidas                   3               *SUS*

to the ~ldm/etc/scour.conf file.  This should leave 3 days of frontal
data on your system.

I think that that about does it.  Please let me know if you run into
anything else that is strange.

Tom