19990922: McIDAS-XCD installation/configuration at UNCA

>From: Jim Heimbach <address@hidden>
>Organization: UNCA
>Keywords: 199909141652.KAA04088 McIDAS

Jim,

>     I finally got some time to work on McIDAS-XCD and remain 
>puzzled, even after your long response.  Knowing how Murphy's
>laws work it is probably something very simple.  If you have the
>time could you nose around and see if there is anything obvious?

I logged onto your machine and set about getting XCD working.  The
main problem was that you had not installed XCD.  You built it, but
did not install it.  The broken pipe messages were the result of
the LDM trying to run programs that did not exist.

Since you asked me to login, I decided to perform what I considered to
be necessary cleanup in your 'mcidas' account.

Let me give you a blow-by-blow of what I did:

<login as 'ldm'>

o stopped the LDM

o added the ~ldm/decoders directory to 'ldm's PATH in .cshrc;
  then I sourced the modified .cshrc.

o copied files batch.k from the /home/ldm-mcid/bin directory to ~ldm/decoders
  this is useful since you need to edit batch.k to set McIDAS environment
  variables to match your system setup.  Since batch.k is included in each
  ldm-mcidas distribution, any changes that you make to the file will
  be lost the next time you install ldm-mcidas.  The best solution is
  to simply put it in a directory in the 'ldm's PATH.

o edited ~ldm/decoders/batch.k and ~ldm/decoders/xcd_run to make sure that
  the McIDAS environment variables were correct.

o edited ~ldm/etc/pqact.conf and toned down the verbosity of reporting
  for the ldm-mcidas decoder lwtoa3 (from '-v -x' to '-v').  I also
  added entries for processing of FSL2 (wind profiler data).  You
  are not requesting this data, but I figured that the entry would
  be there to decode it if you ever did request it

o edited ~ldm/etc/ldmd.conf.  I changed the explicit naming of the
  location of xcd_run to let it be selected by virtue of 'ldm's PATH.
  The entry is now:

  exec   "xcd_run MONITOR"

  The copy of xcd_run that will be used is the one in ~ldm/decoders

<login as 'mcidas'>

o since I found that XCD was not installed, I installed it:

  cd ~mcidas/mcidas7.6/src
  make install.mcxall

o went to the ~mcidas/workdata directory to check out what was there.
  I found that you had copied what appeared to be all of the files
  from ~mcidas/data to ~mcidas/workdata.  Since this is a waste of
  disk space and will make upgrading McIDAS harder in the future, I
  removed all of the files that were not needed.

o since XCD had not been installed when you went through the XCD configuration
  step, none of the configuration parameters were setup.  I had to redo
  redirect.k LIST              (to see if the REDIRECTions had been setup)
  dataloc.k LIST               (to see if data locations had been setup)
  the XCD configuration:

  cd ~mcidas/workdata
  batch.k XCD.BAT
  batch.k XCDDEC.BAT

o at this point, I decided to investigate the contents of the output
  data directory, /data/mcidas/data.  There I found yet another copy of
  what appeared to be all files from the ~mcidas/data directory.  Again,
  since this will make it harder to upgrade McIDAS in the future, I
  removed all unnecessary files.  I took care to backup files that I
  could not definitely ascertain were not modified.  I did this by
  copying them to ~mcidas/back.  The /data/mcidas/data directory is now
  the repository for McIDAS data files produced by ldm-mcidas (lwtoa3
  and nldn2md right now) and XCD decoders.  This is the way an installation
  should look.

<as the user 'ldm'>

o rotate the LDM log files and start the LDM:

  ldmadmin newlog
  ldmadmin start

After the above, XCD came running correctly.  A quick look at processes
that are running as 'ldm' shows:

vortex% ps -u ldm
   PID TTY      TIME CMD
 20129 ?        0:00 dmraob.k
 20075 ?        0:00 startxcd
 20083 ?        0:00 rpc.ldmd
 20077 ?        0:00 pqact
 20074 ?        0:01 pqexpire
 20130 ?        0:00 dmsyn.k
 20081 ?        0:01 rpc.ldmd
 20076 ?        0:00 pqbinsta
 20078 ?        0:00 pqing
 20080 ?        0:00 rpc.ldmd
 20084 ?        0:00 ingetext
  8451 pts/3    0:00 csh
 20123 ?        0:00 startxcd
 20128 ?        0:01 dmsfc.k
 20125 ?        0:00 ingetext
 20124 ?        0:00 ingebin.
 20131 ?        0:00 dmmisc.k
 20086 ?        0:00 ingebin.
 20073 ?        0:00 rpc.ldmd

The programs startxcd.k, dmraob.k, dmsyn.k, dmsfc.k, dmmisc.k,
ingetext.k, and ingebin.k are McIDAS-XCD routines.  ingebin.k and
ingetext.k are the ingestors.  The are run by virtue of the pqact.conf
entries:

DDPLUS|IDS      ^.*     PIPE  <- starts ingetext.k to read all textual products
        xcd_run DDS
HRS     ^.*     PIPE          <- starts ingebin.k to read all binary products
        xcd_run HRS

startxcd.k is the XCD supervisory routine.  Its job is to start and
keep running the XCD data monitor routines: dmraob.k, dmsyn.k, dmsfc.k,
dmmisc.k, HRS model data; you get your textual data from a satellite
feed (true?)).  Since a number of McIDAS data files need to be scoured
(e.g. MDXX files; XCD-produced GRID files; and XCD-produced text
files), I setup scouring through a crontab entry for the user
'mcidas'.  This entry runs the scouring shell script,
~mcidas/workdata/mcscour.sh and keeps 3 days of MD files, etc.  Since
you don't seem to have that much disk space available in /data, you
will need to keep an eye on disk usage to make sure that you don't run
out of space.  If you do, you will need to cut down the number of days
of MD files that you keep online.  You set this by editing
~mcidas/workdata/mcscour.sh and changing number at the end of the

One of the last things that I did was take a look at your McIDAS routing
table:

cd ~mcidas/workdata
route.k LIST

I noted that:

o you were not creating GOES-East/West composite imagery
o you did not have an entry for the Mollweide global water vapor imagery

I enabled the east-west compositing by running:

route.k REL C

I added the appropriate entry for the Mollweide H2O product by running:

route.k ADD UY AREA 110 119 CC=3 SYS=2152 2153 2154 \"Mollweide Composite H2O

As I get ready to send you this email, I note that the new routing table
entries are having their desired effects:

S Pd         Description         Range       Last      Received  Post Process C
- -- ------------------------- --------- ------------ ---------- ------------ -
  CI GOES-8/9 IR Composite       80-89   AREA0080     99265 2238     none     3
  CV GOES-8/9 VIS Composite      90-99   AREA0090     99265 2239     none     3
  CW GOES-8/9 H2O Composite      70-79   AREA0070     99265 2236     none     3

 ...

  UY Mollweide Composite H2O    110-119  AREA0110     99265 2236     none     3


The last thing I would ask you is whether or not you are still receiving
NIDS data from WSI?  I see lots of AREA files in the NIDS AREA ranges in
ROUTE.K, but I see that they appeared to have stopped last December.  If
you are no longer receiving NIDS data, then I would strongly recommend
that you remove the old AREA files to free up some space on /data:

cd /data/mcidas/data
rm AREA0[3456789]*

Finally, I decided to check on the setup of the McIDAS ADDE remote server.
I noted a couple of things that I changed:

o the entry in /etc/passwd for the user 'mcadde' needed to have its HOME
  directory changed to be /home/mcidas.  Its default shell needed
  to be changed from /bin/csh to /bin/false (no logins permitted). These
  entries would make no difference if you are using NIS+, however.

o the mcserv and mccompress entries in /etc/inetd.conf needed to be
  changed since the HOME directory for the user 'mcadde' was changed
  (to match my installation instructions)

o as I prepared to send a HUP to inetd, I noted that you had two copies
  of inetd running!  This is a big no-no.  I killed the one that appeared
  to be the most dormant.  I then sent the other one a HUP so it would
  reread /etc/inetd.conf

o I tested out your remote ADDE server by pointing to it from my
  machine here at Unidata:

  DATALOC ADD RTIMAGES cyclone.atms.unca.edu
  DATALOC ADD RTPTSRC cyclone.atms.unca.edu

  And running some tests:

  DSINFO IMAGE RTIMAGES           (worked fine)
  IMGIDSP RTIMAGES/GE-IR LATLON=32 82 MAG=-2 EU=IMAGE REFRESH='EG;MAP H;BAR 
SU=|IRTEMP'
  SFCPLOT T OLAY FRAME

  These commands all worked correctly proving that your system is ingesting
  and decoding data correctly by both ldm-mcidas (lwtoa3 and nldn2md) and
  XCD (dmsfc.k which makes the surface MD files) decoders.

Again, you need to keep an eye on disk space to make sure that you don't
fill up /data.  As I leave, you have about 600 MB of space left, but that
can disappear in a hurry.

Tom

>From address@hidden  Thu Sep 23 06:44:13 1999

     Many thanks for giving things a good looking over and lending 
a guiding hand.  I will be digesting your e-mail and discussing this
with the University guru.  This is not the sort of thing that
someone should be doing helter-skelter on an hour-here-and-hour-there
basis as I am doing.  The department has started a new search and
hopefully a greater time commitment for UNIDATA will be possible.
     I see by the last command that you spent 3:49 working at this.
I owe you an apology for fouling things up badly enough
that you lost a half man day over this.
      - Jim H