20000606: running McIDAS-X ADDE commands vi cron

>From: Angel Li <address@hidden>
>Organization: RSMAS/University of Miami
>Keywords: 200006061956.e56JuLT23087 McIDAS-X ADDE cron mcbatch.sh mcrun.sh

Angel,

Long time no hear...

>I need to set up a procedure that will automatically get data from
>an ADDE server at Wisconsin. Is this problem solved by writing a
>script with DATALOC, IMGLIST, etc commands?

Yes.  It is best done by setting up the DATALOC entry(ies) in a McIDAS-X
session as the user that will eventually run the cron job.  This will
insure that the DATALOC values get written into the table that will be
read by commands when they go looking for where they should get the
data they need.  The shell scripts I include at the end of this email
will then use these definitions to locate the desired data.

>Can cron run this job?

Yes.  I have included examples below.

>I am going thru the McIDAS learning guide as quickly as possible
>trying to get up to speed on the "new" way.

I understand.  After reading this message, you might want to skim
through the Unidata McIDAS-X online documentation for setting up the
'mcidas' and other user accounts.  In there, I discuss running McIDAS
data file scouring from cron, and I include an example file,
mcscour.sh, in the McIDAS-X distribution.

Since I have gotten essentially the same question more than once
lately, I have decided to include two new Bourne shell scripts in my
next McIDAS-X release, Version 7.70 (which I hope to have out at the
end of July/beginning of August).  I have included those Bourne shell
scripts at the end of this email.

>Also, could you send me your standard pqact.conf? I need to start
>accessing buoy data with ADDE commands and not sure where the data should
>be stored.

The pqact.conf entries for McIDAS-XCD decoding of data are extremely
simple, and are presented in:

Building, Installing, and Configuring McIDAS-XCD
http://www.unidata.ucar.edu/packages/mcidas/mcx/mcidas-xcd.html
  McIDAS-XCD Software Installation 
  http://www.unidata.ucar.edu/packages/mcidas/mcx/xcd_install.html
    Installation procedures
    http://www.unidata.ucar.edu/packages/mcidas/mcx/xcd_install_procs.htm
      Running with the Unidata LDM 
      http://www.unidata.ucar.edu/packages/mcidas/mcx/xcd_start.html

In the final link above, you will see that the pqact.conf entry for
McIDAS-XCD decoding is:

# Entries for XCD decoders
DDPLUS|IDS      ^.*     PIPE
        xcd_run DDS
HRS     ^.*     PIPE
        xcd_run HRS

'xcd_run' is a Bourne shell script included in the Unidata McIDAS-X
distribution;  it gets installed in the ~mcidas/bin directory.

You will also see on that same page that one starts the McIDAS-XCD
supervisory routines at LDM startup; this is in ldmd.conf:

exec    "pqexpire"
exec    "xcd_run MONITOR"
exec    "pqact"
exec    "pqbinstats"
exec    "pqsurf"

The 'exec  "xcd_run MONITOR"' invocation is what does the job.

>Thanks and hope everyone is doing well.

Everyone is plodding along as usual.  I hope things are going well at
RSMAS also.

Below, I have included two Bourne shell scripts: mcbatch.sh and mcrun.sh.

mcbatch.sh can be used to run a McIDAS BATCH file outside of a McIDAS-X
session.  I put this together for those sites that are comfortable writing
McIDAS BATCH scripts and want to be able to run them from cron.

The invocation syntax for mcbatch.sh is:

chmod +x mcbatch.sh
mcbatch.sh McIDAS_BATCH_FILE_NAME

Example:

mcbatch.sh PROFILER.BAT

A cron entry might look like:

00 21 * * * /home/mcidas/workdata/mcbatch.sh PROFILER.BAT

The second shell script, mcrun.sh, is virtually identical to mcbatch.sh
except that it is designed to be copied to a file of the user's choice
and edited.

The invocation syntax for mcrun.sh is:

chmod +x mcrun.sh
mcrun.sh

A cron entry might look like:

00 21 * * * /home/mcidas/workdata/mcrun.sh

The crux of both of these shell scripts is the setting of several
environment variables that define things that McIDAS commands will need
to run outside of a McIDAS-X session.

Since I just jammed together these shell scripts, they have may be a minor
bug or two; please forgive me and send me in questions and comments.

--------------------------- mcbatch.sh ---------------------------------------
#!/bin/sh -f

#--------------------------------------------------------------------------
#
# Name:     mcbatch.sh
#
# Purpose:  Bourne shell script used to run McIDAS BATCH files outside
#           of a McIDAS-X session.
#
# Syntax:   mcbatch.sh MCBATCHFILE.BAT
#
# Notes:    Environment variables that need to be set for McIDAS commands
#           to run outside of a McIDAS-X session:
#
#             MCHOME - the home directory for the user running mcbatch.sh
#
#             MCDATA - the user's McIDAS-X working directory.  NOTE: this
#                      directory should also be the first directory
#                      specified in MCPATH.
#
#             MCPATH - colon separated list of directories that contain
#                      McIDAS data files, ancillary data files, and help
#                      files
#
#             PATH -   a colon separated list of directories that
#                      Unix will search when looking for executables.
#                      NOTE: the ~mcidas/bin directory should be first in
#                      the list of directories that will be searched.
#
#             LD_LIBRARY_PATH - the search path for sharable libraries;
#                      this should be the same search path as the one
#                      used by the McIDAS session
#
#             MCBATCHFILE.BAT - the name of the McIDAS BATCH file that
#                      you want to run.
#
#           This is a Bourne shell script.  All statements must conform to
#           Bourne shell scripting syntax.
#
#           The example PATH and LD_LIBRARY_PATH definitions are set
#           for Sun Solaris.  AIX and HP-UX users need to change
#           LD_LIBRARY_PATH everywhere in this file to what is appropriate
#           on their systems
#
#             AIX:  LD_LIBRARY_PATH -> LIBPATH
#             HPUX: LD_LIBRARY_PATH -> SHLIB_PATH
#
#
# History:  20000607 - Written for Unidata McIDAS-X, -XCD 7.70
#
#--------------------------------------------------------------------------

# Define macros needed for McIDAS-7.X environment
#
# MCHOME - set this to the HOME directory of the user McIDAS was installed as
# 
# MCDATA - this will be set according to who is running the script:
# 
#          user       MCDATA
#          ----------+---------------------
#          mcidas     $HOME/workdata
#          other      $HOME/mcidas/data
#
# MCPATH - this is a colon-separated list of directories McIDAS will search
#          when looking for anciallary data (e.g., map databases, enhancements,
#          stretch tables, etc.) and data (e.g., AREA, MDXX, GRID, and TEXT
#          files.
#
#          The $MCDATA directory should _always_ be the first MCPATH directory!
#
# MCLOG  - the name of the file to log output to; defaulted to
#          $MCDATA/mcbatch.log
#
# MCTABLE_READ - a quoted, semi-colon-separated list of directories to search
#          for McIDAS-X DATALOCation information.  This is typically
#          MCTABLE_READ="$MCDATA/MCTABLE.TXT;$MCHOME/data/ADDESITE.TXT"
#          but can be expanded to suit the user's needs.
#
# PATH   - Unix shell search PATH with $MCHOME/bin being the first directory.
#
# LD_LIBRARY_PATH - most likely not needed, but put in for future use
#

#
# First, define MCHOME as the HOME directory for the user under which
# McIDAS-X is installed.
#

MCHOME=/home/mcidas

#
# Find out who is trying to run the script.  Don't allow 'root'!
#

case `id` in
'uid=0('*)
        echo "ERROR: cannot execute as user 'root'"
        exit 1
        ;;
uid=?'(mcidas)'*        |\
uid=??'(mcidas)'*       |\
uid=???'(mcidas)'*      |\
uid=????'(mcidas)'*     |\
uid=?????'(mcidas)'*    |\
uid=??????'(mcidas)'*   |\
uid=???????'(mcidas)'*  )
        MCDATA=$HOME/workdata
        ;;
*)
        MCDATA=$HOME/mcidas/data
        ;;
esac

#
# Set MCPATH using MCDATA and MCHOME
#

MCPATH=${MCDATA}:$MCHOME/data:$MCHOME/help

#
# Define MCLOG to be $MCDATA/mcbatch.log
#

MCLOG=$MCDATA/mcbatch.log

#
# Set MCTABLE_READ based on the existence of $MCDATA/MCTABLE.TXT.  Users
# should modify this section to match their setup if it does not follow
# the recommendations in the Unidata McIDAS-X online documentation
# for configuring user accounts.
#

if [ -f "${MCDATA}/MCTABLE.TXT" ]; then
  MCTABLE_READ="${MCDATA}/MCTABLE.TXT;${MCHOME}/data/ADDESITE.TXT"
else
  MCTABLE_READ="${MCHOME}/data/ADDESITE.TXT"
fi

#
# Setup PATH so that the McIDAS-X executables can be found
#

PATH=$MCHOME/bin:$PATH

#
# Set LD_LIBRARY_PATH to include all directories (other than those searched
# by default) that are needed to be searched to find shared libraries.
#

LD_LIBRARY_PATH=$MCHOME/lib:$LD_LIBRARY_PATH

#
# Send all textual output to the log file
#

exec 2>$MCLOG 1>&2

#
# Export the environment variables specified above
#

export MCHOME MCDATA MCLOG MCPATH PATH LD_LIBRARY_PATH

#
# Now run 'mcenv' to create a McIDAS-X environment in which the McIDAS
# binary batch.k will run.
#

cd $MCDATA
mcenv << EOF

  batch.k $1
  exit

EOF

# Done
exit 0
--------------------------- mcbatch.sh ---------------------------------------



--------------------------- mcrun.sh -----------------------------------------

#!/bin/sh -f

#--------------------------------------------------------------------------
#
# Name:     mcrun.sh
#
# Purpose:  Bourne shell script used to run a sequence of McIDAS commands
#           outside of a McIDAS-X session.
#
# Syntax:   mcrun.sh
#
# Notes:    Environment variables that need to be set for McIDAS commands
#           to run outside of a McIDAS-X session:
#
#             MCHOME - the home directory for the user running mcrun.sh
#
#             MCDATA - the user's McIDAS-X working directory.  NOTE: this
#                      directory should also be the first directory
#                      specified in MCPATH.
#
#             MCPATH - colon separated list of directories that contain
#                      McIDAS data files, ancillary data files, and help
#                      files
#
#             PATH -   a colon separated list of directories that
#                      Unix will search when looking for executables.
#                      NOTE: the ~mcidas/bin directory should be first in
#                      the list of directories that will be searched.
#
#             LD_LIBRARY_PATH - the search path for sharable libraries;
#                      this should be the same search path as the one
#                      used by the McIDAS session
#
#             MCBATCHFILE.BAT - the name of the McIDAS BATCH file that
#                      you want to run.
#
#           This is a Bourne shell script.  All statements must conform to
#           Bourne shell scripting syntax.
#
#           The example PATH and LD_LIBRARY_PATH definitions are set
#           for Sun Solaris.  AIX and HP-UX users need to change
#           LD_LIBRARY_PATH everywhere in this file to what is appropriate
#           on their systems
#
#             AIX:  LD_LIBRARY_PATH -> LIBPATH
#             HPUX: LD_LIBRARY_PATH -> SHLIB_PATH
#
#
# History:  20000607 - Written for Unidata McIDAS-X, -XCD 7.70
#
#--------------------------------------------------------------------------

# Define macros needed for McIDAS-7.X environment
#
# MCHOME - set this to the HOME directory of the user McIDAS was installed as
# 
# MCDATA - this will be set according to who is running the script:
# 
#          user       MCDATA
#          ----------+---------------------
#          mcidas     $HOME/workdata
#          other      $HOME/mcidas/data
#
# MCPATH - this is a colon-separated list of directories McIDAS will search
#          when looking for anciallary data (e.g., map databases, enhancements,
#          stretch tables, etc.) and data (e.g., AREA, MDXX, GRID, and TEXT
#          files.
#
#          The $MCDATA directory should _always_ be the first MCPATH directory!
#
# MCLOG  - the name of the file to log output to; defaulted to
#          $MCDATA/mcrun.log
#
# MCTABLE_READ - a quoted, semi-colon-separated list of directories to search
#          for McIDAS-X DATALOCation information.  This is typically
#          MCTABLE_READ="$MCDATA/MCTABLE.TXT;$MCHOME/data/ADDESITE.TXT"
#          but can be expanded to suit the user's needs.
#
# PATH   - Unix shell search PATH with $MCHOME/bin being the first directory.
#
# LD_LIBRARY_PATH - most likely not needed, but put in for future use
#

#
# First, define MCHOME as the HOME directory for the user under which
# McIDAS-X is installed.
#

MCHOME=/home/mcidas

#
# Find out who is trying to run the script.  Don't allow 'root'!
#

case `id` in
'uid=0('*)
        echo "ERROR: cannot execute as user 'root'"
        exit 1
        ;;
uid=?'(mcidas)'*        |\
uid=??'(mcidas)'*       |\
uid=???'(mcidas)'*      |\
uid=????'(mcidas)'*     |\
uid=?????'(mcidas)'*    |\
uid=??????'(mcidas)'*   |\
uid=???????'(mcidas)'*  )
        MCDATA=$HOME/workdata
        ;;
*)
        MCDATA=$HOME/mcidas/data
        ;;
esac

#
# Set MCPATH using MCDATA and MCHOME
#

MCPATH=${MCDATA}:$MCHOME/data:$MCHOME/help

#
# Define MCLOG to be $MCDATA/mcrun.log
#

MCLOG=$MCDATA/mcrun.log

#
# Set MCTABLE_READ based on the existence of $MCDATA/MCTABLE.TXT.  Users
# should modify this section to match their setup if it does not follow
# the recommendations in the Unidata McIDAS-X online documentation
# for configuring user accounts.
#

if [ -f "${MCDATA}/MCTABLE.TXT" ]; then
  MCTABLE_READ="${MCDATA}/MCTABLE.TXT;${MCHOME}/data/ADDESITE.TXT"
else
  MCTABLE_READ="${MCHOME}/data/ADDESITE.TXT"
fi

#
# Setup PATH so that the McIDAS-X executables can be found
#

PATH=$MCHOME/bin:$PATH

#
# Set LD_LIBRARY_PATH to include all directories (other than those searched
# by default) that are needed to be searched to find shared libraries.
#

LD_LIBRARY_PATH=$MCHOME/lib:$LD_LIBRARY_PATH

#
# Send all textual output to the log file
#

exec 2>$MCLOG 1>&2

#
# Export the environment variables specified above
#

export MCHOME MCDATA MCLOG MCPATH PATH LD_LIBRARY_PATH

#
# Now run 'mcenv' to create a McIDAS-X environment in which the McIDAS
# binary commands will run.
#
# It is the user's responsibility to put the commands they want to run
# in the following.
#

cd $MCDATA
mcenv << EOF

# put McIDAS-X commands you want to run here, one command per line.

# Example (note that these lines are commented out!!):
#
# dataloc.k ADD RTGINI adde.unidata.ucar.edu
# imgdisp.k RTGINI/GE1KVIS STA=KMIA EU=IMAGE SF=YES REFRESH='EG;MAP H'
# frmsave.k 1 miamivis.gif

# done
  exit

EOF

# Done
exit 0
--------------------------- mcrun.sh -----------------------------------------

Tom Yoksas