20040408: batch jobs interferring with current .NAM

>From: Owen Cooper <address@hidden>
>Organization: Aeronomy Laboratory/NOAA
>Keywords: 200404081605.i38G5UCT028198 McIDAS scripts REDIRECT MCPATH

Hi Owen,

>I have several MCIDAS .sh jobs running around the clock.  I find that 
>these jobs interfere with my current Mcidas session that is open and  
>running on the screen in front of me.

Hmm...

>The problem is with the REDIRECT command.

I have to admit that I didn't expect you to be changing REDIRECTions in
a cron-initiated, scripted McIDAS job.  Since REDIRECTions are stored
in a file named LWPATH.NAM, and since LWPATH.NAM is typically located
in the ~mcidas/workdata directory for the user 'mcidas' or the
~/mcidas/data directory for other users, AND since the recommended set
of REDIRECTions has a REDIRECTion for LWPATH.NAM as '.' (the current
working directory), you will run into conflicts by changing
REDIRECTions in scripts running at the same time, or scripts running at
the same time that an interactive session is active.  But, you already
know about the conflicts, hence your email.

>Let's assume that I'm sitting 
>in front of Mcidas with my redirection set to PROJECT.NAM.
>Then my .sh job fires off at a fixed time, it makes it's images, but 
>then when it's finished my open Mcidas sesssion is no longer set to 
>PROJECT.NAM.  If I issue the command REDIRECT LIST, it does in fact list 
>the contents of PROJECT.NAM, but Mcidas can't find any of the files that 
>are linked to PROJECT.NAM.  I have to acutally type REDIRECT REST 
>PROJECT.NAM for Mcidas to be able to once again find the file.
>How can I set up my .sh jobs so that they don't interfere with the 
>redirection on my open Mcidas session?

I can think of two ways of accomplishing what you want:

- don't change REDIRECTions in your shell scripts (control access through
  MCPATH only)

- after mcenv is started, figure out the name of the .mctmp subdirectory
  (this will be the last directory in MCPATH), CD to it, and then
  set your REDIRECTions

The first approach is much cleaner _if_ you can use it.

The second approach should require some parsing of the MCPATH environment
variable.  This shouldn't be too hard, but I just did a little test
and don't get the results I expected:

mcenv -i 64 -g 32 -f 800x1100 << EOF
echo $MCPATH
exit

/home/yoksas/mcidas/data:/home/mcidas/aix/data:/home/mcidas/aix/help

I expected this to be something like:

/home/yoksas/mcidas/data:/home/mcidas/aix/data:/home/mcidas/aix/help:/home/yoksas/.mctmp/524296

I am perplexed about this, but I have developed a way around the problem.

>I have attached a sample .sh file.  Note that I issue a REDIRECT command 
>before the batch job is called so that the batch file knows where to 
>find the data.  When the batch file has run I set the redirection to 
>EMPTY.NAM which has no entries.  I do this so that my current session 
>doens't use the .NAM file from the .sh job and start clobbering files in 
>other directories.

I have modified the schell script you sent me with one way of doing
what I think you want.  Please note that this scheme depends on
there being a REDIRECTion in place in your "standard" LWPATH.NAM
file (e.g., ~mcidas/workdata/LWPATH.NAM; or ~user/mcidas/data/LWPATH.NAM)
that reads:

LWPATH.NAM   .

Here is the modified script:

#!/bin/sh -f
#--------------------------------------------------------------------------
#
# Name:     mcbatch.sh
#
# Purpose:  Bourne shell script used to run McIDAS BATCH files outside
#           of a McIDAS-X session.
#
# Syntax:   mcbatch.sh UTC_hour_of_image BATCHfile.BAT
#
# Notes:    Environment variables that need to be set for McIDAS commands
#           to run outside of a McIDAS-X session:
#
#             MCHOME - the home directory for the user running mcbatch.sh
#
#             MCDATA - the user's McIDAS-X working directory.  NOTE: this
#                      directory should also be the first directory
#                      specified in MCPATH.
#
#             MCPATH - colon separated list of directories that contain
#                      McIDAS data files, ancillary data files, and help
#                      files
#
#             PATH -   a colon separated list of directories that
#                      Unix will search when looking for executables.
#                      NOTE: the ~mcidas/bin directory should be first in
#                      the list of directories that will be searched.
#
#             LD_LIBRARY_PATH - the search path for sharable libraries;
#                      this should be the same search path as the one
#                      used by the McIDAS session
#
#             MCBATCHFILE.BAT - the name of the McIDAS BATCH file that
#                      you want to run.
#
#           This is a Bourne shell script.  All statements must conform to
#           Bourne shell scripting syntax.
#
#           The example PATH and LD_LIBRARY_PATH definitions are set
#           for Sun Solaris.  AIX and HP-UX users need to change
#           LD_LIBRARY_PATH everywhere in this file to what is appropriate
#           on their systems
#
#             AIX:  LD_LIBRARY_PATH -> LIBPATH
#             HPUX: LD_LIBRARY_PATH -> SHLIB_PATH
#
#
# History:  20000607 - Written for Unidata McIDAS-X, -XCD 7.70
#           20040409 - Modified to create a temporary directory and use
#                        a copy of LWPATH local to it
#
#--------------------------------------------------------------------------

# Define macros needed for McIDAS-7.X environment
#
# MCHOME - set this to the HOME directory of the user McIDAS was installed as
# 
# MCDATA - this will be set according to who is running the script:
# 
#          user       MCDATA
#          ----------+---------------------
#          mcidas     $HOME/workdata
#          other      $HOME/mcidas/data
#
# MCPATH - this is a colon-separated list of directories McIDAS will search
#          when looking for anciallary data (e.g., map databases, enhancements,
#          stretch tables, etc.) and data (e.g., AREA, MDXX, GRID, and TEXT
#          files.
#
#          The $MCDATA directory should _always_ be the first MCPATH directory!
#
# MCLOG  - the name of the file to log output to; defaulted to
#          $MCDATA/mcbatch.log
#
# MCTABLE_READ - a quoted, semi-colon-separated list of directories to search
#          for McIDAS-X DATALOCation information.  This is typically
#          MCTABLE_READ="$MCDATA/MCTABLE.TXT;$MCHOME/data/ADDESITE.TXT"
#          but can be expanded to suit the user's needs.
#
# PATH   - Unix shell search PATH with $MCHOME/bin being the first directory.
#
# LD_LIBRARY_PATH - most likely not needed, but put in for future use
#

SHELL=sh
export SHELL

#
# First, define MCHOME as the HOME directory for the user under which
# McIDAS-X is installed.
#

MCHOME=/home/mcidas/solaris

#
# Find out who is trying to run the script.  Don't allow 'root'!
#

case `id` in
'uid=0('*)
        echo "ERROR: cannot execute as user 'root'"
        exit 1
        ;;
uid=?'(mcidas)'*        |\
uid=??'(mcidas)'*       |\
uid=???'(mcidas)'*      |\
uid=????'(mcidas)'*     |\
uid=?????'(mcidas)'*    |\
uid=??????'(mcidas)'*   |\
uid=???????'(mcidas)'*  )
        MCDATA=$HOME/workdata
        ;;
*)
        MCDATA=$HOME/mcidas/data
        ;;
esac

#
# Create a temporary directory under $HOME/.mctmp
#

TMPDIR=$HOME/.mctmp/$$
mkdir $TMPDIR

#
# Set MCPATH using TMPDIR, MCDATA, and MCHOME
#

MCPATH=${TMPDIR}:${MCDATA}:$MCHOME/data:$MCHOME/help

#
# Define MCLOG to be $MCDATA/mcbatch.log
#

MCLOG=$MCDATA/mcbatch_NEXRAD_$3$4.log

#
# Set MCTABLE_READ based on the existence of $MCDATA/MCTABLE.TXT.  Users
# should modify this section to match their setup if it does not follow
# the recommendations in the Unidata McIDAS-X online documentation
# for configuring user accounts.
#

if [ -f "${MCDATA}/MCTABLE.TXT" ]; then
  MCTABLE_READ="${MCDATA}/MCTABLE.TXT;${MCHOME}/data/ADDESITE.TXT"
else
  MCTABLE_READ="${MCHOME}/data/ADDESITE.TXT"
fi

#
# Setup PATH so that the McIDAS-X executables can be found
#

PATH=$MCHOME/bin:$PATH

#
# Set LD_LIBRARY_PATH to include all directories (other than those searched
# by default) that are needed to be searched to find shared libraries.
#

LD_LIBRARY_PATH=$MCHOME/lib:$LD_LIBRARY_PATH

#
# Send all textual output to the log file
#

exec 2>$MCLOG 1>&2

#
# Export the environment variables specified above
#

export MCHOME MCDATA MCLOG MCPATH MCTABLE_READ PATH LD_LIBRARY_PATH

#
# Now run 'mcenv' to create a McIDAS-X environment in which the McIDAS
# binary batch.k will run.
#

# Original code example commented out
# mcenv << EOF
#   redirect.k REST ICARTT04.NAM
#   mcenv -f 1@700x800 batch.k $1 $2 $3 $4 $5 CONTINUE=YES
#   redirect.k REST EMPTY.NAM
#   exit
# EOF

cd $TMPDIR

mcenv -f 1@700x800 << EOF

redirect.k REST ICARTT04.NAM
batch.k $1 $2 $3 $4 $5 CONTINUE=YES
exit

EOF

cd $MCDATA
rm -rf $TMPDIR

# Done
exit 0

Other NOTES:

- you do not need to run the second redirect.k since the REDIRECTions that
  were in place for the batch.k run will disappear when the script exits

- you do not need to nest the 'mcenv' invocations

- the script should make sure to remove the temporary directory and its
  contents before it exits.  This should be done carefully so that
  no other directory is deleted!

>Thanks

Sorry for missing the possibility that you might be using REDIRECTs
in your shell scripts.

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.