20031008: McIDAS Installation and configuration at ULM (cont.)

>From: address@hidden
>Organization: ULM 
>Keywords:  200310011647.h91Gl8k1011698 McIDAS LDM GEMPAK 

Hi Adam,

Here is what I did to setup McIDAS on cyclone:

1) change the location of where data was being written by GEMPAK decoders:

   - changed from /data/gempak and /data/ldm/gempak to /data/ldm/gempak
     This required numerous changes to the entries in ~ldm/pqact.conf,
     but they were essentially changing entries like /data/gempak to 
     data/gempak.  This was made possible by ~ldm/data being a link to 
     /data/ldm and ~ldm/logs being a link to /data/ldm/logs.

   - Chiz setup GEMPAK data scouring to follow his recommendations:

     - put the GEMPAK log files in /data/ldm/gempak/logs and
       make a link to where you did have them to that director

     - added a cron entry to rotate GEMPAK data files:

#
# rotate GEMPAK logs
0 17 * * * util/dcrotatelog.csh >/dev/null 2>&1


2) setup McIDAS-XCD to decode data into /data/ldm/mcidas (this
   all follows the installation and configuration instructions
   in my.unidata.ucar.edu/content/software/mcidas).  All of this
   was done as the user 'mcidas' and the order that they were
   done is important:

   - decided where McIDAS-XCD and ldm-mcidas decoders should write
     output data files: /data/ldm/mcidas

   - copied ~mcidas/data/EXAMPLE.NAM to ~mcidas/data/LOCAL.NAM and
     edited LOCAL.NAM to set the directory where the various McIDAS
     data files will be found on cyclone

   - copied ~mcidas/data/DSSERVE.BAT to ~mcidas/data/LSSERVE.BAT and
     edited LSSERVE.BAT to setup dataset definitions to match were
     files are found on cyclone.  Create the ADDE info file for
     NEXRAD data serving:

     cd ~mcidas/workdata
     cp NNEXRAD.CFG ULMNEXR.CFG

     - edit ULMNEXR.CFG to set the location and name masks to match
       where these data are being stored on cyclone,

DIRMASK=/data/gempak/nexrad/NIDS/\ID/\TYPE
FILEMASK=\TYPE_*

   - copied ~mcidas/data/DATALOC.BAT to ~mcidas/data/LOCDATA.BAT and
     edited LOCDATA.BAT to set the name of the ADDE servers for various
     datasets.  The datasets that you will host locally (through files
     created by McIDAS-XCD, ldm-mcidas, and NEXRAD filing) are identified
     to be accessible through the remote ADDE sever that was setup
     on cyclone (see below)

   - make the file REDIRECTions defined in ~mcidas/data/LOCAL.NAM
     active:

     cd ~mcidas/workdata
     redirect.k REST LOCAL.NAM

   - define the McIDAS string XCDDATA to point at the directory chosen
     for McIDAS-XCD and ldm-mcidas output:

     cd ~mcidas/workdata
     te.k XCDDATA \"/data/ldm/mcidas

   - copy needed files to the directory that XCD and ldm-mcidas will
     write output to:

     cp ~mcidas/data/SCHEMA /data/ldm/mcidas
     cp ~mcidas/data/SYSKEY.TAB /data/ldm/mcidas
     cp ~mcidas/workdata/ROUTE.SYS /data/ldm/mcidas

     Make sure that they are readable and writable by 'ldm' which is
     in the same group as 'mcidas':

     chmod 664 /data/ldm/mcidas/SCHEMA
     chmod 664 /data/ldm/mcidas/SYSKEY.TAB
     chmod 664 /data/ldm/mcidas/ROUTE.SYS

   - run two McIDAS BATCH files that will setup the McIDAS side of
     McIDAS-XCD decoding:

     batch.k XCD.BAT
     batch.k XCDDEC.BAT
     
   - define the McIDAS ADDE datasets that will be accessible on cyclone.
     The definitions were setup in a previous step in ~mcidas/data/LSSERVE.BAT:

     batch.k LSSERVE.BAT

   - setup pointing at datasets that will be hosted on cyclone and those
     that are accessible from remote machines.  This was previously setup
     in ~mcidas/data/LOCDATA.BAT:

     batch.k LOCDATA.BAT


At this point, McIDAS-XCD was ready to start decoding data, and the
data files it will create were ready to be served.  Also, NEXRAD data
(and FNEXRAD composite radar images, and NLDN lightning data, and FSL
wind profiler data) which will be created over in the LDM configuration
are also ready to be served even though they might not yet exist.

Needed next was configuring the LDM files ~ldm/etc/ldmd.conf and
~ldm/etc/pqact.conf to send data to McIDAS-XCD decoders.  This was all
done as the user 'ldm' while the LDM was turned off:

3) copy and configure needed McIDAS utility scripts to the ~ldm/util
   directory:

   cp ~mcidas/mcidas2003/src/xcd_run ~ldm/util
   cp ~mcidas/workdata/mcscour.sh ~ldm/util

   Configure entries in xcd_run and mcscour.sh to match the McIDAS
   installation, where to write log files, etc.

   Make sure that the ~ldm/util directory is in the PATH for the user
   'ldm' (edit and source .cshrc)

4) upgrade your ldm-mcidas installation to v2003.  I reorganized
   your ~ldm/ldm-mcidas directory to be able to quickly change
   to a new version of ldm-mcidas:

$ ls -alt ~ldm/ldm-mcidas
total 20
drwxrwx---   18 ldm      ldm          4096 Oct  8 19:50 ..
drwxrwx---    4 ldm      ldm          4096 Oct  4 14:52 .
drwxrwxr-x    6 ldm      ldm          4096 Oct  4 14:52 ldm-mcidas-2002b
lrwxrwxrwx    1 ldm      ldm            11 Oct  4 14:51 lib -> runtime/lib
lrwxrwxrwx    1 ldm      ldm            11 Oct  4 14:51 man -> runtime/man
lrwxrwxrwx    1 ldm      ldm            11 Oct  4 14:51 etc -> runtime/etc
lrwxrwxrwx    1 ldm      ldm            11 Oct  4 14:51 bin -> runtime/bin
lrwxrwxrwx    1 ldm      ldm            15 Oct  4 14:51 runtime -> 
ldm-mcidas-2003
drwxrwxr-x    6 ldm      ldm          4096 Oct  4 14:50 ldm-mcidas-2003
-rw-rw-r--    1 ldm      ldm          2316 Jun 19  2000 README

5) add an entry to ~ldm/etc/ldmd.conf to run the McIDAS script xcd_run:

   ~ldm/etc/ldmd.conf:

exec    "xcd_run MONITOR"

5) add the needed entries for XCD and ldm-mcidas decoding to
   ~ldm/etc/pqact.conf:

############################################################################
#
# McIDAS-XCD decoding
#
#########################################################################
DDPLUS  .*      PIPE
        xcd_run DDS
HRS     .*      PIPE
        xcd_run HRS

#########################################################################
#
# FNEXRAD section
#
#########################################################################

#
#### PNG compressed FNEXRAD image (AREA) files
#

# NEXRCOMP 6 km National BREF mosaic
FNEXRAD ^pnga2area Q5 (RL) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        /home/ldm/ldm-mcidas/bin/pnga2area -vl /home/ldm/logs/ldm-mcidas.log
        data/gempak/nexrad/NEXRCOMP/6KN0R-NAT/\4_\6_\7

# NEXRCOMP 10 km National RCM mosaic
FNEXRAD ^pnga2area Q5 (RN) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        /home/ldm/ldm-mcidas/bin/pnga2area -vl /home/ldm/logs/ldm-mcidas.log
        data/gempak/nexrad/NEXRCOMP/10KRCM-NAT/\4_\6_\7
#
#### PNG compressed FNEXRAD image (GINI) files
#

# NEXRAD Level III radar composites in PNG-compressed GINI format
FNEXRAD ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
        PIPE    -close
        /home/ldm/ldm-mcidas/bin/pngg2gini -vl /home/ldm/logs/ldm-mcidas.log
        data/gempak/nport/RADAR/\1/\2/\2_\3_\4

#########################################################################
#
# NLDN section
#
#########################################################################

#
####                      NLDN lightning product
#

NLDN    
^([12][0-9][0-9][0-9]|[0-9][0-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
        PIPE    -close 
        /home/ldm/ldm-mcidas/bin/nldn2md -vl /home/ldm/logs/ldm-mcidas.log
        -d data/mcidas 70 NLDN DIALPROD=LD \1\2 \3\400 DEV=CCN

#########################################################################
#
# FSL2 section
#
#########################################################################

#
#### IDD FSL2 FSL Wind Profiler netCDF file to MD action
#

# Hourly summary
FSL2    ^FSL\.NetCDF\.NOAAnet\.windprofiler\.01hr
        PIPE    -close
        /home/ldm/ldm-mcidas/bin/proftomd -vl /home/ldm/logs/ldm-mcidas.log
        -d data/mcidas U2 WPRO 81

# 6-minute
FSL2    ^FSL\.NetCDF\.NOAAnet\.windprofiler\.06min
        PIPE    -close
        /home/ldm/ldm-mcidas/bin/proftomd -vl /home/ldm/logs/ldm-mcidas.log
        -d data/mcidas U6 WPR6 91


6) setup cron scouring of files created by McIDAS-XCD and ldm-mcidas:

#
# Rotate ldm-mcidas log files
#
0 * * * * bin/newlog logs/ldm-mcidas.log 4
#
# Scour McIDAS-XCD produced data files
#
00 20 * * * util/mcscour.sh
#
# Rotate ADDE access log files just past 12 midnight on Saturdays
#
1 0 * * 6 bin/newlog logs/SERVER.LOG 3; chmod 666 logs/SERVER.LOG

7) setup cron scouring of NNEXRAD NEXRAD Level III products and FNEXRAD
   composite images:

#
# Prune nexrad tree
#
15 * * * * util/prune_nexrad.csh >/dev/null 2>&1
30 * * * * util/prune_nexrcomp.csh /home/ldm/data/gempak/nexrad/NEXRCOMP 
>/dev/null 2>&1
45 * * * * util/prune_nexrcomp.csh /home/ldm/data/gempak/nport/RADAR >/dev/null 
2>&1

   The ~ldm/util/prune_nexrad.csh and ~ldm/util/prune_nexrcomp.csh scripts
   had to be edited to setup PATH correctly.

8) stopped and restarted the LDM so that McIDAS-XCD and ldm-mcidas decoding
   would begin:

ldmadmin stop && ldmadmin start


Here are some things that we did not look at:

1) decoding into WXP formats
2) scouring of data decoded into WXP formats

Also, Chiz will be logging onto cyclone to run some tests of GEMPAK
routines to make sure that they are working correctly.

I exercised McIDAS access to locally-created data files and to remotely
hosted datasets.  All seemed to be working as it should.  The one
thing missing is the ability to contact your remote ADDE server from
machines not on the geos.ulm.edu domain.  To enable this, your firewall
has to open ports 500, 503, and 112.

After the disk filling incident -- my fault for not finishing off the
scouring stuff fast enough --, cyclone seems to be running well.  Scouring
is now working as expected, so you are keeping about 37 GB of /data
free:

$ df -k
Filesystem           1k-blocks      Used Available Use% Mounted on
/dev/sda2             16389652   1682772  13874312  11% /
/dev/sda1               101089     23988     71882  26% /boot
/dev/sdc1             70557052  29834672  37138284  45% /data
/dev/sdb1             17496684   1029892  15578000   7% /home
none                    515176         0    515176   0% /dev/shm

Please let me know if you have questions on what I listed above, or
if you see something amiss on cyclone.

Cheers,

Tom