[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
20031008: McIDAS Installation and configuration at ULM (cont.)
- Subject: 20031008: McIDAS Installation and configuration at ULM (cont.)
- Date: Wed, 08 Oct 2003 19:07:39 -0600
>From: address@hidden
>Organization: ULM
>Keywords: 200310011647.h91Gl8k1011698 McIDAS LDM GEMPAK
Hi Adam,
Here is what I did to setup McIDAS on cyclone:
1) change the location of where data was being written by GEMPAK decoders:
- changed from /data/gempak and /data/ldm/gempak to /data/ldm/gempak
This required numerous changes to the entries in ~ldm/pqact.conf,
but they were essentially changing entries like /data/gempak to
data/gempak. This was made possible by ~ldm/data being a link to
/data/ldm and ~ldm/logs being a link to /data/ldm/logs.
- Chiz setup GEMPAK data scouring to follow his recommendations:
- put the GEMPAK log files in /data/ldm/gempak/logs and
make a link to where you did have them to that director
- added a cron entry to rotate GEMPAK data files:
#
# rotate GEMPAK logs
0 17 * * * util/dcrotatelog.csh >/dev/null 2>&1
2) setup McIDAS-XCD to decode data into /data/ldm/mcidas (this
all follows the installation and configuration instructions
in my.unidata.ucar.edu/content/software/mcidas). All of this
was done as the user 'mcidas' and the order that they were
done is important:
- decided where McIDAS-XCD and ldm-mcidas decoders should write
output data files: /data/ldm/mcidas
- copied ~mcidas/data/EXAMPLE.NAM to ~mcidas/data/LOCAL.NAM and
edited LOCAL.NAM to set the directory where the various McIDAS
data files will be found on cyclone
- copied ~mcidas/data/DSSERVE.BAT to ~mcidas/data/LSSERVE.BAT and
edited LSSERVE.BAT to setup dataset definitions to match were
files are found on cyclone. Create the ADDE info file for
NEXRAD data serving:
cd ~mcidas/workdata
cp NNEXRAD.CFG ULMNEXR.CFG
- edit ULMNEXR.CFG to set the location and name masks to match
where these data are being stored on cyclone,
DIRMASK=/data/gempak/nexrad/NIDS/\ID/\TYPE
FILEMASK=\TYPE_*
- copied ~mcidas/data/DATALOC.BAT to ~mcidas/data/LOCDATA.BAT and
edited LOCDATA.BAT to set the name of the ADDE servers for various
datasets. The datasets that you will host locally (through files
created by McIDAS-XCD, ldm-mcidas, and NEXRAD filing) are identified
to be accessible through the remote ADDE sever that was setup
on cyclone (see below)
- make the file REDIRECTions defined in ~mcidas/data/LOCAL.NAM
active:
cd ~mcidas/workdata
redirect.k REST LOCAL.NAM
- define the McIDAS string XCDDATA to point at the directory chosen
for McIDAS-XCD and ldm-mcidas output:
cd ~mcidas/workdata
te.k XCDDATA \"/data/ldm/mcidas
- copy needed files to the directory that XCD and ldm-mcidas will
write output to:
cp ~mcidas/data/SCHEMA /data/ldm/mcidas
cp ~mcidas/data/SYSKEY.TAB /data/ldm/mcidas
cp ~mcidas/workdata/ROUTE.SYS /data/ldm/mcidas
Make sure that they are readable and writable by 'ldm' which is
in the same group as 'mcidas':
chmod 664 /data/ldm/mcidas/SCHEMA
chmod 664 /data/ldm/mcidas/SYSKEY.TAB
chmod 664 /data/ldm/mcidas/ROUTE.SYS
- run two McIDAS BATCH files that will setup the McIDAS side of
McIDAS-XCD decoding:
batch.k XCD.BAT
batch.k XCDDEC.BAT
- define the McIDAS ADDE datasets that will be accessible on cyclone.
The definitions were setup in a previous step in ~mcidas/data/LSSERVE.BAT:
batch.k LSSERVE.BAT
- setup pointing at datasets that will be hosted on cyclone and those
that are accessible from remote machines. This was previously setup
in ~mcidas/data/LOCDATA.BAT:
batch.k LOCDATA.BAT
At this point, McIDAS-XCD was ready to start decoding data, and the
data files it will create were ready to be served. Also, NEXRAD data
(and FNEXRAD composite radar images, and NLDN lightning data, and FSL
wind profiler data) which will be created over in the LDM configuration
are also ready to be served even though they might not yet exist.
Needed next was configuring the LDM files ~ldm/etc/ldmd.conf and
~ldm/etc/pqact.conf to send data to McIDAS-XCD decoders. This was all
done as the user 'ldm' while the LDM was turned off:
3) copy and configure needed McIDAS utility scripts to the ~ldm/util
directory:
cp ~mcidas/mcidas2003/src/xcd_run ~ldm/util
cp ~mcidas/workdata/mcscour.sh ~ldm/util
Configure entries in xcd_run and mcscour.sh to match the McIDAS
installation, where to write log files, etc.
Make sure that the ~ldm/util directory is in the PATH for the user
'ldm' (edit and source .cshrc)
4) upgrade your ldm-mcidas installation to v2003. I reorganized
your ~ldm/ldm-mcidas directory to be able to quickly change
to a new version of ldm-mcidas:
$ ls -alt ~ldm/ldm-mcidas
total 20
drwxrwx--- 18 ldm ldm 4096 Oct 8 19:50 ..
drwxrwx--- 4 ldm ldm 4096 Oct 4 14:52 .
drwxrwxr-x 6 ldm ldm 4096 Oct 4 14:52 ldm-mcidas-2002b
lrwxrwxrwx 1 ldm ldm 11 Oct 4 14:51 lib -> runtime/lib
lrwxrwxrwx 1 ldm ldm 11 Oct 4 14:51 man -> runtime/man
lrwxrwxrwx 1 ldm ldm 11 Oct 4 14:51 etc -> runtime/etc
lrwxrwxrwx 1 ldm ldm 11 Oct 4 14:51 bin -> runtime/bin
lrwxrwxrwx 1 ldm ldm 15 Oct 4 14:51 runtime ->
ldm-mcidas-2003
drwxrwxr-x 6 ldm ldm 4096 Oct 4 14:50 ldm-mcidas-2003
-rw-rw-r-- 1 ldm ldm 2316 Jun 19 2000 README
5) add an entry to ~ldm/etc/ldmd.conf to run the McIDAS script xcd_run:
~ldm/etc/ldmd.conf:
exec "xcd_run MONITOR"
5) add the needed entries for XCD and ldm-mcidas decoding to
~ldm/etc/pqact.conf:
############################################################################
#
# McIDAS-XCD decoding
#
#########################################################################
DDPLUS .* PIPE
xcd_run DDS
HRS .* PIPE
xcd_run HRS
#########################################################################
#
# FNEXRAD section
#
#########################################################################
#
#### PNG compressed FNEXRAD image (AREA) files
#
# NEXRCOMP 6 km National BREF mosaic
FNEXRAD ^pnga2area Q5 (RL) (.*) (.*) (.*) (.*) (........) (....)
PIPE -close
/home/ldm/ldm-mcidas/bin/pnga2area -vl /home/ldm/logs/ldm-mcidas.log
data/gempak/nexrad/NEXRCOMP/6KN0R-NAT/\4_\6_\7
# NEXRCOMP 10 km National RCM mosaic
FNEXRAD ^pnga2area Q5 (RN) (.*) (.*) (.*) (.*) (........) (....)
PIPE -close
/home/ldm/ldm-mcidas/bin/pnga2area -vl /home/ldm/logs/ldm-mcidas.log
data/gempak/nexrad/NEXRCOMP/10KRCM-NAT/\4_\6_\7
#
#### PNG compressed FNEXRAD image (GINI) files
#
# NEXRAD Level III radar composites in PNG-compressed GINI format
FNEXRAD ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
PIPE -close
/home/ldm/ldm-mcidas/bin/pngg2gini -vl /home/ldm/logs/ldm-mcidas.log
data/gempak/nport/RADAR/\1/\2/\2_\3_\4
#########################################################################
#
# NLDN section
#
#########################################################################
#
#### NLDN lightning product
#
NLDN
^([12][0-9][0-9][0-9]|[0-9][0-9])([0-3][0-9][0-9])([0-2][0-9])([0-5][0-9])([0-5][0-9])
PIPE -close
/home/ldm/ldm-mcidas/bin/nldn2md -vl /home/ldm/logs/ldm-mcidas.log
-d data/mcidas 70 NLDN DIALPROD=LD \1\2 \3\400 DEV=CCN
#########################################################################
#
# FSL2 section
#
#########################################################################
#
#### IDD FSL2 FSL Wind Profiler netCDF file to MD action
#
# Hourly summary
FSL2 ^FSL\.NetCDF\.NOAAnet\.windprofiler\.01hr
PIPE -close
/home/ldm/ldm-mcidas/bin/proftomd -vl /home/ldm/logs/ldm-mcidas.log
-d data/mcidas U2 WPRO 81
# 6-minute
FSL2 ^FSL\.NetCDF\.NOAAnet\.windprofiler\.06min
PIPE -close
/home/ldm/ldm-mcidas/bin/proftomd -vl /home/ldm/logs/ldm-mcidas.log
-d data/mcidas U6 WPR6 91
6) setup cron scouring of files created by McIDAS-XCD and ldm-mcidas:
#
# Rotate ldm-mcidas log files
#
0 * * * * bin/newlog logs/ldm-mcidas.log 4
#
# Scour McIDAS-XCD produced data files
#
00 20 * * * util/mcscour.sh
#
# Rotate ADDE access log files just past 12 midnight on Saturdays
#
1 0 * * 6 bin/newlog logs/SERVER.LOG 3; chmod 666 logs/SERVER.LOG
7) setup cron scouring of NNEXRAD NEXRAD Level III products and FNEXRAD
composite images:
#
# Prune nexrad tree
#
15 * * * * util/prune_nexrad.csh >/dev/null 2>&1
30 * * * * util/prune_nexrcomp.csh /home/ldm/data/gempak/nexrad/NEXRCOMP
>/dev/null 2>&1
45 * * * * util/prune_nexrcomp.csh /home/ldm/data/gempak/nport/RADAR >/dev/null
2>&1
The ~ldm/util/prune_nexrad.csh and ~ldm/util/prune_nexrcomp.csh scripts
had to be edited to setup PATH correctly.
8) stopped and restarted the LDM so that McIDAS-XCD and ldm-mcidas decoding
would begin:
ldmadmin stop && ldmadmin start
Here are some things that we did not look at:
1) decoding into WXP formats
2) scouring of data decoded into WXP formats
Also, Chiz will be logging onto cyclone to run some tests of GEMPAK
routines to make sure that they are working correctly.
I exercised McIDAS access to locally-created data files and to remotely
hosted datasets. All seemed to be working as it should. The one
thing missing is the ability to contact your remote ADDE server from
machines not on the geos.ulm.edu domain. To enable this, your firewall
has to open ports 500, 503, and 112.
After the disk filling incident -- my fault for not finishing off the
scouring stuff fast enough --, cyclone seems to be running well. Scouring
is now working as expected, so you are keeping about 37 GB of /data
free:
$ df -k
Filesystem 1k-blocks Used Available Use% Mounted on
/dev/sda2 16389652 1682772 13874312 11% /
/dev/sda1 101089 23988 71882 26% /boot
/dev/sdc1 70557052 29834672 37138284 45% /data
/dev/sdb1 17496684 1029892 15578000 7% /home
none 515176 0 515176 0% /dev/shm
Please let me know if you have questions on what I listed above, or
if you see something amiss on cyclone.
Cheers,
Tom