20030114: NLDN ldm-mcidas decoders and McIDAS-XCD setup at USF (cont.)

>From: "Happel, Shelly" <address@hidden>
>Organization: USF
>Keywords: 200301091811.h09IBGt26938 ldm-mcidas nldn2md

Hi Shelly,

>Thanks tom!! I ended that vi session.

OK, thanks.

re: having both a FILE and PIPE action for the same pqact.conf entry

>Yes, I made those changes just yesterday on the NLDN pqact.conf file.

That was bad, but it wasn't the only problem.

>I wasn't finding any of the MD files using the PIPE command, so I placed the
>FILE command in there and it started dumping some data into /home/ldm

This was a good thought, but you should have removed the line with the
PIPE action.

>- what
>the HECK are those files? Are they uncoded lightning data?

If you had a correct FILE action for NLDN products, then yes, you should
see the undecoded lightning data.

>I'm so confused
>about where the LDM puts things (my head is sore from beating it against the
>wall!)

Don't hit it too hard :-)

>Thanks Tom!!

Here goes (I made a LOT of changes/additions):

o One problem was that the ldm-mcidas decoders that you were using were
  all bombing out.  I took a look at the size of the decoder binaries
  in ~ldm/decoders, and compared them with the ones in the binary
  ldm-mcidas release for v2002b, and I noticed that yours were way too
  small to be the correct ones.  So, I downloaded the correct set
  of ldm-mcidas binaries (for Linux 2.4) into the ~ldm/ldm-mcidas
  directory and unpacked the compressed tar file.  I then copied
  the decoders that you needed into the ~ldm/decoders directory.
  This took care of one problem.

o Another problem was that multiple actions in ~ldm/pqact.conf
  were incomplete.  You should review the corrections I made
  to all of these to get the flavor of what should be done.

o Another problem was the read/write permissions on the directories
  /var/data/ldm and /var/data/ldm/mcidas and on the file
  /var/data/ldm/mcidas/ROUTE.SYS.  The file ROUTE.SYS was owned by
  'root', so it could not be written by anyone else.  I corrected
  this by making the owner 'mcidas' in group 'mcidas'.

While poking around, I noted that the environment for your 'mcidas'
user was incorrect (or, at least did not follow my recommendations).
I made a mod in ~mcidas/.cshrc to:

1) set the McINST_ROOT environment variable to $MCHOME
2) set MCDATA to be $MCHOME/workdata
3) changed MCTABLE_WRITE to match recommendations

The changing of MCDATA changes MCPATH since MCPATH is defined using
the value of MCDATA as the first directory in its list.  The value
of MCTABLE_WRITE affects what file will be updated with McIDAS
DATALOC information.  The file that will be modified is
/home/mcidas/data/ADDESITE.TXT.  I found that this file was owned
by 'root', so I changed it to be owned by 'mcidas' in group 'mcidas'.

I also noticed that while you had built McIDAS-XCD, you had not installed
it.  I finished this step (which depended on the correct definition
of McINST_ROOT):

<as 'mcidas'>
cd mcidas2002/src
make install.xcdall VENDOR=-g77

After finishing the XCD installation, I decided to do the setup necessary
to allow XCD data decoding to be done (if you choose to do so):

<as 'mcidas'>
cd data
cp EXAMPLE.NAM LOCAL.NAM
<edit LOCAL.NAM and change directory locations to match your setup>
cd ~/workdata
redirect.k REST LOCAL.NAM

te.k XCDDATA \"/var/data/ldm/mcidas

batch.k XCD.BAT
batch.k XCDDEC.BAT

This created several files in /var/data/ldm/mcidas that will be used
by McIDAS-XCD data monitors/decoders if/when you decided to start
decoding data into McIDAS format.  More on this at the end of this
email.

Once ROUTE.SYS was owned by 'mcidas', not 'root', and once a McIDAS
REDIRECTion was setup so that McIDAS knew how to reference the
copy of ROUTE.SYS in /var/data/ldm/mcidas, I could look at the contents
of that copy of ROUTE.SYS.   What I saw made me think that the file
was very old since it was missing several entries.  I decided to update
it using the template DROUTE.BAT contained in the McIDAS distribution:

<as 'mcidas'>
cd ~mcidas/data
cp DROUTE.BAT USFROUTE.BAT>
<look through USFROUTE.BAT and make sure that there were not changes I
 wanted to make>
cd ~/mcidas/workdata
batch.k USFROUTE.BAT
route.k REL C                   <- enable creating GOES-East/West composites
route.k REL N                   <- enable creating Mollweide/imagery composites

I then decided to setup ADDE access to the data you are now ingesting:

cd ~mcidas/data
cp DSSERVE.BAT LSSERVE.BAT
<edit LSSERVE.BAT and modify to match USF setup>
cd ~mcidas/workdata
batch.k LSSERVE.BAT

Nex up was to setup the ADDE remote server on metlab:

<as 'root'>
cd /home/mcidas
sh mcinet2002.sh install mcadde

<as 'mcidas'>
create the file ~mcidas/.mcenv

Then,

cd ~mcidas/workdata
dataloc.k ADD CIMSS metlab.cas.usf.edu
dataloc.k ADD RTIMAGES metlab.cas.usf.edu

Now when users access images from the CIMSS or RTIMAGES dataset, they
will be doing so from metlab.cas.usf.edu.  I verified that I can
point to your remote ADDE server and access these datasets from
my machine here in Boulder.

Continuing on with XCD setup:

<as 'ldm'>
cd ~ldm/decoders
cp ~mcidas/mcidas2002/src/xcd_run .      <- to run XCD data monitors/decoders
cp ~mcidas/mcidas2002/data/mcscour.sh .  <- to scour McIDAS data files

<edit xcd_run and mcscour.sh; no changes were needed in xcd_run; I
changed the number of XCD-produced data files to keep in mcscour.sh>

<add crontab entries to: run mcscour.sh, rotate the ldm-mcidas.log files,
rotate the ADDE server access log files (these do not exist yet)>

o add a line to ~ldm/etc/ldmd.conf to run the XCD data monitors:

  exec  "xcd_run MONITOR"

o add lines to ~ldm/etc/pqact.conf to send text and binary data to
  XCD data ingesters:

  IDS|DDPLUS    ".*"    PIPE
        xcd_run DDS
  HRS   ".*"    PIPE
        xcd_run HRS


The next time the LDM is stopped and restarted the XCD data decoding
will start.  If you don't want this, you should comment out the new
lines in ~ldm/etc/ldmd.conf and ~ldm/etc/pqact.conf.

A couple of observations:

1) ingestion and decoding of Unidata-Wisconsin imagery data is now working.
   The output AREA files can be found in /var/data/ldm/mcidas.

2) no NLDN data has been decoded since there has been none to decode so
   far.  As soon as an NLDN data file is received that has some flash
   data in it, an MDXX file will be created for it/them in /var/data/ldm/mcidas.
   The MDXX files for NLDN lightning will lie in the name space
   MDXX0071 - MDXX0080, inclusive.

3) XCD decoding of observational data will start the next time the LDM
   is stopped and restarted.  GRID decoding will not start until the
   user 'mcidas' enables it.  This is done as follows:

   <login as 'mcidas'>
   cd ~mcidas/workdata
   decinfo.k SET DMGRID ACTIVE

   Since I setup mcscour.sh to run from an 'ldm' cron entry, your disk
   won't fill from the XCD-produced data.

4) you are requesting FNEXRAD data from FSU, but they are refusing
   to send it to you.  You need to contact the IDD admin at FSU and
   get him/her to allow your request for FNEXRAD data.  I have
   setup scouring for FNEXRAD composite data in the cron initiated
   C shell script, ~ldm/util/prune_nexrcomp.csh.  You should take
   a look at this to get an idea of how it works and is configurable.

   Also, you are requesting ALL NEXRAD data from FSU, but they also
   refusing this request.  All of the NEXRAD data is a LOT of data.
   You should consider exactly what you want to ingest before you ask
   them to allow your feed request.  I have, however, setup scouring
   for the NEXRAD Level III data (in ~ldm/util/prune_nexrad.csh), so
   the disk usage won't go nonlinear.

5) ADDE serving of FNEXRAD and NEXRAD data on metlab may have to be
   adjusted once you start getting the FNEXRAD and NNEXRAD data.

Tom

>From address@hidden Wed Jan 15 07:46:51 2003
>Subject: RE: 20030114: thank you!!!
 
Hey Tom! 

Thank you SO much. It's going to take me some time to go through what
you've written here. It's all so complicated and I'm far from qualified
to be at the helm, but I'm trying!!! I feel like kneeling before you in
gratitude. I knew the system was incorrectly functioning but I couldn't
sleuth the solution and my unix pals here are even less familiar than I
am with LDM and mcidas. Thank you thank you thank you thank you!!!

You may be hearing more from me when I get to fully delve into the
system!!

Shelly