20000307: McIDAS-XCD setup and LDM questions at Colgate (cont .)

>From: Adam Burnett <address@hidden>
>Organization: Colgate
>Keywords: 200002291726.KAA26578 McIDAS-X 7.60

Adam,

>I'm sorry to keep bugging you.  I always enter into this software
>installation thing with the goal of not bugging Tom.  My xcd is still not
>working.  If it's not too much trouble, I would like to take you up on your
>offer to log on a look around.

No problem.

>I've been tinkering for a while and may have
>things really out of place or redundant copies of things in many
>directories.

I will clean up things as I go.  I will also try and detail all of  the
things I find and changes I make.

>The important information about my machine is:
>
>location of key files:
>
>xcd_run:       /data2/ldm/decoders
>ldm configuration files:       /data2/ldm/etc
>folder where AREA and lightning MDs are being written: /data3/mcfiles
>folder where I'm trying to write xcd output:   /data3/xcd

OK, here goes.

<login as 'ldm'>
ldmadmin stop     (had to do this twice)

<login as 'mcidas'>
o the copy of SCHEMA in /data3/xcd was a link to /data2/mcfiles; this
  must have been a typo since it should have been a link to /data3/mcfiles.
  I did:

  cp ~mcidas/data/SCHEMA /data3/mcfiles
  cd /data3/xcd
  rm SCHEMA
  ln /data3/mcfiles/SCHEMA .

Basically, the only reason that MDXX files were not being decoded was
the bad link for SCHEMA in /data3/xcd.  That did not, however, stop
me from making further changes ;-).

o The configuration files for the various feeds (HRS.CFG, DDS.CFG,
  PPS.CFG, IDS.CFG) should only live in the ~mcidas/workdata
  directory.  I deleted the copies in /data3/xcd.  I also deleted
  ALLOC.WWW (a remnant from a McIDAS session of the past) and scour.log
  (this file will be written in the ~mcidas/workdata directory)

o after whoever was running a McIDAS session exited (you?), I fixed a
  REDIRECTion typo (changed AREA8* from
  /data2/mcidas/mcidas/data/tutorial to /data2/mcidas/data/tutorial)

o I noticed that you have setup McIDAS data scouring out of the 'mcidas'
  account (done by running mcscour.sh from cron).  I saw that the copy
  of mcscour.sh that was being run was in ~mcidas/data.  I looked at
  this file and made a couple of changes.  In particular, I changed
  MCDATA from /data3/xcd to $MCHOME/workdata.  In looking through your
  setup for 'mcidas' and 'ldm', I found copies of mcscour.sh in several
  places: /data2/mcidas/workdata, /data2/mcidas/data,
  /data2/ldm/decoders.  I recommend deciding on use of one of these:
  the one in /data3/ldm/decoders.  This would mean that you would need
  to setup a cron invocation for 'ldm' like the one that is there for
  'mcidas' and then turn off (remove) the one that is there for
  'mcidas'.  The reason I recommend doing this relates to additional
  scouring that will eventually have to be done in the 'ldm' account.

  I setup cron to run /data2/ldm/decoders/mcscour.sh at the same time
  as the cron entry for 'mcidas' had.  I did this after editing
  data2/ldm/decoders/mcscour.sh and setting the environment variables
  to match your setup.

McIDAS ADDE remote server setup:

o the entry in /etc/passwd for 'mcadde' had its default shell set to
  /bin/csh.  I changed this to /bin/false so 'mcadde' can not login
  (security)

o the contents of ~mcidas/.mcenv had a couple of problems:

  PATH was set with a list of directories that should have been colon
  separated; they were separated by spaces.  Also there was no '='
  sign after PATH.  I changed these.

  There were definitions for CC, FC, etc. in .mcenv AFTER the export
  line.  I removed these definitions as they are not needed.  .mcenv
  is read by the ADDE mcservsh script by the user 'mcadde' when
  an ADDE request is sent to gissun.colgate.edu.  It is not used
  otherwise on your machine since your 'mcidas' user is using the
  C shell (which is fine; don't change this).

o the directory ~mcidas/mcidas had read/write permissions of 700.  In
  order for the ADDE remote server to work, it had to have permissions
  of 775; I changed them.

The ADDE remote server now works fine.  I tested this from home by doing
the following:

DATALOC ADD RTIMAGES GISSUN.COLGATE.EDU
DSINFO IMAGE RTIMAGES

        Dataset Names of Type: IMAGE in Group: RTIMAGES

Name         NumPos   Content
------------ ------   --------------------------------------
ANTARCTIC       10    Antarctic IR Composite
EDFLOATER-I     10    Educational Floater
EDFLOATER-II    10    Educational Floater II
GE-IR           10    GOES-East North America IR
GE-IRTOPO       10    GOES-East IR/TOPO Composite
GE-VIS          10    GOES-East North America VIS
GE-VISTOPO      10    GOES-East VIS/TOPO Composite
GE-WV           10    GOES-East North America H2O
GEW-IR          10    GOES-East/West IR Composite
GEW-IRTOPO      10    GOES-East/West IR/TOPO Composite
GEW-VIS         10    GOES-East/West VIS Composite
GEW-VISTOPO     10    GOES-East/West VIS/TOPO Composite
GEW-WV          10    GOES-East/West H2O Composite
GW-IR           10    GOES-West Western US IR
GW-IRTOPO       10    GOES-West IR/TOPO Composite
GW-VIS          10    GOES-West Western US VIS
GW-VISTOPO      10    GOES-West VIS/TOPO Composite
GW-WV           10    GOES-West Western US H2O
MDR             10    Manually Digitized Radar
MDRTOPO         10    MDR/TOPO Composite
MOLL-IR         10    Mollweide Composite IR
MOLL-IRTOPO     10    Mollweide IR/TOPO Composite
MOLL-WV         10    Mollweide Composite H2O
RESFLOATER      10    Research Floater

DSINFO -- done

I also verified that I could load an image from your server:

IMGDISP RTIMAGES/GW-IR STA=KDEN MAG=2 REFRESH='EG;MAP H'
Beginning Image Data transfer, bytes= 81140
IMGDISP: loaded frame  1
EG;MAP H
IMGDISP: done
Erased graphic frame(s) 1-1
MAP: completed frame  1

<login as 'ldm'>

o as mentioned above, I setup cron to run /data2/ldm/decoders/mcscour.sh

o I edited /data2/ldm/decoders/xcd_run and made sure that environment
  variables were setup as they should be (some mods; please review)

o I noticed that you were running the ldm-mcidas decoders from the
  /data2/ldm/ldm-mcidas-7.6.2/bin directory.  I feel that it is best to
  copy the needed decoders to an LDM directory where they will not get
  overwritten by a new install of ldm-mcidas.  So, I copied batch.k,
  lwtoa3, nids2area, nldn2md, and proftomd from the ldm-mcidas bin
  directory to the /data2/ldm/decoders directory.  batch.k is a Bourne
  shell script that is used in McIDAS ROUTE PostProcess BATCH files
  (like compositing imagery).  I edited batch.k and set environment
  variables to match your setup (should sound very familiar by now :-)
  so that you can run ROUTE PP BATCH invocations.

o I removed the /data2/ldm/ldm-mcidas-7.6.2/bin directory from the .cshrc
  setting of path.  I removed the /data2/ldm/ldm-mcidas-7.6.2/lib
  directory from the setting for LD_LIBRARY_PATH.  I commented out the
  definition of HOME (this is done for you by the shell).

o after sourceing .cshrc, I started the LDM with:

  ldmadmin start

  As this hung (ldmadmin did not exit after things were started as you
  noted before), I backgrounded the process and let things come up
  fully.  After all decoders were running, I killed the ldmadmin process
  (but no others!).  We are looking into this here at the UPC.

<back as 'mcidas'>

I noted that MDXX files are now being decoded as expected.  I exercised
this a bit remotely by doing the following from my home machine:

DATALOC ADD RTPTSRC GISSUN.COLGATE.EDU
SFCPLOT T USA

I got the surface plot of temperatures over the continental US as expected,
so the MDXX decoding is apparently working correctly.

Now, on to other things.  I see that you are converting NIDS files to
McIDAS AREAs using nids2area.  This appears to be working fine, but
(now why is there a but?) you will want to switch over to storing the
files directly and turning off the ldm-mcidas decoding.  The 7.6 Fkey
menu actions for loading/looping NIDS data uses ADDE commands; the
MCGUI still uses the old, non-ADDE ALOOP command.  ALOOP runs DF (old
and going away soon) to load images from AREA files.  This was why
nids2area was needed.  My NIDS ADDE servers, however, can give access
to the McIDAS ADDE image display routine, IMGDISP, directly from the
raw data file.  The raw data files take up LOTs less room than the AREA
file equivalents, so you will be able to keep a lot more images online
with the same disk usage.  The downside is that the MCGUI will then not
be able to load the images anymore, at least until I upgrade MCGUI to
use ADDE commands (coming up).

A middle ground would be to keep decoding the NIDS images into AREA
files; start saving them as raw files; and setting up the ADDE server
to be able to access the raw files.   This would entail:

o an entry in ~ldm/etc/pqact.conf
o sending a HUP to the pqact process telling it to reread pqact.conf
o setting up scouring of the raw NIDS files (if the radars are operating
  in storm mode, you will be getting 12 images an hour for each of the
  20 NIDS products, or 240 products per hour times 24 hours per day).
  We typically keep about 60 of each type of NIDS raw image on line
  at any time.  This allows for a quite nice loop of images.

If you are interested in moving to the ADDE way of accessing NIDS data,
I can setup things for you.  I will probably not be able to jump on this
immediately, however, but let me know anyway.

>Thanks Again - let me know what I can do from this end

Please let me know if you see any strange behaviors on your system.
There may be something left to do with your setup (I don't think that
there is but...).

Tom

>From address@hidden  Tue Mar  7 10:43:20 2000

>WOW!

>Looks like you spent half your day with my machine.  THANKS.

>I was going to ask about the mcadde account.  I didn't know how to set it up
>with no login.  Also, I was reading about the use of ADDE to access NIDS.  I
>was going to wait until I had everything else working before trying that.  I
>guess its time for me to learn the ADDE way of doing things.

>Thanks again

>Adam