20040405: LDM ingest, ldm-mcidas decoding, McIDAS access to imagery

>From: "Luis A. Lopez" <address@hidden>
>Organization: UPRM
>Keywords: 200404051637.i35GbNCT003268

Hi Luis,

>I'm getting the statistics log every hour (aprox), does this means that
>ldm is downloading something from unidata compuerts ??.

Yes.  There are a number of ways to see that your LDM is getting fed
from an upstream IDD node:

- review the real time statistics your machine is reporting and verify
  that it contains current information:

http://my.unidata.ucar.edu/content/software/idd/rtstats/siteindex.php?climatemaster.uprm.edu

- use the LDM facility 'ldmadmin watch' to see data product headers
  as they are received

- use the LDM 'notifyme' facility to see what data has been received
  on your system in the past 'n' seconds and then watch as new products
  are received:

<as 'ldm'>

notifyme -vxl- -o 3600              <- show all products received in the
                                       past 3600 seconds

>Im not sure what I need configure in the pqact.conf file, I checked the
>file that you recommend:
>
>http://my.unidata.ucar.edu/content/software/mcidas/mcidd/ldm-mcidas-pqact.conf
> .all
>
>but I'm not sure what to configure.  Thank you again for your help.

Right now, it looks like you are only requesting the Unidata-Wisconsin
datastream images.  At least, that is the only feed that your 'rtstats'
program is reporting statistics for.  Give this, you would incorporate
the MCIDAS feed actions from the ldm-mcidas-pqact.conf.all example file
into your ~ldm/etc/pqact.conf.  When incorporating the actions you MUST
make sure to not simply cut them from ldm-mcidas-pqact.conf.all and
paste them into ~ldm/etc/pqact.conf.  The reason is that the cut and
past operation does not preserve tab characters as whitespace on some
systems, and pqact.conf entries require the use of tabs in certain
places.  The easiest thing to do if you are a 'vi' user, is to edit
~ldm/etc/pqact.conf, go to the bottom, and then read in the contents
of ldm-mcidas-pqact.conf.all.

There are a number of things you need to decide before you finalize
how you want data decoded.  Some of the issues are:

- how much data do you want to keep online

- where do you have enough disk space to keep what you want

- how you want to configure McIDAS to access data

The decisions represent a bit of give and take between the setup of
the LDM and that of McIDAS.  The easiest way to get started, however,
is to do the following:

1) decide where you want McIDAS data to reside.  The typical location
   is the ~ldm/data/mcidas directory.  Also, it is typical for
   ~ldm/data to be a link to a directory on a file system where
   you have plenty of room.  It is atypical for ~ldm/data to be
   a directory on the same file system as ~ldm.  It is _not_
   recommended that the actual location of the data be on an
   NFS mounted file system.

   So, if you have plenty of disk, you might already have a file
   system where you can create a /data directory.  You would then
   make ~ldm/data a link to /data/ldm:

   <as 'ldm'>
   mkdir /data/ldm
   ln -s /data/ldm ~ldm/data

   You must insure that /data/ldm is readable and writable by both
   'ldm' and 'mcidas'.  This is easy enough if you follow
   recommendations and put 'ldm' and 'mcidas' in the same group, and
   set the umask in both accounts to be 002.

2) make the directory in which you want to store McIDAS data.  I
   will continue using the example above:

   <as 'mcidas'>
   mkdir /data/ldm/mcidas

3) the decoder for the UNIWISC (aka MCIDAS) datastream images supports
   two ways of storing data.  One way is rooted in the historic way
   that McIDAS image files are named.  The ldm-mcidas-pqact.conf.all
   action:

MCIDAS  ^pnga2area Q. (..) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        pnga2area -vl /usr/local/ldm/logs/ldm-mcidas.log
        -a etc/SATANNOT
        -b etc/SATBAND
        -d data/mcidas -r \1,\2

   processes images to match this naming/location method.  It requires
   the user 'mcidas' to copy two files from the Unidata McIDAS
   distribution into the directory where the images are to be
   written:  ROUTE.SYS and SYSKEY.TAB:

   <as 'mcidas'>
   cp ~mcidas/data/SYSKEY.TAB ~ldm/data/mcidas
   cp ~mcidas/workdata/ROUTE.SYS ~ldm/data/mcidas
   chmod 664 ~ldm/data/mcidas/SYSKEY.TAB ~ldm/data/mcidas/ROUTE.SYS

   This is the easiest method for getting some decoding working
   for the Unidata-Wisconsin images that you are ingesting.

   This and the other method of decoding the images in the Unidata-Wisconsin
   images requires that you have installed the ldm-mcidas decoders.
   We covered this in a previous email, but suffice it to say that
   you would have at this point copied the ldm-mcidas decoder pnga2area
   to the ~ldm/decoders directory and insured that ~ldm/decoders is
   in the PATH of the user 'ldm'

4) The next step is to setup McIDAS ADDE to be able to use the images
   that should be decoded into the ~ldm/data/mcidas directory.  This
   is covered in the Unidata McIDAS web pages, but I can quickly
   repeat the procedure here:

   <as 'mcidas'>
   cd ~mcidas/data

   cp EXAMPLE.NAM LOCAL.NAM
   - edit LOCAL.NAM and set the directory portion of each line to
     the directory you just setup for Unidata-Wisconsin image decoding.
     Following the example, this would be /data/ldm/mcidas.
   cp DSSERVEOLD.BAT LSSERVE.BAT
   - if you were an experienced McIDAS user, you would know enough to
     be able to modify the entries in the LSSERVE.BAT copy, but let's
     skip that for now

   cd ~mcidas/workdata
   redirect.k REST LOCAL.NAM
   batch.k LSSERVE.BAT

   The last two steps setup information in McIDAS to find the decoded
   image files ('redirect.k REST LOCAL.NAM'), and to create ADDE
   datasets to be able to access those images by dataset group
   name and descriptor.

At this point, if the LDM has been running and ingesting and decoding
images into ~ldm/data/mcidas (which is /data/ldm/mcidas), you should
be able to run the following commands in McIDAS and get output
reflecting the fact that you now have data:

<as 'mcidas'>
cd ~/workdata
dmap.k AREA
DSINFO IMAGE RTIMAGES

Try the above and let me know the results.  After this is working,
we can go through an equivalent procedure that decodes data into
a more meaningful data hierarchy, and then sets up McIDAS ADDE
to be able to use the imagery in that hierarchy.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.

>From address@hidden  Fri Apr  9 14:33:18 2004

Hi Tom:

  Thanks for your answer. I'll let you know later how I'm going with all
your instructions, I need the aprove from the researchers of what data
they are interested on,because we don't have enougth disk space right
now.

Luis