20000628: LDM setup at INM (cont.)

>From: Pepo Juega <address@hidden>
>Organization: Instituto Nacional de Meteorologia
>Keywords: 200006220829.e5M8TZT11974 LDM

Pepo,

>Well, one step further. I added the line to allow ANY from you
>guys, but as I say in the last lines, this whole setup lies behind
>a firewall, so I guess no messing arround with port 388 would be
>permitted.

OK.  The only reason I wanted you to add the allow was for us to be
able to do some remote checking.  If your system(s) is(are) behind
a firewall, and if access to port 388 is not enabled, this will not
be possible.  Also, since things are starting to work, it will not
be necessary.

re: reverse name lookup for meteosat.inm.es and alfa.inm.es

>They have now, I changed the /etc/hosts accordingly and now I can
>ldmping each other.
>---------------------------------------------------------
>meteosat:/export/home/ldm:>ldmping alfa
>Jun 28 07:19:09      State    Elapsed Port   Remote_Host  
>Jun 28 07:19:09 RESPONDING   0.026893  388   alfa
>alfa:/export/home/mcd/ext/ldm:>ldmping meteosat
>Jun 28 07:18:57      State    Elapsed Port   Remote_Host       
>Jun 28 07:18:57 RESPONDING   0.046424  388   meteosat
>---------------------------------------------------------

OK!  If you can do an ldmping, you can do just about anything.

re: insert a product into the first machine's LDM queue (use pqinsert)

>Done. Now a question. Where should the files transferred reside?

The files put into the LDM queue on meteosat can be anywhere.  The
pqinsert will put a copy of the file, with a header into the product
queue so it can be sent.  As for the files on the receiving machine,
you tell pqact where to put them.

>Is this the DATAPATH in ldmadmin.conf?

No, DATAPATH will typically be the ~ldm/data directory on the receiving
machine.

pqact.conf entries allow you to write data to anyplace you like.  A
number of sites choose to write data into the ~ldm/data/...
directories.  Typically ~ldm/data is a link to some other directory
where there is lots of space for ingested/decoded products to be put.

>What I did is:
>
>pqact.conf in both machines
>---------------------------------------------
>MCIDAS  ^(GRID.*)
>        FILE    data/GRIB/%m%d%h%m.grb
>        -l /export/home/ldm/logs/ldmd.log -v
>        -d /export/home/ldm/data
>
># AREA files:
>#
>MCIDAS  ^(AREA.*)
>        FILE    data/AREA/%m%d%h%m.area
>        -l /export/home/ldm/logs/ldmd.log -v
>        -d /export/home/ldm/data
>---------------------------------------------
>ldmadmin pqactcheck
>/export/home/ldm/etc/pqact.conf is syntactically correct

This looks fine (aside from inclusion of -l, -f, and -d which are not
used by FILE) IF the pqinsert run on meteosat was using a file
that is in the same directory as pqinsert is run from.  What I mean
is this:

o if you are in the ~ldm/data directory and you do a:

  pqinsert -vxl- -f MCIDAS GRID0001

  the header sent along with the product will be 'GRID0001'

o if you were in the ~ldm directory and you do a:

  pqinsert -vxl- -f MCIDAS data/GRID0001

  the header sent along with the product will be data/GRID0001

The patterns you list above say that the header of the product 
received have to start with GRID or AREA; if the directory path
is included the patterns will not match the product header.

>Then I fed alfa from meteosat with pqinsert -f MCIDAS data/AREA1525

So, in this case the product header will be 'data/AREA1525', not
AREA1525.  This is why the action is failing.  If, on the other
hand you had done:

cd data
pqinsert -f MCIDAS AREA1525

The pattern ^(AREA.*) would have matched.

>ldmd.log on meteosat:
>-----------------------------------------------------------------------
>Jun 28 08:49:13 meteosat rpc.ldmd[17414]: Starting Up (built: Jun 16
>2000 11:27:
>01)
>Jun 28 08:49:13 meteosat pqact[17416]: Starting Up
>Jun 28 08:49:13 meteosat pqexpire[17415]: Starting Up
>Jun 28 08:49:20 meteosat alfa[17422]: Connection from alfa.inm.es
>Jun 28 08:49:20 meteosat alfa(feed)[17422]: Starting Up:
>20000628074924.990 TS_E
>NDT {{MCIDAS,  ".*"}}
>Jun 28 08:49:20 meteosat alfa(feed)[17422]: topo:  alfa.inm.es MCIDAS
>----------------------------------------------------------------------
>
>and ldmd.log on alfa:
>----------------------------------------------------------------------
>Jun 28 08:49:24 alfa rpc.ldmd[7707]: Starting Up (built: Jun 15 2000
>14:34:03)
>Jun 28 08:49:24 alfa pqexpire[7708]: Starting Up
>Jun 28 08:49:24 alfa pqact[7709]: Starting Up
>Jun 28 08:49:24 alfa meteosat[7710]: run_requester: Starting Up:
>meteosat.inm.es
>Jun 28 08:49:24 alfa meteosat[7710]: run_requester: 20000628074924.990
>TS_ENDT {
>{MCIDAS,  ".*"}}
>Jun 28 08:49:25 alfa meteosat[7710]: FEEDME(meteosat.inm.es): OK
>----------------------------------------------------------------------
>
>It looks fine, but ... where is the file ???

The header didn't match, so it did not get FILEd.  You should have been
able to observe the receipt of the product on alfa by running a 
notifyme:

<on alfa>
notifyme -vxl- -f MCIDAS -o 3600

<on meteosat>
notifyme -vxl- -f MCIDAS -o 3600 -h alfa.inm.es

If you want to keep the directory portion of the file as part of the
product header, you could change your match regular expression in
pqact.conf to:

# GRID files:
#
MCIDAS  ^.*GRID.*
        FILE    -close  data/GRIB/%m%d%h%m.grb

# AREA files:
#
MCIDAS  ^.*AREA.*
        FILE    -close  data/AREA/%m%d%h%m.area

and the product would be filed.  Note that I got rid of the -d, -l, and
-v flags since they are relevant for the FILE action.  These flags are
typically associated with a decoder that is run by the pqact action.

Also note that I included the -close flage on the FILE action.  This tells
the LDM to close the file after it is written.  Without this flag, the
file is kept open.  Without it the next product of the same type might
be appended to the end of the file just written.

>alfa:/export/home/ldm:>ls -laR data
>data:
>total 12712
>drwxrwx---   4 ldm      staff        512 Jun 28 08:16 .
>drwxrwxr-x  14 ldm      staff       1024 Jun 28 08:49 ..
>drwxrwx---   2 ldm      staff        512 Jun 28 08:15 AREA
>drwxrwx---   2 ldm      staff        512 Jun 28 08:15 GRIB
>-rw-rw----   1 ldm      staff    25600000 Jun 28 08:49 ldm.pq
>
>data/AREA:
>total 4
>drwxrwx---   2 ldm      staff        512 Jun 28 08:15 .
>drwxrwx---   4 ldm      staff        512 Jun 28 08:16 ..
>
>data/GRIB:
>total 4
>drwxrwx---   2 ldm      staff        512 Jun 28 08:15 .
>drwxrwx---   4 ldm      staff        512 Jun 28 08:16 ..

There are nothing in the subdirectories since the patterns did not
match.

Once you get a successful transfer and FILE, you should start thinking
about compressing the products before they get distributed (assuming,
of course, that the network bandwidth is limited; if it is not, then
don't worry about compressing the product).

I could imagine that the process on the injection machine would look
like:

<on meteosat>

cd ~ldm
compress data/AREA1515
compress data/GRID1234
pqinsert -f MCIDAS data/AREA1525.Z
pqinsert -f MCIDAS data/GRID1234.Z
uncompress data/AREA1515
uncompress data/GRID1234

The pqact.conf entries on the receiving machines would then be something
like:

# GRID files:
#
MCIDAS  ^.*GRID.*
        PIPE    -close
        uncompressit.sh data/GRIB/%m%d%h%m.grb.Z

# AREA files:
#
MCIDAS  ^.*AREA.*
        PIPE    -close
        uncompressit.sh data/AREA/%m%d%h%m.area.Z

where uncompressit.sh is an executable shell script located in the PATH
of the user running the LDM.  This might look something like:

#!/bin/sh

cat > $1
uncompress $1

The need for the shell script is that you will not be able to redirect
the output from uncompress in pqact.conf; redirection is a shell
capability, not an LDM pqact one.  The 'cat > $1' will write stdin to
the file named by $1; uncompress will then uncompress the file in $1.

If you are going to compress the images, you might be interested in the
PNG compression that we developed for the images in AREA format that we
send through our IDD.  PNG compression works better in almost every
case than compress, gzip, bzip2 and McIDAS delta encoding.  PNG
compressed images are also displayable by applications like ImageMagic
and by web browsers IF the header bytes before the PNG compressed
section are stripped off.  I can give you more information on this as
your tests evolve.

One of the nice things about the PNG compression routine, area2png, and
uncompression routine, pnga2area, is that they can be told to extract
information from the AREA file header and use it in the naming of the
compressed/uncompressed files.  Syntax like you use in your pqact entry
above (e.g., %m%d%h%m) will be satisfied by values from the image
itself:  %H is the hour of the image; %M is the minute of the hour of
the image; etc.  This turns out to be very useful for our GEMPAK users,
and will be very useful to McIDAS-X 7.70 users since SSEC adopted
support of imagery in files not constrained by the AREAnnnn naming
convention.


>Txs,
>Pepo

Tom