[IDD #VVO-154186]: NOAA LDM MADIS

Hi Alex,

re:
> Thanks so much, I am now at the point of testing.

Very good.

The first thing I would do is:

- make sure that the LDM is running on your "upstream" machine (the machine
  that will send products to the machine(s) that will REQUEST data)

- from the "downstream" machine (the machine that will do the REQUEST) run
  the LDM utility 'notifyme' directed at the upstream machine

  For instance:

  <as 'ldm' on the downstream machine>
  notifyme -vl- -f ANY -h ldm02.airdat.com -o 3600

  OR, if there is no forward DNS (name -> IP) for ldm02.airdat.com:

  notifyme -vl- -f ANY -h 172.16.1.132 -o 3600

  assuming, of course, that the IP address of ldm02.airdat.com is 172.16.1.132.

  What are the results?

  A successful 'notifyme' REQUEST will result in a line that looks like:

Oct 13 18:13:48 notifyme[11529] NOTE: NOTIFYME(ldm02.airdat.com): OK

  If the ALLOW on the upstream machine is incorrect, and if firewall(s)
  are configured to allow inbound traffic on port 388 on your upstream
  machine, then you would get a denying message.  If the firewall(s)
  do not allow inbound traffic on the upstream machine, you will likely
  get nothing.

- when you have success making 'notifyme' requests from the downstream
  to the upstream, you will undoubtedly move on to configuring your
  LDM on the downstream to REQUEST the data that you want from the
  upstream

re:
> Our biggest thing is we tend to over
> think how this works since we are all linux engineers here.

:-)

re:
> I have a box called ldm02 which will be serving data allow    EXP     
> (ldm99.airdat.com|10.4.0.215)

NB: it is best practice to escape '.'s (periods) in fully qualified
hostnames/IP addresses.  For example:

allow   EXP   ^ldm99\.airdat\.com$

This will help make sure that you are allowing the machine(s) that you
actually want to allow.

re:
> I also have a box called ldm99 with request   ANY     "test([0-9]).txt" 
> (ldm02.airdat.com|172.16.1.132)

OK.  In your scenario, ldm99.airdat.com is your upstream and ldm02.airdat.com is
your downstream.

re:
> I am trying to grab any text(0-9).txt wild card from ldm02 with ldm99 to test 
> it.

Note that the extended regular expression you are using in your example REQUEST 
line
does not match your comment about what you want to get:

request ANY "test([0-9]).txt"

"trying to grab any text(0-9).txt"

Saying "text(0-9).txt" instead of "test(0-9).txt" may be just a simple typo.

re:
> My big issue is the pqact.conf on figuring that out.

???

re:
> I may have something wrong so far I just need ldm99 to grab the files I put 
> in the data
> dir for ldm02.

I do not understand what you mean.

The way that the data distribution process works is:

- product(s) are inserted into the LDM queue on the upstream machine

- downstream machine(s) are configured to REQUEST some/all of the
  products that are inserted into the LDM queue on the upstream machine

- the LDM configuration file on the downstream machine (~ldm/etc/ldmd.conf)
  has an EXEC line that runs the LDM utility 'pqact' telling it to use
  actions in a (un)specified pattern-action file

  If the pattern action file to use is not specified on the EXEC "pqact...
  line, then the default is assumed, ~ldm/etc/pqact.conf.

- if it is so desired (a machine running an LDM may simply be a relay for
  data -- it REQUESTs products from one or more upstreams and provides
  feeds to zero or more downstreams), the pattern-action file is setup
  to do something with products that are received

  The most common thing that a pattern-action file action will do is
  FILE the products that are received.

re:  
> Can you help me shine any light onto this?

Assuming that you have products on your upstream machine named test0.txt,
test1.txt, test2.txt, ... test9.txt that you want to feed to your
downstream, and assuming that there is a process on the upstream that
inserts the products to be fed into the upstream's LDM queue, then
the operative configuration lines could look like:

upstream machine, ldm02.airdat.com ~ldm/etc/ldmd.conf file:

ALLOW EXP ^ldm99\.airdat\.com$

downstream machine, ldm99.airdat.com ~ldm/etc/ldmd.conf file:

EXEC "pqact"
 ...
REQUEST EXP "test[0-9].txt" ldm02.airdat.com

downstream machine's ~ldm/etc/pqact.conf file:

EXP<tab>(*)
<tab>FILE -close<tab>\1

Assuming that the user running the LDM on the downstream machine
has write permission in the ~ldm/var/data directory, AND assuming
that the Product ID assigned to the file that was inserted in the
LDM queue on the upstream is simply the file's name (sans directory
information), then the test[0-9].txt files will be written to ~ldm/var/data.

Your pattern-action file actions can specify exactly where to write
output.  For instance, if you have a /data/ldm directory structure
on your downstream machine, and if you want to write the files you
receive into that directory, your pattern-action file action might
look like:

EXP<tab>(*)
<tab>FILE -close<tab>/data/ldm/\1

Now, we should "talk" about the process of inserting products into
an LDM queue:

The LDM utility 'pqinsert' is used to insert products (e.g., files)
into the LDM queue on an upstream machine.  'pqinsert' allows
the user to construct/specify what a product's Product ID will
be.  If you setup your LDM account correctly/fully, then you will
be able to do a 'man pqinsert' to see the man page for 'pqinsert'.

NB: 'pqinsert' allows the user to specify a product's Product ID
and the datastream that the product will be associated with.

Let's assume that you have files in the /data/madis directory
that you want to send to one or more downstreams, and that the
files in the /data/madis directory have names like test[0-9].txt.
Let's further assume that you want to send the files in the EXP
datastream, and you want the Product IDs to be the base names of
the files.  Your 'pqinsert' invocation run by the user running the
LDM (typically 'ldm') on the upstream may look like:

pqinsert -vl- -f EXP -p test0.txt /data/madis/test0.txt
            ^     ^         ^         ^
            |     |         |         |___ the file to insert into the LDM queue
            |     |         |_____________ the Product ID to assign to the 
product
            |     |_______________________ the feed to send the product in
            |_____________________________ send log information to STDOUT


Before going further, I have to ask if the above is addressing what you were
asking in your email...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: VVO-154186
Department: Support IDD
Priority: Normal
Status: Closed