[IDD #NDJ-769600]: ldm question

Hi Lance,

re:
> Okay, so here's the skinny on our LDM setup.  I have a suspicion that it
> is configured... less than optimally... but my predecessors must have
> had their reasons:
> 
> We have a system on Mauna Loa called kenobi.  We have four local systems
> that talk to kenobi: puu, puka, wikiwiki, castor.  All four systems
> collect different data (as you can see from the attached ldm.conf
> files).  The first three collect data for internal use, the fourth we
> just manage on behalf of LASP- they drop in daily via scp and copy what
> they need (I'm not privy to the details of how they do this).  For
> simplicity, I'm going to ignore castor for now.

OK.

re:
> pqact.conf is identical for all systems, so I've attached one copy only.

This should make things simple.

re:
> In each of my ldmd.conf files I have unique "request" lines, which
> obviously makes combining those trivial.

I distilled the operative actions from each of the ldmd.conf files that
you attached to your email.  The setup is pretty simple:

puka-ldmd.conf:

exec    "pqact -v -l /local/d/ldm/logs/pqact.log -q /local/d/ldm/data/ldm.pq 
/local/d/ldm/etc/pqact.conf"
request ANY "^chip/.*" kenobi.mlso.ucar.edu

puu-ldmd.conf:

exec    "pqact -v -l /local/d/ldm/logs/pqact.log -q /local/d/ldm/data/ldm.pq 
/local/d/ldm/etc/pqact.conf"
exec    "pqexpire -a .50"
request ANY "^pchip/.*|^plowl/.*|^pextra/.*|^fchip/.*|^fmk4/.*|^fpics/.*" 
kenobi.mlso.ucar.edu

wikiwiki-ldmd.conf:

exec    "pqact -v -l /local/d/ldm/logs/pqact.log -q /local/d/ldm/data/ldm.pq 
/local/d/ldm/etc/pqact.conf"
request ANY "^pics/.*|^mk4/.*|^misc/.*" kenobi.mlso.ucar.edu

Here are some questions and comments:

- are all the data on kenobi.mlso.ucar.edu that are to be transferred to
  puka, puu, and wikiwiki in the EXP feed type?

  If yes, I would be explicit about using the feed type in the REQUEST line(s).

- currently, the writing of the data is spread across 4 machines.  Will
  consolidating all requests to a single machine cause problems?

  I can't tell if this is likely to be true since I don't know what the
  Product IDs for the products look like, so I don't know what the 
  file names generated by the pqact.conf action:

  EXP ^(.*)   FILE    -overwrite  /local/d/mlso_data/\1

  will look like.  I am guessing, however, that since the beginning
  part of the Product IDs are different (based on the REQUEST lines
  in the ldmd.conf files on puka, puu, and wikiwiki), then the
  putput file names will be different.

- the REQUEST lines from all three ldmd.conf files are pathological.  The
  inclusion of the .* at the end of each pattern is not needed and, in fact,
  not desired.

  For instance, the following REQUEST from puka:

  request ANY "^chip/.*" kenobi.mlso.ucar.edu

  is much better written as:

  request ANY "^chip/" kenobi.mlso.ucar.edu

- the REQUEST lines look pretty simple.  The multiple requests from
  puu could be combined into something much simpler.  For instance:

  change:

  request ANY "^pchip/.*|^plowl/.*|^pextra/.*|^fchip/.*|^fmk4/.*|^fpics/.*" 
kenobi.mlso.ucar.edu

  to:

  request ANY "^(pchip/|plowl/|pextra/|fchip/|fmk4/|fpics/)" 
kenobi.mlso.ucar.edu

  If the trailing '/' in each pattern is superfluous (i.e., if the characters 
preceeding
  the '/' are not repeated elsewhere in the Product ID), this can be condensed 
to:

  request ANY "^(pchip|plowl|pextra|fchip|fmk4|fpics)" kenobi.mlso.ucar.edu

- the 'pqexpire' invocation in the ldmd.conf file on puu can safely be commented
  out *** assuming that you are running a non-archaic version of the LDM ***

- the pqact.conf file has 4 actions for products in the IDS|DDPLUS datastream.
  If none of the data from kenobi are tagged as being in IDS|DDPLUS, these
  actions can be commented out.

I have a hunch that the products being transferred from kenobi have Product IDs
that are sufficiently different to allow you to simply combined all of the
REQUEST lines into a single ldmd.conf REQUEST entry... BUT I can not be sure
until I can see what the Product IDs look like.

Request:

- either capture a significantly long run of Product IDs and send us the
  results:

  <as 'ldm'>
  notifyme -vl logs/kenobi_products.log -f EXP

- OR, and preferably, ALLOW machines from the unidata.ucar.edu subdomain
  to REQUEST data from kenobi.  This is done by uncommenting the generic
  ALLOW for unidata.ucar.edu machines in the ldmd.conf file on kenobi:

  allow ANY
    ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$))

We can chat about this on the phone if you like (x8642).

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: NDJ-769600
Department: Support IDD
Priority: Normal
Status: Closed