20030725: LDM access by anonymous FTP (cont.)

>From: Mark Bradford <address@hidden>

re: FTP from sites outside the UCAR security perimeter work
>Weird, but apparently true; I just transferred the distribution using my
>home Linux box as an intermediary.  Building now.

OK, sounds good.

re: what to change in ldmamdin when 'uname -n' is not a FQDN
>Cool, will do.  Thanks again.

>For the time being, I will just fire it up with my old ldmd.conf; is there
>a concise document describing ways to optimize ldmd.conf for LDM 6, or
>do I need to actually (*gasp*) read and comprehend the LDM 6 docs?

There is no concise document about how to tune one's ldmd.conf file.
This is mainly because we are still learning things that should and
shouldn't be done.  The general guidelines, however, are:

1) eliminate problematic regular expressions in both feed requests (ldmd.conf)
   and in pqact actions (pqact.conf).  Problematic regular expressions
   are those that have unneeded '.*'s.  Here is one simple example:

   This is bad:

   request      CONDUIT ".*[09]$"       upstream_feed_host

   This is good and it does the same job:

   request      CONDUIT "[09]$" upstream_feed_host

   An unneeded ".*" is one that occurs at the beginning or at the end
   of the regular expression.

2) it is up to the user to split feed requests when needed and combine
   them when not.  What I mean by this is probably best illustrated by
   way of an example.  Let's assume that a downstream site is electronically
   far from its upstream feed host, and that the downstream host wants
   to get all of the feeds in the compound feed type 'UNIDATA" (which
   is IDS|DDPLUS|HDS|UNIWISC).

   This will probably be bad:

   request      UNIDATA ".*"    upstream_feed_host

   Splitting the request into several is probably called for:

   request      IDS|DDPLUS      ".*"    upstream_feed_host
   request      HDS     ".*"    upstream_feed_host
   request      UNIWISC ".*"    upstream_feed_host

   The need for one splitting or combining feds depends on the network
   pipe into his/her site; the network pipe out of his/her upstream
   feed host; and how far apart the sites are.

   For all sites at UCAR, the original feed request should be OK:

   request      UNIDATA ".*"    upstream_feed_host

   The overriding principle in splitting and combining feed requests is
   to end up with the minimun number of requests that will get you
   the data reliably.  This is more art than science at this point.

3) users need to add a new 'exec' line to their ldmd.conf file:

   exec "rtstats -h rtstats.unidata.ucar.edu"

   It is probably "best" (most tidy) if this is the last 'exec' line
   in the user's ldmd.conf file.

   If sites are using feed existing feed types to move their own data
   to a cooperating site, we would appreciate if they not report this
   in their rtstats line.  One can specify which feeds to report stats
   for by specifying the '-f' flag.  Please see the rtstats man page
   for more information.

4) I have noticed that a number of sites have had more than one feed
   request for the same data.  Here is one example:

   request      WMO     ".*"    upstream_feed_host
   request      HDS     ".*"    upstream_feed_host

   In LDM-5 this would not be an "error" since LDM-5 collected all
   feed requests to a single upstream host into a single rpc.ldmd.
   Since LDM-6 does not do this collection, having more than one
   feed request for the same data will result in twice the amount
   of data arriving at the user's site.  This is generally a bad
   practice since it is a waste of bandwidth.

5) LDM-6 supports the notion of PRIMARY and ALTERNATE feed requests.
   A PRIMARY feed request is one where the receiving LDM is telling
   the sending LDM to send down products that match the requests
   regular expression without asking if you want them.  An ALTERNATE
   feed request is where the downstream LDM tells the upstream LDM
   to ask "me" if I want a product before it is sent.  The combination
   of PRIMARY and ALTERNATE feed requests can be used as kind of an
   automatic failover.  Here is an example:

   request      HDS     ".*"    upstream_feed_host PRIMARY
   request      HDS     ".*"    upstream_feed_host2 ALTERNATE

   One would set a PRIMARY feed (the default) to the upstream host that
   is most likely to be able to provide the data the quickest.  The
   ALTERNATE request would then result in a single query for each product
   which would most likely result in a decline by the downstream site
   since it will most likely have already received the product from the
   upstream PRIMARY host.

   By the way, the use of PRIMARY and ALTERNATE feed requests can
   be abused.  Just remember that the ALTERNATE feed request does
   use your and your upstream feed site resources.

6) if a site is upgrading from an LDM version previous to LDM-5.1.3,
   s/he will need to delete and remake her/his LDM queue.  The queue
   in versions 5.1.3+ are compatible with the current releases of
   LDM-6.

7) document your ldmd.conf file!!  You would be suprised by what a
   mess most ldmd.confs I have seen are.  Some were to the point of
   being unreadable.  A little documentation now saves a lot of
   headache down the road.

That is all I can come up with right now...

Tom