20030308: high latencies with DDPLUS feed

>From:  Robert Mullenax <address@hidden>
>Organization:  NMSU/NSBF
>Keywords:  200303090454.h294sGB2028582 IDD latency

Robert,

>Is anyone else experiencing very high latencies with the DDPLUS|IDS feed?

I checked the latencies on several machines that are reporting real
time statistics, and don't see any systematic latency problems.  Who
is your upstream IDD feed site currently?  It may be that your data
originates from a machine that is not reporting real time statistics,
so we can't tell if it is having problems.

>I have tried all my different sources and the latency with the HDS (not
>unusual) and IDS|DDPLUS (ver unusual) is 57-62 minutes.  It doesn't seem
>to be a bandwidth issue locally and the NNEXRAD feed has almost 0 latency.  

You can check on the latencies at sites reporting real time statistics
at the following:

Sites running LDM-5:

http://www.unidata.ucar.edu/staff/chiz/latency/stats

Sites running LDM-6:

http://www.unidata.ucar.edu/staff/chiz/rtstats

In either case, I recommend clicking on 'Statistics by Host' and then
selecting an IDD host by name.  After doing that, you will be presented
with a list of feeds the site is getting and different views of data
reception: latency, log(latency), volume, # products, and topology.
Clicking on the latency link for any feed will result in a time series
plot of the latency in seconds for that feed for that host.

For sites running LDM-6.x, the differential latency from that host to
its upstream feeder can be displayed by clicking on 'topology' and then
selecting the upstream hostname link _if it is displayed as a link_.
You can only calculate differential latencies if a site's upstream host
is also reporting real time statistics.

FYI, LDM-6 is in its final stages of test.  We expect to announce its
general availability real soon now (documentation is being worked on.

If you would like to get a jump on LDM-6, and if you don't mind having
to load new test versions as they become available, then I recommend
that you load our latest release candidate, LDM-6.0.2:

I. Download the LDM-6 release candidate and unpack it

<login to your IDD machine as 'ldm'>
cd ~ldm
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> your_full_email_address
  cd pub/ldm/test
  binary
  get ldm-6.0.2.tar.Z
  quit

zcat ldm-6.0.2.tar.Z | tar xvf -

II. Define build the LDM-6 release candidate

<make sure that the Unix environment variable LDMHOME is correctly
defined in your .cshrc file; 'source' the file>

cd ldm-6.0.2/src
./configure                 <- LDM-6.x is only available as source
make                        <- if there were no configure errors
make install                <- if there were no errors in the make step
sudo make install_setuids   <- must be done as 'root'; do what is necessary
cd ../bin

III.  Modify the LDM-6 startup script, ldmadmin, to match your current
      setup

<compare entries in ldmadmin with those in the ~ldm/bin/ldmadmin you
are currently using.  Things to look out for are:

1) if a 'uname -n' does not return a fully qualified hostname for your
   machine, you will have to set the machine name in the hostname
   section of ldmadmin:

change:

chop($hostname = `uname -n`);
# $hostname = "your.hostname.here";

to:

# chop($hostname = `uname -n`);
$hostname = "psnldm.nsbf.nasa.gov";

2) modify the LDM product queue size, $pq_size, to match your existing
   queue

3) modify the number of LDM log files, $numlogs, to match the number of
   logs you want to keep on disk at any time


IV. Add reporting of real time statistics in your ldmd.conf file

add:

exec    "rtstats -h rtstats.unidata.ucar.edu"

in the "exec" section of ~ldm/logs/ldmd.conf

V.  Examine your ldmd.conf file request lines and adjust them where necessary

1) look for "pathological" regular expressions in the requests.  For instance
   the regular expression in this request line:

request NNEXRAD ".*(FTG)"       upstream_feed_host

   is pathological.  The leading '.*' is not needed, and causes a LOT
   of extra work for certain system's (e.g. Sun Solaris) regular expression
   packages.  An equivalent, and non-pathological regular expression is:

request NNEXRAD "(FTG)" upstream_feed_host

   The general guideline we can offer is to make your regular expression
   as simple as possible.  Leading and trailing '.*' fields sorrounding
   "stuff" that will match is not needed and undesirable.

   NOTE: the following is _NOT_ a pathological regular expression:

request NNEXRAD ".*"   upstream.feed.host

   Regular expression packages pretty much all know that a ".*" pattern
   means that no matching has to be done.

2) LDM-6 does not accumulate request lines with the same upstream
   host into a single request like LDM-5 did.  For instance, the following
   two ldmd.conf lines are used differently by the different versions
   of the LDM:

request NNEXRAD ".*"    upstream.feed.host
request HDS     ".*"    upstream.feed.host
  
   LDM-5 accumulates thes into a single rpc.ldmd to get the data:

request NNEXRAD|HDS     ".*"    upstrea.feed.host

   LDM-6 uses two separate rpc.ldmds to get the data.  The change
   was made to ease the process of splitting feed requests.  If
   too much data is put on a single feed request, then produts
   may have increased latencies since they are waiting for transmit
   until others in the same request have been processed.  If
   the upstream feed host is electronically "close" to your machine,
   then this issue practically disappears.  If, on the other hand,
   the upstream feed host is electronically distant from the
   local host, then it has a big impact.

   The process of organizing feed requests into the fewest number
   (to keep the number of processes down on both the local host
   AND on the upstream feed site) while not putting too much
   on a single request is an iterative one.  The guidance I can
   offer is best illustrated by one example:

LDM-5 feed request:

request UNIDATA ".*"    upstream.feed.host
request FLS2    ".*"    upstream.feed.host

LDM-6 feed requests equivalent but much better for keeping latencies down:

request DDPLUS|IDS|FSL2 ".*"    upstream.feed.host
request HRS     ".*"    upstream.feed.host
request UNIWISC ".*"    upstream.feed.host

   I am _not_ suggesting that LDM-6 is less efficient than LDM-5 when
   moving data; it is _much_ more efficient.  I _am_ suggesting that
   more attention needs to be paid to request ldmd.conf request lines
   so that overall latencies can be kept to a minimum.  With
   proper configuration of ldmd.conf request entries and reasonable
   network connectivity, there is no reason why any continental US site
   can not have near-zero latencies for _any_ feed.  This, by the way,
   includes the CONDUIT and CRAFT feeds.  CONDUIT, for comparison,
   contains about 75% of the total volume of IDD feeds.  We are able
   to relay all of this data to sites with suitable bandwidth with
   little to no latencies (I'm talking about one to a few seconds!).
   

VI.  Switch to using the new LDM

cd ~ldm
ldmadmin stop
rm runtime
ln -s ldm-6.0.2 runtime

<make sure that all rpc.ldmd processes have exited>

rehash
ldmadmin start

    Note that LDM-6's ldmadmin will not return you to the Unix command
    prompt until all rpc.ldmd processes have exited. This is nice so
    we can stop admonishing users to "wait for all rpc.ldmd processes
    to exit"!

After you LDM-6 release candidate is up and running, visit the real time
statistics page(s) to see how you are doing.  This will take a few minutes
to show up, but not many.

Please let me know if/when you are going to upgrade to LDM-6

Tom

>From address@hidden Wed Mar 12 12:41:25 2003

Tom,

At this time with the ULDB thing still going on I don't want to change
the LDM.  After it is finished I do want to get LDM 6 installed
as soon as I can.

I don't really understand how we end up with the latencies we do sometimes.
I was using ldm.meteo.psu.edu and sunset.aos.wic.edu..they had no
issues.

Anyway at least as of last night it was toleable again.

Thanks,
Robert