[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
20030308: high latencies with DDPLUS feed
- Subject: 20030308: high latencies with DDPLUS feed
- Date: Sun, 09 Mar 2003 10:53:02 -0700
>From: Robert Mullenax <address@hidden>
>Organization: NMSU/NSBF
>Keywords: 200303090454.h294sGB2028582 IDD latency
Robert,
>Is anyone else experiencing very high latencies with the DDPLUS|IDS feed?
I checked the latencies on several machines that are reporting real
time statistics, and don't see any systematic latency problems. Who
is your upstream IDD feed site currently? It may be that your data
originates from a machine that is not reporting real time statistics,
so we can't tell if it is having problems.
>I have tried all my different sources and the latency with the HDS (not
>unusual) and IDS|DDPLUS (ver unusual) is 57-62 minutes. It doesn't seem
>to be a bandwidth issue locally and the NNEXRAD feed has almost 0 latency.
You can check on the latencies at sites reporting real time statistics
at the following:
Sites running LDM-5:
http://www.unidata.ucar.edu/staff/chiz/latency/stats
Sites running LDM-6:
http://www.unidata.ucar.edu/staff/chiz/rtstats
In either case, I recommend clicking on 'Statistics by Host' and then
selecting an IDD host by name. After doing that, you will be presented
with a list of feeds the site is getting and different views of data
reception: latency, log(latency), volume, # products, and topology.
Clicking on the latency link for any feed will result in a time series
plot of the latency in seconds for that feed for that host.
For sites running LDM-6.x, the differential latency from that host to
its upstream feeder can be displayed by clicking on 'topology' and then
selecting the upstream hostname link _if it is displayed as a link_.
You can only calculate differential latencies if a site's upstream host
is also reporting real time statistics.
FYI, LDM-6 is in its final stages of test. We expect to announce its
general availability real soon now (documentation is being worked on.
If you would like to get a jump on LDM-6, and if you don't mind having
to load new test versions as they become available, then I recommend
that you load our latest release candidate, LDM-6.0.2:
I. Download the LDM-6 release candidate and unpack it
<login to your IDD machine as 'ldm'>
cd ~ldm
ftp ftp.unidata.ucar.edu
<user> anonymous
<pass> your_full_email_address
cd pub/ldm/test
binary
get ldm-6.0.2.tar.Z
quit
zcat ldm-6.0.2.tar.Z | tar xvf -
II. Define build the LDM-6 release candidate
<make sure that the Unix environment variable LDMHOME is correctly
defined in your .cshrc file; 'source' the file>
cd ldm-6.0.2/src
./configure <- LDM-6.x is only available as source
make <- if there were no configure errors
make install <- if there were no errors in the make step
sudo make install_setuids <- must be done as 'root'; do what is necessary
cd ../bin
III. Modify the LDM-6 startup script, ldmadmin, to match your current
setup
<compare entries in ldmadmin with those in the ~ldm/bin/ldmadmin you
are currently using. Things to look out for are:
1) if a 'uname -n' does not return a fully qualified hostname for your
machine, you will have to set the machine name in the hostname
section of ldmadmin:
change:
chop($hostname = `uname -n`);
# $hostname = "your.hostname.here";
to:
# chop($hostname = `uname -n`);
$hostname = "psnldm.nsbf.nasa.gov";
2) modify the LDM product queue size, $pq_size, to match your existing
queue
3) modify the number of LDM log files, $numlogs, to match the number of
logs you want to keep on disk at any time
IV. Add reporting of real time statistics in your ldmd.conf file
add:
exec "rtstats -h rtstats.unidata.ucar.edu"
in the "exec" section of ~ldm/logs/ldmd.conf
V. Examine your ldmd.conf file request lines and adjust them where necessary
1) look for "pathological" regular expressions in the requests. For instance
the regular expression in this request line:
request NNEXRAD ".*(FTG)" upstream_feed_host
is pathological. The leading '.*' is not needed, and causes a LOT
of extra work for certain system's (e.g. Sun Solaris) regular expression
packages. An equivalent, and non-pathological regular expression is:
request NNEXRAD "(FTG)" upstream_feed_host
The general guideline we can offer is to make your regular expression
as simple as possible. Leading and trailing '.*' fields sorrounding
"stuff" that will match is not needed and undesirable.
NOTE: the following is _NOT_ a pathological regular expression:
request NNEXRAD ".*" upstream.feed.host
Regular expression packages pretty much all know that a ".*" pattern
means that no matching has to be done.
2) LDM-6 does not accumulate request lines with the same upstream
host into a single request like LDM-5 did. For instance, the following
two ldmd.conf lines are used differently by the different versions
of the LDM:
request NNEXRAD ".*" upstream.feed.host
request HDS ".*" upstream.feed.host
LDM-5 accumulates thes into a single rpc.ldmd to get the data:
request NNEXRAD|HDS ".*" upstrea.feed.host
LDM-6 uses two separate rpc.ldmds to get the data. The change
was made to ease the process of splitting feed requests. If
too much data is put on a single feed request, then produts
may have increased latencies since they are waiting for transmit
until others in the same request have been processed. If
the upstream feed host is electronically "close" to your machine,
then this issue practically disappears. If, on the other hand,
the upstream feed host is electronically distant from the
local host, then it has a big impact.
The process of organizing feed requests into the fewest number
(to keep the number of processes down on both the local host
AND on the upstream feed site) while not putting too much
on a single request is an iterative one. The guidance I can
offer is best illustrated by one example:
LDM-5 feed request:
request UNIDATA ".*" upstream.feed.host
request FLS2 ".*" upstream.feed.host
LDM-6 feed requests equivalent but much better for keeping latencies down:
request DDPLUS|IDS|FSL2 ".*" upstream.feed.host
request HRS ".*" upstream.feed.host
request UNIWISC ".*" upstream.feed.host
I am _not_ suggesting that LDM-6 is less efficient than LDM-5 when
moving data; it is _much_ more efficient. I _am_ suggesting that
more attention needs to be paid to request ldmd.conf request lines
so that overall latencies can be kept to a minimum. With
proper configuration of ldmd.conf request entries and reasonable
network connectivity, there is no reason why any continental US site
can not have near-zero latencies for _any_ feed. This, by the way,
includes the CONDUIT and CRAFT feeds. CONDUIT, for comparison,
contains about 75% of the total volume of IDD feeds. We are able
to relay all of this data to sites with suitable bandwidth with
little to no latencies (I'm talking about one to a few seconds!).
VI. Switch to using the new LDM
cd ~ldm
ldmadmin stop
rm runtime
ln -s ldm-6.0.2 runtime
<make sure that all rpc.ldmd processes have exited>
rehash
ldmadmin start
Note that LDM-6's ldmadmin will not return you to the Unix command
prompt until all rpc.ldmd processes have exited. This is nice so
we can stop admonishing users to "wait for all rpc.ldmd processes
to exit"!
After you LDM-6 release candidate is up and running, visit the real time
statistics page(s) to see how you are doing. This will take a few minutes
to show up, but not many.
Please let me know if/when you are going to upgrade to LDM-6
Tom
>From address@hidden Wed Mar 12 12:41:25 2003
Tom,
At this time with the ULDB thing still going on I don't want to change
the LDM. After it is finished I do want to get LDM 6 installed
as soon as I can.
I don't really understand how we end up with the latencies we do sometimes.
I was using ldm.meteo.psu.edu and sunset.aos.wic.edu..they had no
issues.
Anyway at least as of last night it was toleable again.
Thanks,
Robert