20030401: LDM-6 request setup on rossby.met.sjsu.edu (cont.)

>From: Mike Voss <address@hidden>
>Organization: Unidata Program Center/UCAR
>Keywords: 200304011736.h31Haa7U020077 IDD LDM-6 ldmd.conf request

Hi Mike,

>Thanks for checking on the stats for rossby.met.sjsu.edu.

The real time statistis make it _so_ much easier to keep tabs on the
pulse of the IDD.  Kudos to Chiz!

re: time drifting on rossby
>1) Yes, it turns out my xntpd daemon was not running....hmmm. It is now.

Yup.  This would not be easily known by us without the real time stats!

re: data requests by IP, not fully qualified host name
>2) I had originally split the feeds to try and get better performance and
>insure that the IDS data was timely. But now since LDM 6 is so fast it
>doesn't seem to matter. I changed back to requesting all from a fully
>qualified name.

It is not true that splitting the feeds is never needed.  LDM-6 does
not collect data requests to a single upstream host into a single
rpc.ldmd like LDM-5 did.  Each request line in ldmd.conf results in a
single rpc.ldmd on both the client and server.  This makes it a LOT
easier to split feeds.  It can cause the end user to reevaluate their
feed requests to group feeds into single request lines.  For instance,
a good grouping would be to put FSL2 together with FNEXRAD and/or
UNIWISC.  A "bad" grouping (bad depending on a site's network
connectivity) might be to request UNIDATA from an upstream site.  It is
typically better to separate HDS requests from IDS|DDPLUS and UNIWISC.

The objective in "tuning" ldmd.conf is to end up with the fewest number
of request lines (and resulting rpc.ldmds) that provided reliable data
feeds with minimal latency.  There is no harm in keeping the requests
separated, but it does result in more LDM processes.  This would
probably be negligible on a receive-only system, but can be an issue
for sites relaying a lot of data.

>LDM 6 is great, what the heck did you all do to make it so much better?

Steve Emmerson rewrote the LDM to not using blocking RPCs for PRIMARY
data requests.  Also, for ALTERNATE requests, the upstream feeder asks
the downstream receiver if it wants each product in each stream.  If
the answer is yes, then the entire product is sent in one transaction.
LDM-5, on the other hand, would split up the products larger than 16384
bytes into 16384 byte chunks and send each chunk using a blocking RPC
call.  The result of that was that the feeding LDM was waiting for the
downstream to acknowledge the receipt of a product/piece of a product
before sending another product/piece of a product.  LDM-6 justs blasts
the data down to the receiver.

>The main thing I notice is how fast the queue is built and how fast the
>data catches up to zero latency after rebuilding a queue....simply amazing!!

LDM-6 is much more efficient in using the network to move data than LDM-5.
Your observation matches those of other sites that have made comment
on LDM-6.  I'm glad to hear that things are working nicely!

>Thanks for the great work out there,

I am CCing Steve E. and Chiz on this note so they can see your generous
comments.

Later...

Tom
Unidata WWW Service              http://my.unidata.ucar.edu/content/support