Re: 20000831: Diagnosing data slowness at Penn State (new thread) (fwd)

>To: Jeff Weber <address@hidden>
>From: Arthur A. Person <address@hidden>
>Subject: Re: 20000831:  Diagnosing data slowness at Penn State (new thread)
>Organization: Penn State Department of Meteorology
>Keywords: RPC performance, product latency, LDM tuning

Jeff,

Here's a bit more information on Art's question.  Use whatever you
want from this answer, unless you want to wait until I have more
benchmarks on using multiple connections ...

> What is the maximum bandwidth in products/second as a function of
> turn-around time on the network?  For example, if each product transmitted
> to a downstream site had one turn-around wait associated with it, and the
> average network turn-around time were 40ms, then a maximum of 1/0.040=25
> products/second could be transmitted before products would begin to
> backlog on any particular IDD stream.  I'm asking this because I'm
> wondering if we might be pushing the limit on the number of products per
> stream into navier which ingests all of the Noaaport and NMC2 stuff on one
> stream (is that correct?... I'm assuming one connection to motherlode
> results in all products being sent serially over the network in one
> stream).

For small products (less than 16 Kbytes), one remote procedure call
(RPC) per product is required, which is one turn-around wait.  So your
analysis is correct for an average network turn-around time of 40 ms.
To make sure the LDM processing doesn't add any significant amount to
this, I just tested sending 100000 small (text) products from one host
on our local network to another via the LDM.  This achieved a maximum
of 1340 products per second, showing the LDM won't be the bottleneck
until network latencies are less than 1 ms.  Even after 100000
products were transferred, the rate was still over 900 products per
second in this experimental run.  (For versions prior to LDM 5.1.2
this would have slowed down much more; I'll be benchmarking that
soon.)

I don't think making multiple connections will improve this any, but
I'll try that also, using multiple different /etc/hosts aliases for
the same upstream server to see if more products per second are
transferred in that case.

> Can you enlighten me on interproduct latencies and theoretical maximum
> products/second?  Or am I imagining a problem that wouldn't exist? 

For large products (greater than 16 Kbytes), an initial RPC call is
required to see if the downstream host wants the product, then an RPC
call is required for each "chunk" of the large product;  I think the
chunks are 8 kbytes each.  For small products, the time to send a
product is essentially the time for a single RPC call with response.
The ldmping program can be used to see what the time for a single LDM
RPC call is between two hosts, from which you can calculate the
maximum products/second.  Make sure you get multiple ldmping calls
with the "-i" argument, since the first RPC call which includes the
connection set up time will definitely be slower than subsequent RPC
calls.

For example, here's what we see on our local area network between the
two hosts I used:

 $ ldmping -i 1 buddy
 Sep 05 15:43:43      State    Elapsed Port   Remote_Host           rpc_stat
 Sep 05 15:43:43 RESPONDING   0.013054 59710   buddy        
 Sep 05 15:43:44 RESPONDING   0.000608 59710   buddy        
 Sep 05 15:43:45 RESPONDING   0.000505 59710   buddy        
 Sep 05 15:43:46 RESPONDING   0.000557 59710   buddy        
 Sep 05 15:43:47 RESPONDING   0.000496 59710   buddy        

so I can conclude it takes about .5 ms per RPC call.  This sets a
theoretical maximum of 2000 null products per second, which makes the
1340 DDPLUS products per second seen in the experiment seem
reasonable.

--Russ