This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
>To: Jeff Weber <address@hidden> >From: Arthur A. Person <address@hidden> >Subject: Re: 20000831: Diagnosing data slowness at Penn State (new thread) >Organization: Penn State Department of Meteorology >Keywords: RPC performance, product latency, LDM tuning Jeff, Here's a bit more information on Art's question. Use whatever you want from this answer, unless you want to wait until I have more benchmarks on using multiple connections ... > What is the maximum bandwidth in products/second as a function of > turn-around time on the network? For example, if each product transmitted > to a downstream site had one turn-around wait associated with it, and the > average network turn-around time were 40ms, then a maximum of 1/0.040=25 > products/second could be transmitted before products would begin to > backlog on any particular IDD stream. I'm asking this because I'm > wondering if we might be pushing the limit on the number of products per > stream into navier which ingests all of the Noaaport and NMC2 stuff on one > stream (is that correct?... I'm assuming one connection to motherlode > results in all products being sent serially over the network in one > stream). For small products (less than 16 Kbytes), one remote procedure call (RPC) per product is required, which is one turn-around wait. So your analysis is correct for an average network turn-around time of 40 ms. To make sure the LDM processing doesn't add any significant amount to this, I just tested sending 100000 small (text) products from one host on our local network to another via the LDM. This achieved a maximum of 1340 products per second, showing the LDM won't be the bottleneck until network latencies are less than 1 ms. Even after 100000 products were transferred, the rate was still over 900 products per second in this experimental run. (For versions prior to LDM 5.1.2 this would have slowed down much more; I'll be benchmarking that soon.) I don't think making multiple connections will improve this any, but I'll try that also, using multiple different /etc/hosts aliases for the same upstream server to see if more products per second are transferred in that case. > Can you enlighten me on interproduct latencies and theoretical maximum > products/second? Or am I imagining a problem that wouldn't exist? For large products (greater than 16 Kbytes), an initial RPC call is required to see if the downstream host wants the product, then an RPC call is required for each "chunk" of the large product; I think the chunks are 8 kbytes each. For small products, the time to send a product is essentially the time for a single RPC call with response. The ldmping program can be used to see what the time for a single LDM RPC call is between two hosts, from which you can calculate the maximum products/second. Make sure you get multiple ldmping calls with the "-i" argument, since the first RPC call which includes the connection set up time will definitely be slower than subsequent RPC calls. For example, here's what we see on our local area network between the two hosts I used: $ ldmping -i 1 buddy Sep 05 15:43:43 State Elapsed Port Remote_Host rpc_stat Sep 05 15:43:43 RESPONDING 0.013054 59710 buddy Sep 05 15:43:44 RESPONDING 0.000608 59710 buddy Sep 05 15:43:45 RESPONDING 0.000505 59710 buddy Sep 05 15:43:46 RESPONDING 0.000557 59710 buddy Sep 05 15:43:47 RESPONDING 0.000496 59710 buddy so I can conclude it takes about .5 ms per RPC call. This sets a theoretical maximum of 2000 null products per second, which makes the 1340 DDPLUS products per second seen in the experiment seem reasonable. --Russ