[LDM #XXW-990118]: LDM - Not your average latency problem

Mike,

> >I infer the following from the MRTG and the rtstats(1) plots for
> >SonomaTech: 1) the SonomaTech LDM restarted around noon WDT on the 18th;
> >2) it saturated the pipe; 3) it didn't catch up until just before 1700
> >WDT; 4) throughout the subsequent night, the pipe was occationally
> >saturated; and 5) the pipe became saturated again around the start of the
> >workday at SonomaTech.
> 
> The pipe becomes saturated during daylight hours because the NIMAGE feed
> volume has a diurnal cycle....... it's based largely on visible satellite
> imagery and so it increases during the day.
> 
> >Judging from this and the good performance of the LDM in sending the same
> >data-products from Rossby to here, I think it's safe to say that
> >SonomaTech's pipe just isn't big enough to receive the data-products that
> >it's requesting.
> 
> Yes, this was never in doubt. But they should be able to get half their
> pipe in volume without much latency, and that is the main thing I've been
> trying to figure.  I added back in the NIMAGE feed yesterday to prove a
> point. Without the NIMAGE feed, we were having large latencies ( for
> unknown reasons, I would argue). After the NIMAGE feed was added, and the
> feeds were combined, we got much better performance.
> 
> >Merging all the requests into one was done to validate the hypothesis that
> >a SonomaTech router was favoring higher-volume LDM connections at the
> >expense of lower-volume ones.  That appears to be the case.
> 
> Hmm...could it be a router CPU problem? But if we lower volume a bunch
> (without NIMAGE), you would think it could handle it, even if the feeds
> were split. OK, I'll give you a chance to make a prediction...maybe you can
> get Tom in on it and we can have an office pool.
> 
> GIVEN: the current good performance of the LDM at STI (aside from the
> expected latencies due to their small pipe), and considering their current
> feed volume, and the fact that their feed types are combined in one entry...
> 
> PREDICT: what will happen if we request the feeds separately, AND we do not
> request the NIMAGE feed (a reduction in volume by more than half). Now
> myself, and I'm guessing most others that have played around with the LDM,
> would never predict that large latencies would develop and certain feeds
> might just stop coming in. Any takers? This is the fundamental thing that I
> have been unable to understand.

Unfortunately, you're proposing to change two parameters simultaneously (volume 
and number of connections).  I think I can predict the outcome when only one is 
changed.  If only the volume is decreased, then I expect the average latency to 
decrease.  If only the number of connections is increased, then I expect the 
average latency to *increase* due to a hypothesized problem with Sonomatech's 
IP-mapping router managing its connection tables.

Feel free to test this out -- but I'd only change one parameter at a time.

> Again, I appreciate your indulgence, you can ignore me anytime your
> patience has been exhausted. :-)

Compared to my impatience with myself, you're a piece of cake.  :-)

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: XXW-990118
Department: Support LDM
Priority: Normal
Status: Closed