[IDD #NYU-552405]: LDM and packet shaping -- to Tom Yoksas

Hi Dave,

re:
> Do you remember during the LDM training session we looked at
> Millersville's LDM logs and determined that I was losing CONDUIT data
> due to packet shaping?

Actually, I was left with the impression that there was some kind of
volume limiting going on after comparing the latencies seen in lower
volume feeds like IDS|DDPLUS with the high volume CONDUIT feed.  The log
file didn't tell us much other than the original process ingesting
CONDUIT was _not_ exiting or issuing errors/notices.

> I have been working with our networking people and they say that Twister
> is exempt from all packet shaping.

Hmm...

> I showed them Twisters' LDM
> statistics, but they are not sure where the problem may lie.

If there really is no packet shaping beind done for twister, then I am
also uncertain of what could be starving the CONDUIT ingest process(es).

> In order to
> help track down the cause of our CONDUIT data loss, they are asking me
> to contact you and have you explain how it was determined that packet
> shaping was occurring.

The "classic" signature of volume-based flow limiting (which I commonly
refer to as packet shaping) is very low latencies on feeds that have
small data volumes (e.g., IDS|DDPLUS, FSL2, UNIWISC, etc.)

http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+twister.esci.millersville.edu

while at the same time high volume feeds (e.g., HDS, NEXRAD2, CONDUIT,
etc.) have very high latencies

http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+twister.esci.millersville.edu

By the way, the current latencies for IDS|DDPLUS on twister have been high
for extended periods of time.  This is not what we saw during the workshop.

It was also very interesting that a 5-way split of CONDUIT on twister resulted
in a rapid decrease in latencies followed after not so long a time by those
latencies going back up.

> They also ask if Unidata is doing any shaping
> of your outgoing data streams (which I know you are not)

No, there is no packet shaping being done here at UCAR.

> or if your ISP
> may be doing so (which I know is not happening.)

Again, no.  UCAR internally has a Gbps LAN, and our connection to NLR
and I2 is unimpeded.  If there were any limiting beind one at the
source, we would see a signature in more of the some 400 downstream
connections than just the small number from twister.

> Can you help?

I will get together with our brain trust (Mike Schmidt, Steve Emmerson,
and Steve Chiswell) to see if we can gleen something new/different
from the latency plots we are currently seeing for twister.

> Thanks!

No worries.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: NYU-552405
Department: Support IDD
Priority: Normal
Status: Closed