Re: 20011008: IDD latencies at PSU (cont.)

Hi Art and Jeff,

I've gathered together a few pieces of the various emails I've received
in order to compose an answer.  I'll respond to your comments about a
large queue in a separate email, probably tomorrow.

Art, in your very last email to me you had said you'd send some emails
from Jeff.  Nothing actually followed - perhaps an error, but I was
already cc'ed on two messages from Jeff, so I believe I have them all.

----
Jeff wrote:

> Heh, Here's what is probably a large contributor to the problem.
> OTC has 2 seperate OC-3 circuits for the I1 and I2 connections.
> The I1 connection is saturated inbound most of the day. Outbound is
> not that bad these days. I2 is hardly loaded, but the connections are both
> served on the same Cisco 7513, and that box is probably severely overloaded.
> There are plans to move things to a new box, I'll find out more on that on
> friday.

The traceroutes from here have all shown an I2 route to PSU:

[root@imogene ipv4]# traceroute ldm.meteo.psu.edu
traceroute to ldm.meteo.psu.edu (128.118.28.12), 30 hops max, 38 byte
packets
 1  flra-n140 (128.117.140.252)  0.331 ms  0.279 ms  0.258 ms
 2  vbnsr-n2.ucar.edu (128.117.2.252)  0.895 ms  0.750 ms  0.659 ms
 3  internetr-n243-104.ucar.edu (128.117.243.106)  0.830 ms  0.757 ms 
0.764 ms
 4  denv-abilene.ucar.edu (128.117.243.126)  1.631 ms  3.047 ms  2.038
ms
 5  kscy-dnvr.abilene.ucaid.edu (198.32.8.14)  13.112 ms  12.695 ms 
12.518 ms
 6  ipls-kscy.abilene.ucaid.edu (198.32.8.6)  21.882 ms  21.427 ms 
21.450 ms
 7  clev-ipls.abilene.ucaid.edu (198.32.8.26)  28.472 ms  27.999 ms 
27.768 ms
 8  abilene.psc.net (192.88.115.122)  31.130 ms  31.053 ms  31.417 ms
 9  penn-state.psc.net (198.32.224.66)  35.709 ms  36.520 ms  36.460 ms
10  * * *
11  ldm.meteo.psu.edu (128.118.28.12)  37.320 ms  37.070 ms  37.323 ms
12  ldm.meteo.psu.edu (128.118.28.12)  36.868 ms  38.276 ms  37.222 ms

So, I'm assuming that the saturated inbound I1 connection is not the
problem.  Now, whether the Cisco box is a problem or not, I couldn't
say.  Is it really the case that that box can't handle two OC-3s?

----
> > However, in previous communication with Unidata, I indicated that we get
> > other higher volume streams from Wisconsin and Oklahoma via I2 on time,
> > within seconds.  So, that would indicate it's not a raw bandwidth problem,
> > wouldn't it?

> In general yes, that would point to a problem at Unidata as opposed to PSU.
> However, there's no guarantee that the problem isn't local.

From our end, we've seen no other indications of problems to any other
site (unlike other times, as you know).

----
> I've heard back from Jeff twice and I will include both his responses
> below.  In response to his questions, I did a cat on
> /proc/sys/net/ipv4/tcp_ecn and got "0" back, so I'm assuming this is not
> involved.  Also, if our congested router is suspect, it would have to be
> somehow throttling the stream from motherlode but not others since it
> seems to be the only one affected, but I suppose that's somehow possible.
> Jeff suggests a ttcp test from Unidata to our machine.  Are you familiar
> with this and could we set something like this up on motherlode?
> 

This sounds like a good idea.  We have a machine,
milton.unidata.ucar.edu, that is a very similar architecture to
motherlode.  The connectivity is also very similar.  The only difference
is that milton is located down here at the Foothills Lab (FL), while
motherlode is up at the Mesa Lab (ML).  But, traffic to FL flows through
ML, so we'll be testing the same route the data takes.

The connection between ML and FL is a Gigabit ethernet connection that
is rarely full.  From UCAR to Front Range Gigapop is an OC-3 and from
FRGP to Abilene is also an OC-3.  

Jeff, please let us know what we need to do in order to set up for this
test.

We are in the midst of our User Comm meeting, thus, I'm running back and
forth between the meeting and my email.  This will also be true until
noon local time tomorrow.  However, I will try to be  responsive in
helping to get this set up.

Anne
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************