20030626: 20030624: HDS feed to/from seistan (cont.)

>From: Robert Leche <address@hidden>
>Organization: LSU
>Keywords: 200306161954.h5GJs2Ld016710 LDM-6 IDD

Hi Bob,

>In talking with our telecommunications people:
>
>1) The Louisiana Office of Telecommunications ("LANET")  was contacted with
>the problem and LANET reports Bell South (The states communications provider)
>has an open trouble ticket on the Public Switched sonnet network connecting
>ULM to the LANET. The trouble ticket reports:  CRC, Retransmission errors. 
>This is a  DS-3 Private Virtual Circuit  (PVC) on the Public Switched sonnet
>network connecting ULM to the LANET.

This sounds like the problem we uncovered at ULM.  They contacted their
service provider and rerouted their traffice from I2 to "I1".  We
never did get a reply from them as to what "I1" means.  After they
rerouted away from their problematic I2 connection, we were able to feed
all of HDS to them with virtually no latency.

>LANET indicated this trouble ticket
>has been open for "some time". We do not know what "some time" means in terms
>of  days or months. CRC, and retransmission errors are consistent with delays
>in network traffic.

Is this also affecting the LSU connection?  If not, there is still a
problem to be solved.

>2) Concerning Ping (ICMP):
>   A) LSU has limitation's placed on ICMP payload sizes to limit "the
>Ping Of Death" hacks. So it is interesting that even though LSU has this
>policy in place  we can demonstrate  large ICMP traffic to correctly query 
>systems other than ULM but not ULM.

OK.

>   B) The telecommunications people pointed out that Cisco router interface
>ping (ICMP) buffers have a hard limitation of 18,000 bytes. Unix/Linux systems
>do not have this issue. So the theory goes....  Ping LSU's Cisco border router,
>then LANET's Cisco border router and problems seem apparent. Yet ping an
>UNIX device with a large pay load beyond the Cisco device and travel time
>delays suddenly do not seem excessive.

I understand.  Even still, the pings with large ICMP packets from
seistan (RedHat 7.2 Linux) to zero.unidata.ucar.edu (Sun Solaris SPARC
5.9) show dramatic round trip time increases after the ping packet size
exceeds 20KB.  The 18000 byte limit you note does seem like what were
were seeing when trying to ping laNoc-lsubr.LEARN.la.net.

>3) It would be interesting to know who ULM is feeding HDS from. Chances are,
>the communications circuit they are currently using is the same DS-3 circuit
>that LANET uses.

Right now, ULM is feeding HDS from rainbow.al.noaa.gov (this is a
CU/CIRES lab here in Boulder).  We also fed them with no latency from
emo.unidata.ucar.edu.

>4) Limitations placed on ICMP payload sizes  on any devices in a networks
>path will cause problems in using  ICMP round trip time to measure network
>metrics.  But at this time, I do not have an alternative method to measure
>network latencies. My network guy said network latencies issues are handled
>by the circuit provider. No help there.

The ping packet size issue was just an interesting observation.  The
real issue is the latency when feeding the HDS stream out of LSU as
compared to virtually no latency when feeding the HDS stream _into_
LSU.  This observation is something that the telcomm people should be
able use to help isolate where the throttling is occuring on or near
the LSU campus.  Our being able to feed ULM all of the HDS feed from at
least two other sites and our not being able to feed HDS from seistan
but being able to feed seistan shows us that the problem is not at ULM,
but at LSU.

What did the telecomm folks have to say about the asymmetry seen moving
data to/from srcc.lsu.edu?

Tom