20041129: monster latencies from atm.geo.nsf.gov (cont.)

>From: "Patrick L. Francis" <address@hidden>
>Organization: Bowling Green State University
>Keywords: 200411291316.iATDGZlI028344 IDD latency

Hi Patrick,

First, my earlier email should have suggested that you were requesting
everything from the UNIDATA feed from atm.geo.nsf.gov.  This means that
your request line would look like:

request UNIDATA     ".*"    atm.geo.nsf.gov

'UNIDATA' is a compound feed mnemonic.  It is actually IDS|DDPLUS|HDS|UNIWISC.

>traceroutes into bgnet are throttled... and as mentioned i examined
>the internal network for difficulties. For example, we are receiving
>data from OU (stokes) with minimal delay.
>
>http://my.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NNEXRAD+unidata.bgsu.edu
>
>however,  over the same time period IDD data from atm shows monster latencies.
>
>http://my.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?HDS+unidata.bgsu.edu

Yes, but the volume of NNEXRAD data you are receiving from OU is small:

http://my.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?NNEXRAD+unidata.bgsu.edu

and the volume of the request from atm.geo.nsf.gov would be much higher
_if_ your request is for everything in UNIDATA.  I just did a quick
check of the connection messages in the ~ldm/logs/ldmd.log.* files on
atm.geo.nsf.gov, and I see that my assumption is correct:

Nov 28 03:01:34 atm.geo.nsf.gov unidata(feed)[3309]: up6.c:331: Starting 
Up(6.0.14/6): 20041128030119.632 TS_ENDT {{UNIDATA,  ".*"}}
Nov 28 03:01:34 atm.geo.nsf.gov unidata(feed)[3309]: topo:  unidata.bgsu.edu 
UNIDATA

>this would seem to indicate a problem outside of the BG Network.

Not necessarily.  If there is a some sort of volume limitation on a
feed (like through "packet shaping" being done on a BGSU firewall or
through a problem on a router), and if your request for data from
atm.geo.nsf.gov is like:

request UNIDATA     ".*"    atm.geo.nsf.gov

then the requested volume of data from atm.geo.nsf.gov is up to 60
times more than the NEXRAD Level III data you are getting from OU.  The
fact that the latency plots for IDS|DDPLUS, HDS, and UNIWISC are
identical (not similar, _identical_) is what tipped me off that me that
the feed request to atm.geo.nsf.gov is for everything from UNIDATA.

>Also notice
>the significant drop in latencies to unidata.bgsu from atm.geo since the time
>of my first message.

This drop is dramatic indeed, and the traceroute from atm.geo.nsf.gov
to unidata.bgsu.edu indicates that the problem at the BGSU machine
mentioned previously has disappeared:

/local/ldm% traceroute unidata.bgsu.edu
traceroute to unidata.bgsu.edu (129.1.112.220), 30 hops max, 40 byte packets
 1  colima.nsf.gov (198.181.231.1)  1 ms  1 ms  1 ms
 2  arlg-nsf.maxgigapop.net (206.196.177.137)  2 ms  2 ms  1 ms
 3  dcne-so3-0-0.maxgigapop.net (206.196.178.42)  2 ms  10 ms  18 ms
 4  abilene-rtr.maxgigapop.net (206.196.177.2)  25 ms *  4 ms
 5  nycmng-washng.abilene.ucaid.edu (198.32.8.84)  6 ms  6 ms  9 ms
 6  chinng-nycmng.abilene.ucaid.edu (198.32.8.82)  27 ms  48 ms  28 ms
 7  iplsng-chinng.abilene.ucaid.edu (198.32.8.77)  249 ms  252 ms  241 ms
 8  clmbq-r2-po0-0.bb.oar.net (192.88.192.133)  34 ms  35 ms  35 ms
 9  dytnq-r3-at1-0s16.bb.oar.net (192.88.191.101)  37 ms  37 ms  37 ms
10  bgsu2-r1-at2-0s103.cpe.oar.net (192.88.192.174)  62 ms  62 ms  61 ms
11  129.1.8.252 (129.1.8.252)  62 ms  67 ms  62 ms
12  129.1.8.125 (129.1.8.125)  63 ms  62 ms  63 ms
13  * * *
14  * * *
15  unidata.bgsu.edu (129.1.112.220)  76 ms  106 ms  93 ms

The problem you were seeing earlier was not at or "near"
atm.geo.nsf.gov since other machines feeding the same data as
unidata.bgsu.edu from atm.geo.nsf.gov showed no appreciable latencies
over the time where you were observing problems.  The machine at the
University of North Carolina-Ashville is a good example:

IDS|DDPLUS latency:
http://my.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+storm2.atms.unca.edu

HDS latency:
http://my.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?HDS+storm2.atms.unca.edu

UNIWISC latency:
http://my.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?UNIWISC+storm2.atms.unca.edu

>my concerns are that since other universities feed off of us, usually with 
>great success, any delays from the top would snowball.

Your concern is a valid one!

In a way, it is almost too bad (but not really :-) that the slowdown is
not still occurring.  If it was, I was going to ask you to split your
single UNIDATA feed request to three separate ones:

change:

request WMO     ".*"    atm.geo.nsf.gov

to:

request IDS|DDPLUS  ".*"    atm.geo.nsf.gov
request HDS         ".*"    atm.geo.nsf.gov
request UNIWISC     ".*"    atm.geo.nsf.gov

After a change like this, I would expect that even though there was
still some sort of problem, the latency for the IDS|DDPLUS feed would
fall to levels you were seeing in your NNEXRAD request from OU since
the volume in the IDS|DDPLUS feed is equivalent to the volume of
NNEXRAD data you are requesting.  I would expect that the latencies for
UNIWISC would also decrease, but the ones for HDS would probably not
since it comprises the bulk of the volume in the combined request.

>Hope you had a wonderful thanksgiving :)

It was grand, thanks.  I hope yours was as much fun as mine!!

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.

>From address@hidden  Mon Nov 29 16:37:18 2004

>Please let us know the results of your 'traceroute'.

forgot these earlier... will reply to your last message tomorrow ...
it's almost by bedtime  :)

------------------[ from weather.bgsu.edu ]------------------------
[pfrancis@weather pfrancis]$ /usr/sbin/traceroute atm.geo.nsf.gov
traceroute to atm.geo.nsf.gov (198.181.231.53), 30 hops max, 38 byte packets
  1  router1-112 (129.1.112.252)  0.307 ms  0.255 ms  0.232 ms
  2  10.9.2.1 (10.9.2.1)  0.525 ms  0.431 ms  0.468 ms
  3  10.9.10.12 (10.9.10.12)  0.354 ms  0.297 ms  0.237 ms
  4  129.1.8.126 (129.1.8.126)  0.533 ms  0.454 ms  0.442 ms
  5  gw-bgsu (129.1.8.253)  1.604 ms  1.558 ms  2.110 ms
  6  dytnq-r3-at2-0s6.bb.oar.net (192.88.192.173)  25.119 ms  25.761 
ms  25.201 ms
  7  clmbq-r2-at2-0s13.bb.oar.net (192.88.191.102)  28.993 ms  27.149 
ms  27.254 ms
  8  abilene-IPLSng.ohio-gigapop.oar.net (192.88.192.134)  33.395 
ms  32.685 ms  33.648 ms
  9  chinng-iplsng.abilene.ucaid.edu (198.32.8.76)  44.590 ms  38.635 
ms  48.238 ms
10  nycmng-chinng.abilene.ucaid.edu (198.32.8.83)  58.707 ms  84.901 
ms  62.806 ms
11  washng-nycmng.abilene.ucaid.edu (198.32.8.85)  61.339 ms  61.714 
ms  64.282 ms
12  dcne-abilene-oc48.maxgigapop.net (206.196.177.1)  61.604 ms  60.610 
ms  62.031 ms
13  arlg-so3-1-0.maxgigapop.net (206.196.178.41)  61.171 ms  60.459 
ms  61.173 ms
14  nsf-rtr.maxgigapop.net (206.196.177.138)  61.883 ms  61.537 ms  61.390 ms
15  atm.geo.nsf.gov (198.181.231.53)  61.655 ms  61.956 ms  62.473 ms
[pfrancis@weather pfrancis]$

---------------------[ from unidata.bgsu.edu ]--------------------------
[pfrancis@unidata pfrancis]$ /usr/sbin/traceroute atm.geo.nsf.gov
traceroute to atm.geo.nsf.gov (198.181.231.53), 30 hops max, 38 byte packets
  1  router1-112 (129.1.112.252)  0.286 ms  0.233 ms  0.225 ms
  2  10.9.2.1 (10.9.2.1)  0.493 ms  0.404 ms  0.390 ms
  3  10.9.10.12 (10.9.10.12)  0.293 ms  0.246 ms  0.229 ms
  4  129.1.8.126 (129.1.8.126)  0.395 ms  0.385 ms  0.387 ms
  5  gw-bgsu (129.1.8.253)  3.244 ms  2.377 ms  2.759 ms
  6  dytnq-r3-at2-0s6.bb.oar.net (192.88.192.173)  25.752 ms  24.496 
ms  26.233 ms
  7  clmbq-r2-at2-0s13.bb.oar.net (192.88.191.102)  28.529 ms  27.223 
ms  28.123 ms
  8  abilene-IPLSng.ohio-gigapop.oar.net (192.88.192.134)  32.394 
ms  33.580 ms  36.879 ms
  9  chinng-iplsng.abilene.ucaid.edu (198.32.8.76)  42.701 ms  44.354 
ms  72.150 ms
10  nycmng-chinng.abilene.ucaid.edu (198.32.8.83)  64.432 ms  69.913 
ms  57.960 ms
11  washng-nycmng.abilene.ucaid.edu (198.32.8.85)  61.733 ms  67.642 
ms  70.044 ms
12  dcne-abilene-oc48.maxgigapop.net (206.196.177.1)  60.852 ms  62.229 
ms  62.586 ms
13  arlg-so3-1-0.maxgigapop.net (206.196.178.41)  60.994 ms  61.581 
ms  60.775 ms
14  nsf-rtr.maxgigapop.net (206.196.177.138)  62.236 ms  64.352 ms  62.337 ms
15  atm.geo.nsf.gov (198.181.231.53)  70.127 ms  70.208 ms  73.690 ms
[pfrancis@unidata pfrancis]$
------------------------------------------------------------------------------

>From address@hidden  Tue Nov 30 07:20:25 2004

Pat,
         I have given a higher traffic priority in the packet shaper to all 
traffic to and from weather.bgsu.edu and unidata.bgsu.edu.

Let me know if this improves anything.

Thanks
-Mike