Re: [conduit] Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu starting the last day or two.

Those traceroutes clearly show the issue is on the last 2 hops or inside NOAA.

Ray

On Friday, February 19, 2016 2:39pm, "Gerry Creager - NOAA Affiliate" 
<address@hidden> said:

> _______________________________________________
> conduit mailing list
> address@hidden
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/ If I were the conspiracy theory 
> type, I
> might think to blame the
> balkanization of all paths to the internet from NOAA sites via the Trusted
> Internet Connection stuff. But I'm just hypothesizing. Or, they could be
> running distro on overloaded VMs.
> 
> gerry
> 
> On Fri, Feb 19, 2016 at 1:28 PM, Patrick L. Francis <address@hidden>
> wrote:
> 
>>
>>
>> Art / Pete etc. al. J
>>
>>
>>
>> There seems to be a consistency is potential packet loss from no matter
>> which route is taken into ncep… so whoever you are communicating with,
>> you
>> might have them investigate 140.90.111.36… reference the previous graphic
>> shown and this new one here:
>>
>>
>>
>> http://drmalachi.org/files/ncep/ec2-ncep.png
>>
>>
>>
>> if you are unfamiliar with amazon ec2 routing, the first.. twenty
>> something or so hops are just internal to amazon, and they don’t jump
>> outside until you hit the internet2 hops, which then jump to gigapop, and
>> from there to noaa internal.. so since this amazon box is in ashburn,
>> physically it’s close, and has limited interruptions until that point..
>>
>>
>>
>> the same hop causes more severe problems from my colo boxes, which are
>> hurricane electric direct, which means that in those cases jumping from
>> hurricane electric to 140.90.111.36 has “severe” problems (including
>> packet
>> loss) while jumping from amazon to I2 to gigapop to 140.90.111.36 also
>> encounters issues, but not as severe..
>>
>>
>>
>> hopefully this may help J  Happy Friday J
>>
>>
>>
>> cheers,
>>
>>
>>
>> --patrick
>>
>>
>>
>> -------------------------------------------------------
>>
>> Patrick L. Francis
>>
>> Vice President of Research & Development
>>
>>
>>
>> Aeris Weather
>>
>>
>>
>> http://aerisweather.com/
>>
>> http://modelweather.com/
>>
>>
>>
>> http://facebook.com/wxprofessor/
>>
>>
>>
>> --------------------------------------------------------
>>
>>
>>
>> *From:* address@hidden [mailto:
>> address@hidden] *On Behalf Of *Arthur A Person
>> *Sent:* Friday, February 19, 2016 1:57 PM
>> *To:* Pete Pokrandt <address@hidden>
>> *Cc:* Bentley, Alicia M <address@hidden>; Michael Schmidt <
>> address@hidden>; address@hidden <
>> address@hidden>; _NCEP.List.pmb-dataflow <
>> address@hidden>; Daes Support <address@hidden>
>> *Subject:* Re: [conduit] Large CONDUIT latencies to UW-Madison
>> idd.aos.wisc.edu starting the last day or two.
>>
>>
>>
>> Pete,
>>
>>
>>
>> We've been struggling with latencies for months to the point where I've
>> been feeding gfs 0p25 from NCEP and the rest from Unidata... that is, up
>> untl Feb 10th.  The afternoon of the 10th, our latencies to NCEP dropped to
>> what I consider "normal", an average maximum latency of about 30 seconds.
>> Our networking folks and NCEP have been trying to identify what this
>> problem was, but as far as I know, no problem has been identified or action
>> taken.  So, it appears it's all buried in the mysteries of the internet.
>> I've switched data collection back to NCEP at this point, but I'm on the
>> edge of my seat waiting to see if it reverts back to the old behavior...
>>
>>
>>
>>                   Art
>>
>>
>> ------------------------------
>>
>> *From: *"Pete Pokrandt" <address@hidden>
>> *To: *"Carissa Klemmer - NOAA Federal" <address@hidden>,
>> "Arthur A Person" <address@hidden>, "_NCEP.List.pmb-dataflow" <
>> address@hidden>
>> *Cc: *"address@hidden" <address@hidden>,
>> "Michael Schmidt" <address@hidden>, "Bentley, Alicia M" <
>> address@hidden>, "Daes Support" <address@hidden>
>> *Sent: *Friday, February 19, 2016 12:20:20 PM
>> *Subject: *Large CONDUIT latencies to UW-Madison idd.aos.wisc.edu
>> starting the last day or two.
>>
>> All,
>>
>>
>>
>> Not sure if this is on my end or somewhere upstream, but the last several
>> runs my CONDUIT latencies have been getting huge to the point where we are
>> losing data.
>>
>>
>>
>> I did stop my ldm the other day to add in an alternate feed for Gilbert at
>> allisonhous.com, not sure if that pushed me over a bandwidth limit, or by
>> reconnecting we got hooked up to a different remote ldm, or taking a
>> different path, that shot the latencies up.
>>
>>
>>
>> Seems to be really only CONDUIT, none of our other feeds show this kind of
>> latency.
>>
>>
>>
>> Still looking into things locally, but wanted make people aware. I just
>> rebooted idd.aos.wisc.edu, will see if that helps at all.
>>
>>
>>
>> Here's an ldmping and traceroute from idd.aos.wisc.edu to
>> conduit.ncep.noaa.gov.
>>
>>
>>
>> [ldm@idd ~]$ ldmping conduit.ncep.noaa.gov
>>
>> Feb 19 17:16:08 INFO:      State    Elapsed Port   Remote_Host
>> rpc_stat
>>
>> Feb 19 17:16:08 INFO: Resolving conduit.ncep.noaa.gov to 140.90.101.42
>> took 0.00486 seconds
>>
>> Feb 19 17:16:08 INFO: RESPONDING   0.115499  388   conduit.ncep.noaa.gov
>>
>>
>>
>>
>>
>> traceroute to conduit.ncep.noaa.gov (140.90.101.42), 30 hops max, 60 byte
>> packets
>>
>>  1  r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3)
>>  0.760 ms  0.954 ms  0.991 ms
>>
>>  2  internet2-ord-600w-100G.net.wisc.edu (144.92.254.229)  18.119 ms
>>  18.123 ms  18.107 ms
>>
>>  3  et-10-0-0.107.rtr.clev.net.internet2.edu (198.71.45.9)  27.836 ms
>>  27.852 ms  27.838 ms
>>
>>  4  et-11-3-0-1276.clpk-core.maxgigapop.net (206.196.177.4)  37.363 ms
>>  37.363 ms  37.345 ms
>>
>>  5  noaa-i2.demarc.maxgigapop.net (206.196.177.118)  38.051 ms  38.254 ms
>>  38.401 ms
>>
>>  6  140.90.111.36 (140.90.111.36)  118.042 ms  118.412 ms  118.529 ms
>>
>>  7  140.90.76.69 (140.90.76.69)  41.764 ms  40.343 ms  40.500 ms
>>
>>  8  * * *
>>
>>  9  * * *
>>
>> 10  * * *
>>
>>
>>
>> Similarly to ncepldm
>>
>>
>>
>> [ldm@idd ~]$ ldmping ncepldm4.woc.noaa.gov
>>
>> Feb 19 17:18:40 INFO:      State    Elapsed Port   Remote_Host
>> rpc_stat
>>
>> Feb 19 17:18:40 INFO: Resolving ncepldm4.woc.noaa.gov to 140.172.17.205
>> took 0.001599 seconds
>>
>> Feb 19 17:18:40 INFO: RESPONDING   0.088901  388   ncepldm4.woc.noaa.gov
>>
>> ^C
>>
>>
>>
>> [ldm@idd ~]$ traceroute ncepldm4.woc.noaa.gov
>>
>> traceroute to ncepldm4.woc.noaa.gov (140.172.17.205), 30 hops max, 60
>> byte packets
>>
>>  1  r-cssc-b280c-1-core-vlan-510-primary.net.wisc.edu (144.92.130.3)
>>  0.730 ms  0.831 ms  0.876 ms
>>
>>  2  internet2-ord-600w-100G.net.wisc.edu (144.92.254.229)  18.092 ms
>>  18.092 ms  18.080 ms
>>
>>  3  ae0.3454.core-l3.frgp.net (192.43.217.223)  40.196 ms  40.226 ms
>>  40.256 ms
>>
>>  4  noaa-i2.frgp.net (128.117.243.11)  40.970 ms  41.012 ms  40.996 ms
>>
>>  5  2001-mlx8-eth-1-2.boulder.noaa.gov (140.172.2.18)  42.780 ms  42.778
>> ms  42.764 ms
>>
>>  6  mdf-rtr-6.boulder.noaa.gov (140.172.6.251)  40.869 ms  40.922 ms
>>  40.946 ms
>>
>>  7  * * *
>>
>>  8  * * *
>>
>>
>>
>>
>>
>>
>>
>> Pete
>>
>>
>>
>>
>>
>>
>>
>> --
>> Pete Pokrandt - Systems Programmer
>> UW-Madison Dept of Atmospheric and Oceanic Sciences
>> 608-262-3086  - address@hidden
>>
>>
>>
>>
>>
>> --
>>
>> Arthur A. Person
>> Research Assistant, System Administrator
>> Penn State Department of Meteorology
>> email:  address@hidden, phone:  814-863-1563
>>
>> _______________________________________________
>> conduit mailing list
>> address@hidden
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
> 
> 
> 
> --
> Gerry Creager
> NSSL/CIMMS
> 405.325.6371
> ++++++++++++++++++++++
> “Big whorls have little whorls,
> That feed on their velocity;
> And little whorls have lesser whorls,
> And so on to viscosity.”
> Lewis Fry Richardson (1881-1953)
> 


_______________________________________________
conduit mailing list
address@hidden
For list information or to unsubscribe, visit: 
http://www.unidata.ucar.edu/mailing_lists/