20040319: Logs indicate Latency problem: LSU to ULM (cont.)

>From: address@hidden
>Organization: ULM
>Keywords: 200403081533.i28FXNrV006465 IDD latency

Adam,

>I shut off the nexrad feed that i was injesting.  As of today, i can't make 
>since of the latency graphs.

I just happened to be looking at the ingest stats for your machine, and
saw that they are very bad.  The IDS|DDPLUS feed, the ofeed with the
lowest volume, shows just how bad the feed is at various points throughout
the day:

http://my.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+tornado.geos.ulm.edu

The trace shows that there are substantial periods during the day when
the latency ramps up to > 3000 seconds (at times up to 8000).  This can
be caused by network saturation, equipment malfunction (noisy routers,
etc.), or throughput limiting (packet shaping).

>I have double checked with our network people 
>and they can't seem to find a problem on our side.  I will keep checking.

It would seem unlikely that both LSU and OU would have problems feeding
ULM unless the problem were at/near the ULM side.  Do your network
folks keep statistics on the volume of data flowing through individual
routers?  If yes, are they seeing anything during those periods when
the IDD data flow to ULM slows?

>As a note, is it correct for me to have seistan as the PRIMARY and stokes as 
>the SECONDARY?  I have my pqact set up sortof like this.
>
>request  (FEEDTYPE#1) ".*"    seistan.srcc.lsu.edu   PRIMARY
>request  (FEEDTYPE#1) ".*"    stokes.metr.ou.edu     SECONDARY

Yes, this is correct.  If they were both PRIMARY, you would be getting
twice the volume of data flowing to the machine.  The SECONDARY request
to OU will not consume much bandwidth since only the metadata for each
product will be transferred.

>And the same for the other feed types except for the nexrad and nogaps.  For 
>the nexrad i have ALL from seistan as primary and the surrounding 5 from 
>stokes  secondary.

I don't see any statistics being reported for the NNEXRAD feed.

>If this is not correrct then please correct me.  It has been set like that for
>about 4-5 months now so if it had only started acouple of thursdays ago i 
>don't see that it could be a problem.

Your requests are OK.  The latencies are showing a consistent network
problem that is limiting your capability for getting data through the
IDD.

If you feel confident that there is no problem at ULM and want to see
if the feeds can improve by switching to a different upstream host(s),
then you are enabled to receive all non-restricted feeds on the Unidata
test machine, emo.unidata.ucar.edu.  You could change all feed
requests except NLDN and FNMOC to emo and comment out the redundant
(SECONDARY) feeds to see if your reception improves.  If it does, we
will need to examine the network paths from OU to ULM and LSU to ULM to
see where the problem may be occurring.  If it does not improve, it
will reinforce my notion that the network problem lies at or near ULM.

Whatever you decide to do, please keep us informed so we can help.

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.