20030715: troubleshooting high HDS latencies (cont.)

>From: Unidata Support <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200307041809.h64I9TLd018509 IDD LDM-6

Hi Dave,

As per our previous exchange, the HDS ingest test on heron has been
running for a full day.  The latency plot:

http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?HDS+heron.nmt.edu

shows that the HDS data is getting delivered without loss (at least
the latency for no product exceeds 3600 seconds, and that should mean
that no products are being lost _if_ the upstream feed host has at
least an hours worth of data in its LDM queue).  The latencies are,
however, unacceptably high.

The next test step is for you to switch your ingest of HDS from
yin.engin.umich.edu to rainbow.al.noaa.gov to see if that significantly
changes the latencies.  I don't believe that it will since the latencies
I am seeing for Unidata-Wisconsin products being sent from rainbow
to huron:

http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?UNIWISC+heron.nmt.edu

are also showing values that are higher than what they should be.

Given that the latencies for the IDS|DDPLUS products are close to zero:

http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+heron.nmt.edu

it appears that there is some sort of packet shaping being done most
likely at NMT, and that the packet shaping is configured to slow down
the receipt of "larger" products.  Unidata-Wisconsin products are the
largest ones that you are receiving ranging up to about 1.6 MB, but there
are not that many of them being sent:

http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_num_nc?UNIWISC+heron.nmt.edu

HDS products range from a few hunderd bytes up to a half a MB with an
average of between 15-20 KB and there are lots of them:

http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_num_nc?HDS+heron.nmt.edu

The IDS|DDPLUS products are typically small ranging from a few bytes up
to about 20 KB with an average of a few hundred bytes and there are a
lot of them:

http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_num_nc?IDS|DDPLUS+heron.nmt.edu

If we continue to see significant latencies after switching the HDS
feed to rainbow, it would confirm that the packet shaping is occurring
at or near NMT.  If we find this to be the case, contact needs to
be made with the NMT IT folks to see if they can change their packet
shaping to allow port 388 traffic to flow without hindrance.  Other
universities have been successful in getting this setup after
explaining exactly what the data flowing in the IDD is, and how it is
used for education and research.

I will be watching for the HDS feed switch to rainbow...

Tom

>From address@hidden Thu Jul 17 09:35:41 2003

Tom,

I just switched HDS to rainbow.

Dave