20050121: Latencies on our redundant HDS feed

>From: "Dr. Charles Graves" <address@hidden>
>Organization: SLU
>Keywords: 200501211730.j0LHUiv2002639 LDM IDD

Hi Chuck,

>I am trying to work with our local IT people to figure out why our
>HDS latencies are so large (we are talking about 1000s of seconds).
>The latencies with the other feeds are more reasonable (measured in
>seconds).

See below before rushing off to talk to you IT people.

>Here are some points I have noticed:
>
>  1) Papagyo (our feed from UNL) has small latencies
>
>  2) Our local NOAAPORT feed has latencies around 1 second (or less).

This is a function of the design of the SSEC SDI ingest system.  We
routinely see data products from our NOAAPORT ingester
(noaaport.unidata.ucar.edu) make it into the LDM queue faster than
products ingested by the SDI running on the same machine.

>  3) If I examine the data volume for HDS, I see that papagyo is feeding
>     nearly half of the HDS feed...indicating that at least some of the
>     time the papagyo feed is sending data down the pipe very quickly.

Or, older data (but from the same hour) from papagayo's queue is being
sent down to zephir because the same data ingested from your local
NOAAPORT system has been flushed out of your queue.  A quick test
of this would be to make a larger queue on zephir.  Right now, your
queue is smaller than one hour's worth of data that you are ingesting.

The other thing to remember is that the HDS stream that originates
from the Unidata NOAAPORT ingester contains grids in the OCONUS
stream, and the same stream from the SSEC SDI does not.

>zephir (our ingest macine) is running ldm-6.1.0 with a 400MB data
>queue.

I would bump the queue size to 1 GB, and see if that gets rid of the
high latency values you are seeing in HDS ingest.

>It is a FreeBSD 4.8
>platform with dual 1.2 GHz AMD processors, 1 GB of memory and two mirrored
>15,000 rpm SCSI disks.  Given this system I don't think it is lack of
>computing power (I see no evidence of slow downs in the log files).

The latencies are definitely not an indication of lack of computing
power.

>I suspect it is a network slow down somewhere even though we are using
>Internet II.  I am interested in your options.  You have dealt with this 
>much more than I have.  I also noticed the peaks in latency occur with 
>the data volume peaks...do you think this is a "packet shaping" problem?

If the problem is not spurious latencies for products that have already
been received and flushed out of your queue, it could mean that there
is some sort of packet shaping going on.

>I have an actual live person in our IT department to chat with...is there
>anything I should be asking/telling him?

I would first try changing your queue size to see if that makes a
difference.  It is best to not lead IT folks on wild goose chases.
After all, when there really is a problem, you want them to be
interested :-)

>After we get this straightened out, SLU will look into obtaining other
>feeds (i.e. CONDUIT) as well as, be willing (and able) to feed other sites!

OK.

By the way what are you folks going to do to your NOAAPORT ingest system
when the NWS switches to the DVB-S broadcast of data?

>Thanks for all of your efforts!

No worries.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.