20031125: ldm/craft data latencies at NWS Central Region HQ

Karen,

>Date: Tue, 25 Nov 2003 10:04:53 -0600
>From: "Karen Cooper" <address@hidden>
>Organization: NOAA/NWS/Central Region HQ
>To: William Gery <address@hidden>
>Subject: Re: 20031124: ldm/craft data latencies at NWS Central Region HQ

The above message contained the following:

> I see Bill is getting you an account set up, so I'll just hit some of 
> the highlights.

Actually, I still have the ssh(1)-accessible account

    address@hidden

and I've logged onto ldmradar-2.  Tom and I are currently poking
around.

> There are actually two machines with a "heartbeat" service running 
> between them.  One of them is the active server, and the other is a 
> "warm" spare.  They are ldmserver1.crh.noaa.ogv and ldmserver2.crh.noaa.gov.
>
> At this time we have 20  LDM machines in forecast offices, each of them 
> ingesting a single radar stream and sending it through LDM.  The data is 
> being sent to the 8 CRAFT machines at OU.
>
> Latencies are averaging 15-17 seconds for all of the radars.  There are 
> maximums of 30 seconds (and more), but all 20 seem to be about the same.
> 
> I don't see anything unusual in the logs.  The logs are rather full, 
> because the ldmd.conf is set up to request data from a number of 
> machines that are not yet in place.  However, in testing  Bill pared the 
> file down to a single request from one of the machines with the best 
> connection (bandwidth-wise), and the latencies were the same.
> 
> pqmon shows:
> nprods: 60201
> nfree:    1
> nempty: 37454
> nbytes:  399997488
> maxprods: 60204
> maxfree:   2
> minempty: 7451
> maxext: 5584
> age:   41197
> 
> The server shows connections as LDM-6.
> 
> The rtstats is being collected on the server through localhost requests. 
> I did not set it up to send to Unidata, but that could easily be changed.

Don't change the rtstats(1) reporting yet.

> As for the LDM machines in the forecast offices.... Each of these 
> machines is ingesting a single radar of 88D Level II data.  I don't see 
> anything unusual in their ldmd.log files,  I'm not sure I understand 
> you're question about the latencies on these machines.  They are 
> ingesting the data and inserting it into the ldm queue to send to the 
> regional server.  

Would it be possible for us to log onto one or more LDM computers at the
Forecast Offices?

> Bill will have to answer your questions regarding firewalls and 
> bandwidth, and he is arranging to give you access.

We spoke with Bill today and we're running tests trying to see if
there's sufficient bandwith between the WFO-s and the RHQ.  The large
latencies (even with a single incoming feed) can be accounted for if
there's a slow network-connection somewhere between the WFO-s and the
RHQ.

We're also going to look at your TCP parameters (e.g., TCP window
size).

We'll let you know what we find.

Regards,
Steve Emmerson
Tom Yoksas