[LDM #VEN-289761]: LDM tuning

Hi Roger,

> It's been a few years since we last tried to "tune" our LDM config. With
> the recent LDM software updates, we thought the exercise would be
> useful. Let me first give you a overview of our setup. We operate two
> LDM servers, 'gina' and 'data'. Both are running Ubuntu and have only 8GB
> of RAM. A 6GB tmpfs is setup on each server to hold the ldm queue. Both
> are on our 1GB LAN. We have a NOAAPort dish. The signal is split and
> feeds both gina and data. The gina server is requesting the lightning
> and MADIS feeds. The feeds are then available to data for ingesting.
> We basically have data set up as a redundant LDM server.
> 
> The gina ldmd.conf has noaaportIngester EXEC entries for 224.0.1.1 through
> .8 (we don't care about the new GOES-R channels). Except for lightning
> and MADIS products, everything comes in on our own dish and not via IDD
> feeds. The ldmd.conf splits the incoming traffic into 5 pqact entries
> with their own pqact.conf. One each for MADIS, NAPLN, NEXRAD, and GOES
> products. All other feedtypes use a generic pqact.conf.
> 
> Here are some specific questions about the gina setup:
> 
> 1. Since we only have the lightning and MADIS internet feeds, do we
> really need to have a max latency and time offset of 3600 as suggested?
> According to rstats, we only get about 300 FSL2 and 82 LIGHTNING
> products per hour. The gina server is rarely rebooted and only the
> limited lightning and MADIS feeds are impacted. Is there a reason to
> not change to a value of 1800?

Among other reasons, the LDM product-queue exists to give you time to fix 
problems without losing data, e.g., fixing a networking problem, replacing a 
switch, etc. How large the queue should be is up to you. Just ensure that the 
minimum residence time of the queue is at least the maximum allowable latency 
or you'll risk missing products.

> 2. rstats shows that gina gets over 273,000 products per hour. Assuming
> that we change the latency/offset values to 1800, would it make sense
> to create the queue with half the expected slots or 136,500 slots?

In general, yes. A better means of sizing the queue, however, is to collect 
metrics via the "ldmadmin addmetrics" command and plot them via the "ldmadmin 
plotmetrics" command. Recall that there are two "size" parameters: the number 
of bytes the queue can hold and the number of products. Seeing time-series 
plots of the amount of data and the number of products can help you adjust 
these values.

> The server data has the same basic ldmd.conf and pqact entries as gina.
> It obtains the lightning and MADIS feeds from gina. All other feedtypes
> are via the split feed from our dish.
> 
> Here are some specific questions about the data setup:
> 
> 1. Should the max latency and time offset values on data also be set to
> 1800?

Yes, though it won't make any difference if gina is the only upstream LDM 
server to data.

> 2. Same question about the number of slots as with gina.

Because they receive the same data, the same values should apply.

> 3. data has a Seagate ST1000DX001 hybrid SSHD drive with a TB of space.
> What about creating the queue on this drive and letting it store up to
> an hour of products.

If data is feeding downstream LDM-s, then the amount of time that they can be 
offline to fix a problem will be determined by data's queue; otherwise, there's 
no advantage to data having a larger queue than gini.

> I've tried to find answers to these questions by searching through the
> support archives, but everything seems to be geared for IDD sites. Our
> setup is very different having our own dish and only getting two low
> volume supplemental feeds. We aren't feeding anyone else. We are doing
> the metrics collection on both servers. I've looked at the graphics that
> are generated, but I don't know how to interpret them. I couldn't find
> any tutorial or documentation on how to apply these graphics.

For this issue, the important plots are

    Age: This plots the how long the oldest product in the queue has been in 
the queue. The queue should be large enough so that the minimum age of a 
product in the queue is greater than the maximum allowable latency parameter in 
the registry

    Product Count: This plots the number of products in the queue. If it 
"flatlines" at the top, then the queue is limited by the number of products it 
can hold.

    Queue Size: This plots the amount of data in the queue. It it "flatlines" 
at the top, then the queue is limited by the amount of data it can hold.

> As always, thanks in advance for all you guys do.

You're most welcome.

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: VEN-289761
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.