[LDM #XTK-117583]: optimum queue settings for mostly non IDD LDM

Rodger,

Basically, you want the queue to be large enough so that the minimum 
residence-time is *at least* the value of the "maximum latency" parameter. The 
residence-time is the amount of time that a product spends in the queue before 
being deleted to make room for a new product. The "maximum latency" parameter 
is the amount of time before "now" that an LDM uses to compute the "from" time 
when initiating a connection to an upstream LDM. It's also a rejection 
threshold: products that were created earlier than "now" minus "maximum 
latency" will be rejected and not inserted into the queue. As a consequence, a 
correctly configured LDM will reject a data-product that duplicates one already 
in the queue.

The capacity of the queue is based on two parameters: the queue "size" and the 
number of slots. The queue size is the maximum number of bytes that the queue 
can contain. The number of slots is the maximum number of products that the 
queue can contain.

Your queue has a maximum latency parameter of 22 seconds. This means that you 
can be offline for that long without loosing any data. That's not very long. It 
was computed by the ldmadmin(1) utility based on the minimum residence-time of 
the queue. If you want to increase that safety margin, then you'll need to 
increase the capacity of the queue from 500 MB and around 9800 slots.

Assuming you'd like a minimum residence-time of an hour, you could set the 
"/server/max-latency" parameter to 3600 seconds (regutil -u 3600 
/server/max-latency) and the "/reconciliation-mode" parameter to "increase 
queue" (regutil -s 'increase queue' /reconciliation-mode) and then execute the 
command "ldmadmin vetqueuesize". This is dangerous, however, because 
extrapolation from such a small queue is prone to error. A less error-prone 
method would be to iterate up to the final capacity by first setting the 
reconciliation-mode parameter to "increase queue" and then repeatedly doubling 
the maximum latency parameter and executing the command "ldmadmin vetqueuesize" 
until the desired maximum latency is reached.

Incidentally, the "ldmadmin vetqueuesize" command won't do anything until the 
queue is full (i.e., until a product has been deleted) -- so don't be surprised 
if it doesn't do anything for a while after increasing the maximum latency 
parameter.

One thing to note: the LDM system works best if the entire queue can be kept in 
physical memory. So be careful if and when the queue size approaches the amount 
of physical memory (we try not to exceed about 80% of physical memory).

Are you using the metrics-collecting and plotting capabilities of the LDM 
system (e.g., crontab(1)-executed "ldmadmin addmetrics" and the "ldmadmin 
plotmetrics" command). It's a good way to keep tabs on the LDM in general and 
the product-queue in particular.

Contact me if you have any questions.

> I've read the documentation on tuning the product queue parameters
> with pqmon and read some of the support inquiries. I think that most
> of this is predicated on being IDD sites where latency is the primary
> concern. We have a dish and ingest the majority of what we need.  We are
> using noaaportIngester for ingest. The only IDD feeds we have are for
> the lightning data and from FSL for Madis products. I've run 'ldmadmin
> vetqueuesize' at various times and let it set the latency. Our latency
> is obviously not an issue since we are ingesting directly from NOAAPort
> and not feeding any downstream sites.
> 
> I ran pqmon at 5 second intervals for the past few hours. Here are
> the stats:
> 
> nprods ~9800
> nfree <10 on average with a peak of 18
> nempty single digits or 0 much of the time but did spike at about 5000 one 
> time
> nbytes ~270000000 - 500000000
> maxprods 9802
> maxfree 43
> minempty 0
> maxext ~10000 - 425000000
> age 25 - 200
> 
> Here is our config:
> 
> ldmadmin config
> 
> hostname:              data.awis.com
> os:                    Linux
> release:               4.4.0-31-generic
> ldmhome:               /usr/ldm
> LDM version:           6.13.2
> PATH:                  
> /usr/ldm/ldm-6.13.2/bin:.:/usr/bin:/usr/local/bin:/loc/local/bin:/bin:/loc/wxp_linux:/usr/ldm/bin
> LDM conf file:         /usr/ldm/etc/ldmd.conf
> pqact(1) conf file:    /usr/ldm/etc/pqact.conf
> scour(1) conf file:    /usr/ldm/etc/scour.conf
> product queue:         /dataspare/ldm/queues/ldm.pq
> queue size:            500M bytes
> queue slots:           default
> reconciliation mode:   decrease maximum latency
> pqsurf(1) path:        /dataspare/ldm/queues/pqsurf.pq
> pqsurf(1) size:        2M
> IP address:            0.0.0.0
> port:                  388
> PID file:              /usr/ldm/ldmd.pid
> Lock file:             /usr/ldm/.ldmadmin.lck
> maximum clients:       256
> maximum latency:       22
> time offset:           22
> log file:              /dataspare/ldm/logs/ldmd.log
> numlogs:               7
> log_rotate:            1
> netstat:               /bin/netstat -A inet -t -n
> top:                   /usr/bin/top -b -n 1
> metrics file:          /dataspare/ldm/logs/metrics.txt
> metrics files:         /dataspare/ldm/logs/metrics.txt*
> num_metrics:           4
> check time:            1
> delete info files:     0
> ntpdate(1):            /usr/sbin/ntpdate
> ntpdate(1) timeout:    5
> time servers:          ntp.ucsd.edu ntp1.cs.wisc.edu ntppub.tamu.edu 
> otc1.psu.edu timeserver.unidata.ucar.edu
> time-offset limit:     10
> 
> (Note: this server is not sending rstats as it is in test mode).
> 
> So, given that fact that we have only 2 low volume IDD feeds and get
> everything else directly from our dish and don't feed any other sites,
> how do our numbers look? If needed, we can increase our queue as it
> resides on a separate 1TB drive. The drive is a Seagate SSHD (solid
> state hybrid) that operates at near SSD speeds.
> 
> As always, your insights are appreciated!

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: XTK-117583
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.