[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[TIGGE #RST-559527]: processed oldest product in queue

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [TIGGE #RST-559527]: processed oldest product in queue
  • Date: Thu, 12 Apr 2007 09:23:57 -0600
Doug,

> Go ahead and reduce the size back to 4 GB.  We can't afford the
> reduction in volume
> transmitted.
> 
> Steve,
> 
> Do you have any suggestions on potential methods to keep the
> transmission rate
> up with a larger queue size?

Two things could cause writing to a larger product-queue to
be slower than writing to a smaller queue.

One is the whether or not the entire product-queue can still
be held in memory.  A 10 GB queue would require more than 10
GB of physical memory due to the memory usage of the LDM
processes and system overhead.  The "top" utility should be
able to tell you whether or not the product-queue is
swapping to disk (which is what you want to avoid at all
costs).

The other thing is the number of slots in the product-queue
for the data-products.  The more slots, the more time it
takes to search the relevant data-structures that reference
the slots.  Manuel's "pqmon" output

> >> ldm@tigge-ldm:~/etc> pqmon
> >> Apr 10 09:02:17 pqmon NOTE: Starting Up (7713)
> >> Apr 10 09:02:17 pqmon NOTE: nprods nfree  nempty      nbytes
> >> maxprods maxfree  minempty    maxext  age
> >> Apr 10 09:02:17 pqmon NOTE:  12535   487  963540  3974884336
> >> 122116    3086    853118    429096 846
> >> Apr 10 09:02:17 pqmon NOTE: Exiting

Indicates that his product-queue has held a maximum of 122116
data-products since the queue was created, yet the maximum
number that it can hold is 976562 (nprods + nfree + nempty).
This means that the product-queue has far more slots than it
needs and that the capacity of the product-queue is limited
by the size of the data portion of the queue (given the 
composition of the data-products being written to it).

The effect of this, however, is much less than the effect
caused by swapping the product-queue to disk because it
can't all be held in physical memory.

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: RST-559527
Department: Support IDD TIGGE
Priority: Normal
Status: Closed