[LDM #JOQ-172871]: LDM queue size

Hi Carl,

> I am with the University of Alaska Fairbanks/GINA (Geographic Information
> Network of Alaska) and we are using LDM to provide direct broadcast
> satellite products to National Weather Service offices in here Alaska.
> Everything has been working fine until recently they developed a slow
> network issue that is not resulting in considerable data loss between our
> sites.  Although the root problem is the network slowdown, analysis of our
> LDM setup suggests that our queue is should be larger.  It is currently set
> to 10 GB.  Below is a sample of output from the "pqmon -i 5" command. From
> the output the age of the oldest product in the queue is only around 1100
> seconds which is another indication that our queue is too small.

The LDM works best if the minimum residence time of the product queue is 
greater than the maximum latency parameter in the LDM registry (which is 3600 
seconds by default); otherwise, the product-queue can't guarantee duplicate 
product rejection.

The best way to discover the minimum residence time of the product queue is to 
plot its time-series using the commands "ldmadmin addmetrics" and "ldmadmin 
plotmetrics". See 
<https://www.unidata.ucar.edu/software/ldm/ldm-current/basics/configuring.html#cron>
 and 
<https://www.unidata.ucar.edu/software/ldm/ldm-current/basics/monitoring.html#metrics>
 for more information.

> In your support documents, I've read that the LDM works best when the queue
> can be kept in physical memory and we should try not to exceed 80% of
> that.

Correct -- assuming the computer is only used to relay data and not process any.

> So if we double our queue size to 20 GB, we would need at least that
> much RAM on that server as well.  Does this all seem correct?

If you increase the queue to 20 GB, then you should have at least 20 GB/.8 = 25 
GB of memory.

> $pqmon -i 5
> May 31 19:44:27 pqmon NOTE: Starting Up (20806)
> May 31 19:44:27 pqmon NOTE: nprods nfree  nempty      nbytes  maxprods 
> maxfree  minempty    maxext  age
> May 31 19:44:27 pqmon NOTE:  16317     7  179754  8633409296    196077 313    
>      0 409778760 1107
> May 31 19:44:32 pqmon NOTE:  16390     7  179681  8633475968    196077 313    
>      0 409778760 1112
> May 31 19:44:37 pqmon NOTE:  16423     7  179648  8633489672    196077 313    
>      0 409778760 1117
> May 31 19:44:42 pqmon NOTE:  16432     7  179639  8633571944    196077 313    
>      0 409778760 1122
> May 31 19:44:47 pqmon NOTE:  16523     7  179548  8633602632    196077 313    
>      0 409778760 1127
> May 31 19:44:55 pqmon NOTE:  16528     7  179543  8712167320    196077 313    
>      0 409778760 1134
> May 31 19:45:04 pqmon NOTE:  16625     7  179446  8731669232    196077 313    
>      0 409778760 1143
> May 31 19:45:09 pqmon NOTE:  16685     6  179387  8763574352    196077 313    
>      0 409778760 1148
> May 31 19:45:14 pqmon NOTE:  16708     5  179365  8779823088    196077 313    
>      0 409778760 1153
> May 31 19:45:19 pqmon NOTE:  16726     5  179347  8785962104    196077 313    
>      0 409778760 1158


Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: JOQ-172871
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.