[LDM #KWB-851882]: pqact issues reading pq file

Kevin,

> Let me try to work through the questions here:
> 
> 1.   Yes, noaaportIngester is launched via an EXEC line in the ldmd.conf.
> This is the same file we also EXEC pqact -e and edexBridge.

Good. Then the pqact(1) process will be notified immediately when a new 
data-product is inserted into the product-queue.

> 2.  I can appreciate a need for numbers! ;)  The highest I've seen since
> 0400gmt today is 1147s (this doesn't count the number of products that
> never were acted upon).  Also, it appears to be about the same time every
> day ... 0500gmt,

I agree that 1147 seconds is too much latency.

> 3.  Indeed.  Starting at 0514 a number of them "Processed oldest product in
> queue: 356.194 s"  Jumps to over 1000 s a bit later.  What are these
> indicating?

Conceptually, data-products are added to the head-end of the product-queue and 
move through the queue until they are "pushed off" the tail-end of the queue in 
order to make room for new products. Ideally, pqact(1) processes are processing 
products that are near the head-end of the queue. If a pqact(1) process "gets 
behind", however, then the location of the data-product currently being 
processed will drift toward the tail-end of the queue. When a pqact(1) process 
notices that it just processed the oldest product in the queue (the one at the 
tail-end) then it logs the "Processed oldest product..." message. This 
indicates that the pqact(1) process couldn't keep up with the rate at which 
data-products are arriving. Thus, either the queue needs to be larger or the 
pqact(1) process needs to be faster.

> 4.  No, I am not.  It appears I may have missed a setup step there.  I'll
> add it on this instance if it will help.

Be advised that you'll have to have gnuplot installed in order to use the 
command "ldmadmin plotmetrics".

> 5.  The server has 6GB of physical RAM.  Our pq is 1G and we're configured
> with --disable-max-size.

Huh. Why bother with "--disable-max-size"?

> Interesting theory on the edexBridge.   We do have -log on every FILE in
> the pqact.conf.

If the downstream LDM process is logging the reception of data-products, then 
what does a comparison of the downstream LDM timestamps and the pqact(1) 
timestamps show? That the pqact(1) process is way behind?

> I do not mind if this is made public.  Thanks.

Great! This is a good discussion. Thanks.

> Kevin
> 
> --
> *Kevin P Johnson, RHCE™*
> Principal Engineer with Honors
> AWIPS Lead Systems Engineer
> Mission Operations & Services
> *Raytheon Company*
> 
> +301-495-2257   (office)
> +301-244-8575   (cell)
> address@hidden
> address@hidden
> 
> 
> 8401 Colesville Road
> Suite 800
> Silver Spring, MD 20910
> www.raytheon.com
> 
> 
> 
> *This message contains information that may be confidential and privileged.
> Unless you are the addressee (or authorized to receive mail for the
> addressee), you should not use, copy or disclose to anyone this message or
> any information contained in this message. If you have received this
> message in error, please so advise the sender by reply e-mail and delete
> this message. Thank you for your cooperation.*

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: KWB-851882
Department: Support LDM
Priority: Normal
Status: Closed