[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #XNO-331025]: Queue corruption with very large queues with LDM 6.13.16

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [LDM #XNO-331025]: Queue corruption with very large queues with LDM 6.13.16
  • Date: Tue, 28 Dec 2021 11:14:39 -0700
Hi Gilbert,

> 1. /root% regutil | grep latency
> /server/max-latency : 3600

In that case, the 1.2 TB recommendation from "ldmadmin vetqueuesize" is way off 
base -- unless you're also getting *a lot* of non-IDD, non-NOAAPort data.

> 2.  By "writing to disk" do you mean a downstream LDM process writing to the
> product-queue or a pqact(1) process writing to disk?
> 
> Pqact process writing to the SSD, receiving the feeds from upstream
> server(s).

If and when the NEXRAD2 and NEXRAD3 products are no longer being written to 
disk by pqact(1), see if the products exist in the product-queue by using the 
commands

    pqcat -v -f NEXRAD2 -p <nexrad2-pattern> -o 99999 >/dev/null
    pqact -v -f NEXRAD3 -p <nexrad3-pattern> -o 99999 >/dev/null

where <nexrad2-pattern> and <nexrad3-pattern> are the relevant patterns from 
the relevant pqact(1) configuration-file(s) for the associated products,

If they're in the product-queue, then see if there are any relevant log 
messages from the associated pqact(1) processes -- especially warnings about 
processing the oldest product in the queue because this indicates that the 
associated pqact(1) process can't keep up with the data.

> 3.  Victor Gensini has reported that his server, atlas.niu.edu, has crashed
> > (completely!) twice over the last few days. Stopping LDM now takes several
> > minutes to do.
> >
> > I haven't heard of this. Where's this report?
> 
> You heard it here first. :-)  However, he found the reason why his server
> was crashing; upon further investigation, was a symptom of another issue
> going on.

Not surprised.

> That said, it still takes him a while to shut down his LDM. And I
> am wondering if 6.13.17.3 fixes that delay issue.

It shouldn't.

We've noticed that sometimes the LDM takes a while to terminate -- especially 
on VM-s -- while other times it doesn't. I don't know why.

--Steve

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: XNO-331025
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.