[LDM #WSJ-190258]: queue size question

John,

> We are in the process of moving to a virtual (vmware) linux system for
> down loading and creating our regional satellite/radar images and we've
> had it quit twice in two days after running normally for approximately
> 3 months since its creation.

What is "it"? The VM? The LDM?

 > I noticed that pqmon shows a low max age number (well under 3600 seconds)
> of ~130:
> 
> 20190322T150506.531572Z pqmon[6148] NOTE pqmon.c:358:main() nprods nfree  
> nempty      nbytes  maxprods  maxfree  minempty    maxext  age
> 20190322T150506.531618Z pqmon[6148] NOTE pqmon.c:466:main()  11155     1   
> 12682  1215750360     23837      275         0      3880 131
> 
> I tried increasing the product queue but when I tried to increase it to 5G
> it reverts to 700M.

How did you try? By modifying the relevant registry parameter then deleting and 
remaking the queue?

> I can set it to 3.5G, but the max age only increased t
> o around 300:

Then you'll need to make it much larger.

> 20190322T155936.243586Z pqmon[27704] NOTE pqmon.c:358:main() nprods nfree  
> nempty      nbytes  maxprods  maxfree  minempty    maxext  age
> 20190322T155936.243646Z pqmon[27704] NOTE pqmon.c:466:main()  31748     1  
> 36878  3499797704     39305        2     29321    205624 296
> 
> Here's the command I used to increase the product queue.  I was unable to
> use 3.5G and instead had to use this: regutil -u 3500000000 /queue/size

Ah! The command "regutil -s 3.5G /queue/size" could also have been used.

> The memory of the virtual linux server is 24G.

The queue should be large enough so that the minimum residence time is at least 
one hour. Obviously, the necessary size will depend on your data inflow. 
Periodically obtaining metrics and plotting them as described in the LDM 
documentation is the best way to settle on a size.

> Result of ldmadmin config:
> hostname:              mammatus.ttu.edu
> os:                    Linux
> release:               4.1.12-124.26.3.el7uek.x86_64
> ldmhome:               /home/ldm
> LDM version:           6.13.6
> PATH:                  
> /home/ldm/ldm-6.13.6/bin:/home/ldm/decoders:/home/ldm/util:/home/ldm/bin:/home/ldm/decoders:/home/ldm/util:/home/ldm/bin:/home/ldm/perl5/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/home/ldm/.local/bin:/home/ldm/bin:/home/gempak/GEMPAK6.7.0/os/linux64/bin:/home/gempak/GEMPAK6.7.0/bin
> LDM conf file:         /home/ldm/etc/ldmd.conf
> pqact(1) conf file:    /home/ldm/etc/pqact.conf
> scour(1) conf file:    /home/ldm/etc/scour.conf
> product queue:         /home/ldm/var/queues/ldm.pq
> queue size:            3500000000 bytes
> queue slots:           default
> reconciliation mode:   increase queue
> pqsurf(1) path:        /home/ldm/var/queues/pqsurf.pq
> pqsurf(1) size:        2M
> IP address:            0.0.0.0
> port:                  388
> PID file:              /home/ldm/ldmd.pid
> Lock file:             /home/ldm/.ldmadmin.lck
> maximum clients:       512
> maximum latency:       3600
> time offset:           3600
> log file:              /home/ldm/var/logs/ldmd.log
> numlogs:               7
> log_rotate:            1
> netstat:               /bin/netstat -A inet -t -n
> top:                   /bin/top -b -n 1
> metrics file:          /home/ldm/var/logs/metrics.txt
> metrics files:         /home/ldm/var/logs/metrics.txt*
> num_metrics:           4
> check time:            1
> delete info files:     0
> ntpdate(1):            /usr/sbin/ntpdate
> ntpdate(1) timeout:    5
> time servers:          ntp.ucsd.edu ntp1.cs.wisc.edu ntppub.tamu.edu 
> otc1.psu.edu timeserver.unidata.ucar.edu
> time-offset limit:     10

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: WSJ-190258
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.