[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #QYR-744357]: increase Q size

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [LDM #QYR-744357]: increase Q size
  • Date: Fri, 27 Jul 2012 10:05:40 -0600
Hi Virginia!

> We are running LDM 6.4.2 with a 2000M Queue on a system with 3.8GB and 23G
> available in /home/ldm/data
> 
> Can we increase the Q size to 4000M
> We are not using /dev/shm  for /home/ldm/data/ldm .pq  It is just a regular
> file.
> 
> Developer said:
> I have already tried to increased the queue-size but got an "Illegal size"
> error.
> and also changed Q size in ldmadmin.pl.conf

You've actually asked two questions. The first one is whether or not the size 
of the queue can be greater than the amount of physical memory. While we 
recommend that you have sufficient physical memory to contain the entire queue 
(for performance reasons) we have successfully run instances of the LDM in 
which this wasn't true. I'm afraid that we can't predict what will happen in 
your particular case, however, because it depends on too many variables. If the 
system isn't operationally critical, then I suggest that you try it and see 
(after accumulating sufficient metrics so that you'll know if there's a 
problem, of course).

The second question is why you got an "Illegal size" error when you tried to 
increase the queue-size to 4 GB. This is, undoubtedly, because the type "off_t" 
-- which is used to specify offsets in the mmap() system call -- is 32 bits in 
length rather than 64 (a 32-bit off_t can't specify all possible offsets into a 
4 GB file).

Beginning with LDM version 6.9.8, the building step attempts to maximize this 
"programming environment" constraint. You could try a later version of the LDM 
or try building the version you have on a 64-bit system. The output of the 
configure(1) script will tell you if you succeeded.

If you do upgrade to a later version, then be sure to read carefully the 
CHANGE_LOG file.

> Here is our system memory and disk:
> 
> 
> $ free
> total           used             free       shared
> buffers     cached
> Mem:       3986384    3962304      24080          0     203652    3396300
> -/+ buffers/cache:     362352    3624032
> Swap:      3068372          0    3068372
> 
> $ df -h
> Filesystem            Size  Used Avail Use% Mounted on
> /dev/sda8             981M  342M  590M  37% /
> /dev/sda2              99M   26M   69M  27% /boot
> /dev/sda9              44G   19G   23G  45% /home
> none                  2.0G     0  2.0G   0% /dev/shm
> /dev/sda7             981M   17M  915M   2% /tmp
> /dev/sda5             9.7G  1.5G  7.7G  17% /usr
> /dev/sda3             9.7G  380M  8.8G   5% /var
> 
> tgp48 current Q use
> $ pqmon -i 5
> Jul 27 13:59:39 pqmon NOTE: Starting Up (25470)
> Jul 27 13: pqmon NOTE:             nprods nfree  nempty      nbytes
> maxprods  maxfree  minempty    maxext  age
> 
> 
> Jul 27 13:59:44 pqmon NOTE:  45248   286  442747  1998778440    417495
> 477     70785     86472 2242
> Jul 27 13:59:49 pqmon NOTE:  45259   305  442717  1997955432    417495
> 477     70785     84848 2243
> Jul 27 13:59:54 pqmon NOTE:  45233   311  442737  1996617864    417495
> 477     70785    163520 2241
> Jul 27 13:59:59 pqmon NOTE:  45266   306  442709  1997801688    417495
> 477     70785    161312 2243
> Jul 27 14:00:04 pqmon NOTE:  45278   304  442699  1997952512    417495
> 477     70785    116808 2243

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: QYR-744357
Department: Support LDM
Priority: Normal
Status: Closed