This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Pete, > Looks like updating to CentOS 6.3 and recompiling had no effect. I just > restarted using 6.3 and again, the CPU usage on individual ldmd > processes is very high (50-90%) and data is moving at a crawl. > > I do notice that in 6.11.3 the CPU utilization is almost entirely > consumed by system, rather than user context. That is very odd. We don't see that here at all: our LDM 6.11.3 server handles about 88 downstream connections with a load average around 1 to 2. > Here's what a 'top' looks like on 6.11.2 vs 6.11.3 > > 2/15/2013 > 6.11.3 > > top - 14:51:08 up 20 min, 2 users, load average: 21.46, 17.09, 11.79 > Tasks: 575 total, 17 running, 558 sleeping, 0 stopped, 0 zombie > Cpu0 : 1.6%us, 95.1%sy, 0.0%ni, 3.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu1 : 1.7%us, 88.3%sy, 0.0%ni, 7.0%id, 0.0%wa, 0.0%hi, 3.0%si, > 0.0%st > Cpu2 : 1.7%us, 97.7%sy, 0.0%ni, 0.7%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu3 : 2.0%us, 96.7%sy, 0.0%ni, 1.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu4 : 1.7%us, 95.4%sy, 0.0%ni, 2.6%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0%st > Cpu5 : 1.7%us, 97.7%sy, 0.0%ni, 0.7%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu6 : 1.7%us, 95.3%sy, 0.0%ni, 3.0%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu7 : 1.7%us, 93.4%sy, 0.0%ni, 5.0%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu8 : 1.7%us, 97.0%sy, 0.0%ni, 1.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu9 : 1.7%us, 98.0%sy, 0.0%ni, 0.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu10 : 1.7%us, 97.7%sy, 0.0%ni, 0.7%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu11 : 1.0%us, 98.0%sy, 0.0%ni, 1.0%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu12 : 1.3%us, 94.1%sy, 0.0%ni, 4.6%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu13 : 1.0%us, 92.7%sy, 0.0%ni, 6.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu14 : 1.0%us, 98.4%sy, 0.0%ni, 0.7%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu15 : 1.3%us, 98.3%sy, 0.0%ni, 0.3%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 32874844k total, 19404024k used, 13470820k free, 39280k buffers > Swap: 32767984k total, 0k used, 32767984k free, 17950924k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 28750 ldm 20 0 23.0g 872m 871m R 81.6 2.7 0:59.36 ldmd > 28752 ldm 20 0 23.0g 927m 926m R 80.3 2.9 1:04.70 ldmd > 28810 ldm 20 0 23.0g 154m 152m R 79.6 0.5 0:05.12 ldmd > 28749 ldm 20 0 23.0g 847m 846m R 78.7 2.6 1:03.10 ldmd > 28782 ldm 20 0 23.0g 443m 441m R 73.7 1.4 0:26.03 ldmd > 28746 ldm 20 0 23.0g 865m 864m R 72.0 2.7 1:00.12 ldmd > 28754 ldm 20 0 23.0g 867m 866m R 72.0 2.7 0:59.15 ldmd > 28808 ldm 20 0 23.0g 187m 185m R 69.7 0.6 0:07.03 ldmd > 28753 ldm 20 0 23.0g 884m 882m R 69.4 2.8 0:59.83 ldmd > 28807 ldm 20 0 23.0g 170m 169m R 68.4 0.5 0:07.60 ldmd > 28813 ldm 20 0 23.0g 86m 84m R 54.5 0.3 0:01.65 ldmd > 28812 ldm 20 0 23.0g 81m 80m R 52.9 0.3 0:01.60 ldmd > 28814 ldm 20 0 23.0g 75m 73m R 38.3 0.2 0:01.16 ldmd > 28282 ldm 20 0 23.0g 2876 1720 S 35.7 0.0 0:56.41 ldmd > 28281 ldm 20 0 23.0g 3388 2224 S 35.0 0.0 0:59.65 ldmd > 28294 ldm 20 0 23.0g 14m 12m S 35.0 0.0 1:00.19 ldmd > 28302 ldm 20 0 23.0g 3592 2404 S 35.0 0.0 0:59.76 ldmd > 28278 ldm 20 0 23.0g 20m 18m S 34.7 0.1 1:00.39 ldmd > 28300 ldm 20 0 23.0g 30m 28m S 34.7 0.1 1:01.31 ldmd > 28301 ldm 20 0 23.0g 3720 2528 S 34.4 0.0 0:59.61 ldmd > 28280 ldm 20 0 23.0g 32m 31m S 34.0 0.1 1:00.48 ldmd > 28296 ldm 20 0 23.0g 13m 12m S 34.0 0.0 0:58.95 ldmd > 28279 ldm 20 0 23.0g 3932 2776 S 33.7 0.0 1:00.65 ldmd > 28288 ldm 20 0 23.0g 13m 12m S 33.7 0.0 0:59.04 ldmd > 28290 ldm 20 0 23.0g 14m 12m S 33.7 0.0 0:59.94 ldmd Very high system loads, indeed. > > > 6.11.2 > top - 15:14:45 up 44 min, 3 users, load average: 1.21, 2.22, 4.88 > Tasks: 580 total, 1 running, 579 sleeping, 0 stopped, 0 zombie > Cpu0 : 0.7%us, 15.0%sy, 0.0%ni, 84.4%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu1 : 1.0%us, 15.3%sy, 0.0%ni, 83.1%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0%st > Cpu2 : 1.0%us, 14.8%sy, 0.0%ni, 83.8%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0%st > Cpu3 : 1.3%us, 16.0%sy, 0.0%ni, 82.3%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0%st > Cpu4 : 1.0%us, 15.3%sy, 0.0%ni, 83.1%id, 0.0%wa, 0.0%hi, 0.7%si, > 0.0%st > Cpu5 : 1.0%us, 16.8%sy, 0.0%ni, 81.8%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0%st > Cpu6 : 1.0%us, 16.6%sy, 0.0%ni, 82.5%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu7 : 0.3%us, 15.7%sy, 0.0%ni, 83.7%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0%st > Cpu8 : 0.3%us, 15.6%sy, 0.0%ni, 84.1%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu9 : 0.7%us, 15.2%sy, 0.0%ni, 84.1%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu10 : 0.7%us, 15.2%sy, 0.0%ni, 84.1%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu11 : 0.7%us, 15.3%sy, 0.0%ni, 84.1%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu12 : 0.7%us, 15.2%sy, 0.0%ni, 84.1%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu13 : 0.7%us, 15.2%sy, 0.0%ni, 83.8%id, 0.0%wa, 0.0%hi, 0.3%si, > 0.0%st > Cpu14 : 0.7%us, 15.3%sy, 0.0%ni, 84.1%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Cpu15 : 1.0%us, 14.6%sy, 0.0%ni, 84.4%id, 0.0%wa, 0.0%hi, 0.0%si, > 0.0%st > Mem: 32874844k total, 24002200k used, 8872644k free, 34976k buffers > Swap: 32767984k total, 0k used, 32767984k free, 20543600k cached > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND > 38385 ldm 20 0 23.0g 552m 551m S 2.3 1.7 0:52.59 ldmd > 38405 ldm 20 0 23.0g 642m 640m S 2.3 2.0 0:52.01 ldmd > 38424 ldm 20 0 23.0g 558m 556m S 2.3 1.7 0:54.13 ldmd > 38433 ldm 20 0 23.0g 549m 547m S 2.3 1.7 0:53.95 ldmd > 38497 ldm 20 0 23.0g 526m 525m S 2.3 1.6 0:48.50 ldmd > 38507 ldm 20 0 23.0g 1.7g 1.7g S 2.3 5.5 2:31.76 ldmd > 38510 ldm 20 0 23.0g 2.1g 2.1g S 2.3 6.7 2:54.38 ldmd > 38872 ldm 20 0 23.0g 168m 167m S 2.3 0.5 0:11.16 ldmd > 39034 ldm 20 0 23.0g 176m 174m S 2.3 0.5 0:03.17 ldmd > 38386 ldm 20 0 23.0g 1.2g 1.2g S 2.0 3.8 1:44.74 ldmd > 38387 ldm 20 0 23.0g 572m 570m S 2.0 1.8 0:53.63 ldmd > 38388 ldm 20 0 23.0g 1.2g 1.2g S 2.0 3.8 1:45.35 ldmd > 38391 ldm 20 0 23.0g 644m 642m S 2.0 2.0 0:52.13 ldmd > 38394 ldm 20 0 23.0g 579m 578m S 2.0 1.8 0:52.05 ldmd > 38400 ldm 20 0 23.0g 1.1g 1.1g S 2.0 3.6 1:28.46 ldmd > 38401 ldm 20 0 23.0g 2.1g 2.1g S 2.0 6.7 2:57.24 ldmd > 38403 ldm 20 0 23.0g 1.2g 1.2g S 2.0 3.8 0:52.49 ldmd > 38404 ldm 20 0 23.0g 526m 524m S 2.0 1.6 0:52.45 ldmd > 38408 ldm 20 0 23.0g 539m 537m S 2.0 1.7 0:51.08 ldmd > 38414 ldm 20 0 23.0g 632m 630m S 2.0 2.0 0:48.43 ldmd > 38415 ldm 20 0 23.0g 558m 556m S 2.0 1.7 0:53.70 ldmd > 38419 ldm 20 0 23.0g 526m 525m S 2.0 1.6 0:51.41 ldmd > 38422 ldm 20 0 23.0g 563m 561m S 2.0 1.8 0:52.63 ldmd > 38423 ldm 20 0 23.0g 558m 556m S 2.0 1.7 0:53.92 ldmd > 38426 ldm 20 0 23.0g 563m 561m S 2.0 1.8 0:52.66 ldmd I can't imagine what could be causing LDM 6.11.3 to have much higher system loads than LDM 6.11.2 on your system. > My queue size is 24 Gb, my system RAM is 32 Gb, could that have anything > to do with it? That seems close. Your product-queue should fit in memory. We like to have about twice as much memory as the size of the product-queue but have used less. Our backend server has about 74 GB of memory and a 30 GB queue. > Steve, I can get you ldm access to idd.aos.wisc.edu if you want to poke > around at all. That would help. > I also can send ldmd.log when running 6.11.3 vs 6.11.2. If I get on the system, then I can look at the log file directly. Would you mind if I switched between the LDM-s (or is the system operational)? Regards, Steve Emmerson Ticket Details =================== Ticket ID: VOV-751174 Department: Support LDM Priority: Normal Status: Closed