[LDM #OCY-683651]: pqact crashes

Hi Bob,

Tom here.  Steve and I were doing a Meet to discuss your situation, so I
am up on what is going on...

re:
> I guess I should have included that info.  Here's the hung process:
> 
> 
> [ldm@chs-mets-02-d ~]$ ldmadmin start
> Checking the product-queue...
> Checking pqact(1) configuration-file(s)...
> /usr/local/ldm/etc/pqact.conf: syntactically correct
> Checking LDM configuration-file (/usr/local/ldm/etc/ldmd.conf)...
> 
> Starting the LDM server...

This has happened to me on machines where the setuid root bit was
not set on the lead LDM server, 'ldmd'.  The "hang" is not in 'ldmd',
but, rather in 'ldmadmin'.

re:
> And here's the ps command in another terminal:
> 
> [ldm@chs-mets-02-d ~]$ ps -ef | grep ldm
> 
> loggern+   1505      1  0 17:30 ?        00:00:08 
> /opt/CampbellSci/LoggerNet/ldp_server2 
> --config-file-name=/etc/opt/CampbellSci/ldmp.conf --run-as-daemon=true
> 
> ldm        1976      1  0 17:30 ?        00:00:16 
> /usr/local/metapp/bin/./wrapper 
> /usr/local/metapp/bin/../properties/KMLGenerator.conf 
> wrapper.syslog.ident=kmlgen 
> wrapper.pidfile=/usr/local/metapp/bin/./kmlgen.pid wrapper.daemonize=TRUE 
> wrapper.name=kmlgen wrapper.displayname=KML Generator 
> wrapper.statusfile=/usr/local/metapp/bin/./kmlgen.status 
> wrapper.java.statusfile=/usr/local/metapp/bin/./kmlgen.java.status 
> wrapper.script.version=3.5.33
> 
> ldm        1979      1  0 17:30 ?        00:00:15 
> /usr/local/metapp/bin/./wrapper 
> /usr/local/metapp/bin/../properties/MetApp.conf wrapper.syslog.ident=metapp 
> wrapper.pidfile=/usr/local/metapp/bin/./metapp.pid wrapper.daemonize=TRUE 
> wrapper.name=metapp wrapper.displayname=MetApp Service 
> wrapper.statusfile=/usr/local/metapp/bin/./metapp.status 
> wrapper.java.statusfile=/usr/local/metapp/bin/./metapp.java.status 
> wrapper.script.version=3.5.33
> 
> ldm        2061   1979  0 17:30 ?        00:00:49 
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin/java 
> -showversion -Djava.library.path=../lib -classpath 
> ../lib/wrapper.jar:../apps/MetApp.jar:../apps/MetShared.jar:../apps/FoSUtils.jar:../lib/Serialio.jar:../lib/jspComm.jar:../lib/sqljdbc42.jar:../lib/mail.jar:../lib/activation.jar:../lib/commons-dbcp2-2.1.1.jar:../lib/commons-logging-1.2.jar:../lib/commons-pool2-2.4.2.jar:../lib/commons-io-2.6.jar:../lib/jbex-v1.4.8-basic.jar
>  -Dwrapper.key=ARyPAK_-yP9gXHrT -Dwrapper.port=32000 
> -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 
> -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=1979 
> -Dwrapper.version=3.5.33 -Dwrapper.native_library=wrapper -Dwrapper.arch=x86 
> -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 
> sopp.metapp.MetAppService -propertiesDirectory ../properties
> 
> ldm        2063   1976  0 17:30 ?        00:00:57 
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin/java 
> -showversion -DServer=localhost -DPort=5680 -Djava.library.path=../lib 
> -classpath 
> ../lib/wrapper.jar:../apps/KMLGenerator.jar:../apps/MetShared.jar:../apps/FoSUtils.jar:../apps/AirepClient.jar:../lib/mail.jar:../lib/sqljdbc42.jar:../lib/activation.jar
>  -Dwrapper.key=ARyPAK_-yP9gXHrT -Dwrapper.port=32001 
> -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999 
> -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=1976 
> -Dwrapper.version=3.5.33 -Dwrapper.native_library=wrapper -Dwrapper.arch=x86 
> -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1 
> ats.generator.GeneratorService -propertiesDirectory ../properties
> 
> root      18120  18073  0 22:32 pts/0    00:00:00 sudo su - ldm
> 
> root      18122  18120  0 22:32 pts/0    00:00:00 su - ldm
> 
> ldm       18123  18122  0 22:32 pts/0    00:00:00 -bash
> 
> root      18176  18153  0 22:32 pts/1    00:00:00 sudo su - ldm
> 
> root      18178  18176  0 22:32 pts/1    00:00:00 su - ldm
> 
> ldm       18179  18178  0 22:32 pts/1    00:00:00 -bash
> 
> ldm       18323  18179  0 22:33 pts/1    00:00:00 /bin/perl 
> /usr/local/ldm/bin/ldmadmin start
> 
> ldm       18505  18323  0 22:35 pts/1    00:00:00 sh -c ldmping -l- -i 0 > 
> /dev/null 2>&1
> 
> ldm       18506  18505  0 22:35 pts/1    00:00:00 ldmping -l- -i 0
> 
> ldm       18507  18123  0 22:35 pts/0    00:00:00 ps -ef
> 
> ldm       18508  18123  0 22:35 pts/0    00:00:00 grep --color=auto ldm
> 
> The first 5 entries are local software, then the entries for logging in twice 
> (ldm is a service account, no login privileges).

The process that is missing from this list is 'ldmd'.  This suggests that either
the setuid root bit is not set on the 'ldmd' executable, or that the LDM is
installed on a file system that is not installing setuid root programs to
run as 'root'.

re:
> Here's the mount output.
> 
> [ldm@chs-mets-02-d ~]$ mount
> 
> sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
> devtmpfs on /dev type devtmpfs 
> (rw,nosuid,seclabel,size=1917884k,nr_inodes=479471,mode=755)
> securityfs on /sys/kernel/security type securityfs 
> (rw,nosuid,nodev,noexec,relatime)
> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,seclabel)
> devpts on /dev/pts type devpts 
> (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
> tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
> tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
> cgroup on /sys/fs/cgroup/systemd type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
> pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
> efivarfs on /sys/firmware/efi/efivars type efivarfs 
> (rw,nosuid,nodev,noexec,relatime)
> cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,net_prio,net_cls)
> cgroup on /sys/fs/cgroup/devices type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
> cgroup on /sys/fs/cgroup/blkio type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
> cgroup on /sys/fs/cgroup/pids type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
> cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,cpuacct,cpu)
> cgroup on /sys/fs/cgroup/memory type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
> cgroup on /sys/fs/cgroup/perf_event type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
> cgroup on /sys/fs/cgroup/cpuset type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
> cgroup on /sys/fs/cgroup/freezer type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
> cgroup on /sys/fs/cgroup/hugetlb type cgroup 
> (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
> configfs on /sys/kernel/config type configfs (rw,relatime)
> /dev/mapper/vg1-lv_root on / type xfs 
> (rw,relatime,seclabel,attr2,inode64,noquota)
> selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
> systemd-1 on /proc/sys/fs/binfmt_misc type autofs 
> (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13938)
> mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
> debugfs on /sys/kernel/debug type debugfs (rw,relatime)
> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
> /dev/sda2 on /boot type xfs 
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/sda1 on /boot/efi type vfat 
> (rw,nosuid,nodev,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
> /dev/mapper/vg1-lv_home on /home type xfs 
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_opt on /opt type xfs 
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_var on /var type xfs 
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_www on /var/www type xfs 
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_log on /var/log type xfs 
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_audit on /var/log/audit type xfs 
> (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_tmp on /tmp type xfs 
> (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,noquota)
> tmpfs on /run/user/993 type tmpfs 
> (rw,nosuid,nodev,relatime,seclabel,size=386072k,mode=700,uid=993,gid=990)
> tmpfs on /run/user/994 type tmpfs 
> (rw,nosuid,nodev,relatime,seclabel,size=386072k,mode=700,uid=994,gid=1002)
> tmpfs on /run/user/731480791 type tmpfs 
> (rw,nosuid,nodev,relatime,seclabel,size=386072k,mode=700,uid=731480791,gid=731400513)

OK, thanks.

re:
> LDM is installed in /usr/local/ldm, which I think is on the root volume (/) .

Yup, it sure looks like that is the case.

The symptoms strongly suggest that 'ldmd' is not being allowed to run with
'root' privilege (which is only needed to get port 388, and then it
returns to run as 'ldm').  To test this, please run the following and
send us the output:

ldmd -l-

If your output looks something like:

20201009T221807.932689Z ldmd[165520]                ldmd.c:main:988             
        NOTE  Starting Up (version: 6.13.12; built: Oct  9 2020 16:14:25)
20201009T221807.932826Z ldmd[165520]                priv.c:rootpriv:44          
        ERROR Operation not permitted
20201009T221807.932853Z ldmd[165520]                priv.c:rootpriv:44          
        ERROR Couldn't set effective user-ID to root's (0)
20201009T221807.932891Z ldmd[165520]                
ldmd.c:create_ldm_tcp_svc:484       ERROR Permission denied
20201009T221807.932913Z ldmd[165520]                
ldmd.c:create_ldm_tcp_svc:484       ERROR Couldn't obtain local address 
0.0.0.0:388 for server
20201009T221807.932934Z ldmd[165520]                ldmd.c:cleanup:197          
        NOTE  Exiting
20201009T221807.932959Z ldmd[165520]                ldmd.c:cleanup:256          
        NOTE  Terminating process group

it means that 'ldmd' can not run with setuid permissions, so it can't grab port
388, and then it exits.  I would think that this has something do do woth
the extra security that has been installed on the machine.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: OCY-683651
Department: Support LDM
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.