[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[LDM #OCY-683651]: pqact crashes
- Subject: [LDM #OCY-683651]: pqact crashes
- Date: Fri, 09 Oct 2020 17:02:07 -0600
Hi Bob,
Tom here. Steve and I were doing a Meet to discuss your situation, so I
am up on what is going on...
re:
> I guess I should have included that info. Here's the hung process:
>
>
> [ldm@chs-mets-02-d ~]$ ldmadmin start
> Checking the product-queue...
> Checking pqact(1) configuration-file(s)...
> /usr/local/ldm/etc/pqact.conf: syntactically correct
> Checking LDM configuration-file (/usr/local/ldm/etc/ldmd.conf)...
>
> Starting the LDM server...
This has happened to me on machines where the setuid root bit was
not set on the lead LDM server, 'ldmd'. The "hang" is not in 'ldmd',
but, rather in 'ldmadmin'.
re:
> And here's the ps command in another terminal:
>
> [ldm@chs-mets-02-d ~]$ ps -ef | grep ldm
>
> loggern+ 1505 1 0 17:30 ? 00:00:08
> /opt/CampbellSci/LoggerNet/ldp_server2
> --config-file-name=/etc/opt/CampbellSci/ldmp.conf --run-as-daemon=true
>
> ldm 1976 1 0 17:30 ? 00:00:16
> /usr/local/metapp/bin/./wrapper
> /usr/local/metapp/bin/../properties/KMLGenerator.conf
> wrapper.syslog.ident=kmlgen
> wrapper.pidfile=/usr/local/metapp/bin/./kmlgen.pid wrapper.daemonize=TRUE
> wrapper.name=kmlgen wrapper.displayname=KML Generator
> wrapper.statusfile=/usr/local/metapp/bin/./kmlgen.status
> wrapper.java.statusfile=/usr/local/metapp/bin/./kmlgen.java.status
> wrapper.script.version=3.5.33
>
> ldm 1979 1 0 17:30 ? 00:00:15
> /usr/local/metapp/bin/./wrapper
> /usr/local/metapp/bin/../properties/MetApp.conf wrapper.syslog.ident=metapp
> wrapper.pidfile=/usr/local/metapp/bin/./metapp.pid wrapper.daemonize=TRUE
> wrapper.name=metapp wrapper.displayname=MetApp Service
> wrapper.statusfile=/usr/local/metapp/bin/./metapp.status
> wrapper.java.statusfile=/usr/local/metapp/bin/./metapp.java.status
> wrapper.script.version=3.5.33
>
> ldm 2061 1979 0 17:30 ? 00:00:49
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin/java
> -showversion -Djava.library.path=../lib -classpath
> ../lib/wrapper.jar:../apps/MetApp.jar:../apps/MetShared.jar:../apps/FoSUtils.jar:../lib/Serialio.jar:../lib/jspComm.jar:../lib/sqljdbc42.jar:../lib/mail.jar:../lib/activation.jar:../lib/commons-dbcp2-2.1.1.jar:../lib/commons-logging-1.2.jar:../lib/commons-pool2-2.4.2.jar:../lib/commons-io-2.6.jar:../lib/jbex-v1.4.8-basic.jar
> -Dwrapper.key=ARyPAK_-yP9gXHrT -Dwrapper.port=32000
> -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999
> -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=1979
> -Dwrapper.version=3.5.33 -Dwrapper.native_library=wrapper -Dwrapper.arch=x86
> -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1
> sopp.metapp.MetAppService -propertiesDirectory ../properties
>
> ldm 2063 1976 0 17:30 ? 00:00:57
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.262.b10-0.el7_8.x86_64/jre/bin/java
> -showversion -DServer=localhost -DPort=5680 -Djava.library.path=../lib
> -classpath
> ../lib/wrapper.jar:../apps/KMLGenerator.jar:../apps/MetShared.jar:../apps/FoSUtils.jar:../apps/AirepClient.jar:../lib/mail.jar:../lib/sqljdbc42.jar:../lib/activation.jar
> -Dwrapper.key=ARyPAK_-yP9gXHrT -Dwrapper.port=32001
> -Dwrapper.jvm.port.min=31000 -Dwrapper.jvm.port.max=31999
> -Dwrapper.disable_console_input=TRUE -Dwrapper.pid=1976
> -Dwrapper.version=3.5.33 -Dwrapper.native_library=wrapper -Dwrapper.arch=x86
> -Dwrapper.service=TRUE -Dwrapper.cpu.timeout=10 -Dwrapper.jvmid=1
> ats.generator.GeneratorService -propertiesDirectory ../properties
>
> root 18120 18073 0 22:32 pts/0 00:00:00 sudo su - ldm
>
> root 18122 18120 0 22:32 pts/0 00:00:00 su - ldm
>
> ldm 18123 18122 0 22:32 pts/0 00:00:00 -bash
>
> root 18176 18153 0 22:32 pts/1 00:00:00 sudo su - ldm
>
> root 18178 18176 0 22:32 pts/1 00:00:00 su - ldm
>
> ldm 18179 18178 0 22:32 pts/1 00:00:00 -bash
>
> ldm 18323 18179 0 22:33 pts/1 00:00:00 /bin/perl
> /usr/local/ldm/bin/ldmadmin start
>
> ldm 18505 18323 0 22:35 pts/1 00:00:00 sh -c ldmping -l- -i 0 >
> /dev/null 2>&1
>
> ldm 18506 18505 0 22:35 pts/1 00:00:00 ldmping -l- -i 0
>
> ldm 18507 18123 0 22:35 pts/0 00:00:00 ps -ef
>
> ldm 18508 18123 0 22:35 pts/0 00:00:00 grep --color=auto ldm
>
> The first 5 entries are local software, then the entries for logging in twice
> (ldm is a service account, no login privileges).
The process that is missing from this list is 'ldmd'. This suggests that either
the setuid root bit is not set on the 'ldmd' executable, or that the LDM is
installed on a file system that is not installing setuid root programs to
run as 'root'.
re:
> Here's the mount output.
>
> [ldm@chs-mets-02-d ~]$ mount
>
> sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
> proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
> devtmpfs on /dev type devtmpfs
> (rw,nosuid,seclabel,size=1917884k,nr_inodes=479471,mode=755)
> securityfs on /sys/kernel/security type securityfs
> (rw,nosuid,nodev,noexec,relatime)
> tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,noexec,seclabel)
> devpts on /dev/pts type devpts
> (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
> tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
> tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
> cgroup on /sys/fs/cgroup/systemd type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
> pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
> efivarfs on /sys/firmware/efi/efivars type efivarfs
> (rw,nosuid,nodev,noexec,relatime)
> cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,net_prio,net_cls)
> cgroup on /sys/fs/cgroup/devices type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
> cgroup on /sys/fs/cgroup/blkio type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
> cgroup on /sys/fs/cgroup/pids type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
> cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,cpuacct,cpu)
> cgroup on /sys/fs/cgroup/memory type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
> cgroup on /sys/fs/cgroup/perf_event type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
> cgroup on /sys/fs/cgroup/cpuset type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
> cgroup on /sys/fs/cgroup/freezer type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
> cgroup on /sys/fs/cgroup/hugetlb type cgroup
> (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
> configfs on /sys/kernel/config type configfs (rw,relatime)
> /dev/mapper/vg1-lv_root on / type xfs
> (rw,relatime,seclabel,attr2,inode64,noquota)
> selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
> systemd-1 on /proc/sys/fs/binfmt_misc type autofs
> (rw,relatime,fd=30,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=13938)
> mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
> debugfs on /sys/kernel/debug type debugfs (rw,relatime)
> hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
> /dev/sda2 on /boot type xfs
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/sda1 on /boot/efi type vfat
> (rw,nosuid,nodev,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
> /dev/mapper/vg1-lv_home on /home type xfs
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_opt on /opt type xfs
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_var on /var type xfs
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_www on /var/www type xfs
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_log on /var/log type xfs
> (rw,nosuid,nodev,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_audit on /var/log/audit type xfs
> (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,noquota)
> /dev/mapper/vg1-lv_tmp on /tmp type xfs
> (rw,nosuid,nodev,noexec,relatime,seclabel,attr2,inode64,noquota)
> tmpfs on /run/user/993 type tmpfs
> (rw,nosuid,nodev,relatime,seclabel,size=386072k,mode=700,uid=993,gid=990)
> tmpfs on /run/user/994 type tmpfs
> (rw,nosuid,nodev,relatime,seclabel,size=386072k,mode=700,uid=994,gid=1002)
> tmpfs on /run/user/731480791 type tmpfs
> (rw,nosuid,nodev,relatime,seclabel,size=386072k,mode=700,uid=731480791,gid=731400513)
OK, thanks.
re:
> LDM is installed in /usr/local/ldm, which I think is on the root volume (/) .
Yup, it sure looks like that is the case.
The symptoms strongly suggest that 'ldmd' is not being allowed to run with
'root' privilege (which is only needed to get port 388, and then it
returns to run as 'ldm'). To test this, please run the following and
send us the output:
ldmd -l-
If your output looks something like:
20201009T221807.932689Z ldmd[165520] ldmd.c:main:988
NOTE Starting Up (version: 6.13.12; built: Oct 9 2020 16:14:25)
20201009T221807.932826Z ldmd[165520] priv.c:rootpriv:44
ERROR Operation not permitted
20201009T221807.932853Z ldmd[165520] priv.c:rootpriv:44
ERROR Couldn't set effective user-ID to root's (0)
20201009T221807.932891Z ldmd[165520]
ldmd.c:create_ldm_tcp_svc:484 ERROR Permission denied
20201009T221807.932913Z ldmd[165520]
ldmd.c:create_ldm_tcp_svc:484 ERROR Couldn't obtain local address
0.0.0.0:388 for server
20201009T221807.932934Z ldmd[165520] ldmd.c:cleanup:197
NOTE Exiting
20201009T221807.932959Z ldmd[165520] ldmd.c:cleanup:256
NOTE Terminating process group
it means that 'ldmd' can not run with setuid permissions, so it can't grab port
388, and then it exits. I would think that this has something do do woth
the extra security that has been installed on the machine.
Cheers,
Tom
--
****************************************************************************
Unidata User Support UCAR Unidata Program
(303) 497-8642 P.O. Box 3000
address@hidden Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage http://www.unidata.ucar.edu
****************************************************************************
Ticket Details
===================
Ticket ID: OCY-683651
Department: Support LDM
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata
inquiry tracking system and then made publicly available through the web. If
you do not want to have your interactions made available in this way, you must
let us know in each email you send to us.