[LDM #LWU-278473]: LDM 6.12.11 crashing awhile after downstream sites connect

Art,

That's exactly what I'm looking for.

Unfortunately, it appears to indicate that the problem occurred in the logging 
module as a result of a REQUEST for data being reduced in scope due to ALLOW 
entries. Because the logging module is used extensively without problems, I 
suspect that there's either a problem in the C runtime library on your system 
OR some downstream LDM is sending some very, very weird (and possibly invalid) 
REQUEST message.

Feel free to revert to an earlier version of the LDM while I investigate.

> Steve,
> 
> There was no core, so I enabled it with limit and re-ran it.  This was on a 
> different
> machine instance than the other one I sent, but the same setup.
> 
> The /var/log/messages contained this:
> 
> May 28 16:14:02 iddrs3a kernel: ldmd[22120] general protection ip:36fd247e2c 
> sp:7fff030bf8f0 error:0 in libc-2.12.so[36fd200000+18a000]
> 
> Is the below what you're looking for as a stack dump?
> 
> [ldm@iddrs3a ~]$ gdb bin/ldmd core.22120
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-75.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/local/ldm/ldm-6.12.11/bin/ldmd...done.
> [New Thread 22120]
> Missing separate debuginfo for /usr/local/ldm/ldm-6.12.11/lib/libldm.so.0
> Try: yum --enablerepo='*-debug*' install 
> /usr/lib/debug/.build-id/dd/788405b5ac471e21b3452b0e76c5e1f8355695
> Missing separate debuginfo for
> Try: yum --enablerepo='*-debug*' install 
> /usr/lib/debug/.build-id/b7/4f8095168b93495573fcca21245eb3900f05ea
> Reading symbols from /usr/local/ldm/ldm-6.12.11/lib/libldm.so.0...done.
> Loaded symbols for /usr/local/ldm/ldm-6.12.11/lib/libldm.so.0
> Reading symbols from /usr/lib64/libxml2.so.2...(no debugging symbols 
> found)...done.
> Loaded symbols for /usr/lib64/libxml2.so.2
> Reading symbols from /lib64/libz.so.1...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libz.so.1
> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols 
> found)...done.
> [Thread debugging using libthread_db enabled]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libm.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libc.so.6...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/libdl.so.2...(no debugging symbols found)...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib64/libnss_files.so.2
> Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib64/libnss_dns.so.2
> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols 
> found)...done.
> Loaded symbols for /lib64/libresolv.so.2
> Core was generated by `ldmd -I 0.0.0.0 -P 388 -M 256 -m 10800 -o 7200 -q 
> /usr/local/ldm/var/queues/ldm'.
> Program terminated with signal 11, Segmentation fault.
> #0  0x00000036fd247e2c in vfprintf () from /lib64/libc.so.6
> Missing separate debuginfos, use: debuginfo-install 
> glibc-2.12-1.149.el6_6.7.x86_64 libxml2-2.7.6-17.el6_6.1.x86_64 
> zlib-1.2.3-29.el6.x86_64
> (gdb) bt
> #0  0x00000036fd247e2c in vfprintf () from /lib64/libc.so.6
> #1  0x00000036fd26f752 in vsnprintf () from /lib64/libc.so.6
> #2  0x00007fae07b1bed4 in log_vadd (fmt=0x7fae07b1f190 "[%s:%d] Subscription 
> reduced by one or more %s: %s -> %s", args=0x7fff030c0270)
> at log.c:315
> #3  0x00007fae07b1c121 in log_add (fmt=0x7fae07b1f190 "[%s:%d] Subscription 
> reduced by one or more %s: %s -> %s") at log.c:376
> #4  0x00007fae07afd158 in logIfReduced (origSub=0x844b20, currSub=0x852c00, 
> entity=0x7fae07b1f1e1 "ALLOW entries") at forn.c:40
> #5  0x00007fae07b01fe5 in feed_or_notify (xprt=0x8426c0, want=<value 
> optimized out>, isNotifier=0, maxHereis=4294967295) at ldm_server.c:264
> #6  0x00007fae07b0243a in feedme_6_svc (feedPar=<value optimized out>, 
> rqstp=<value optimized out>) at ldm_server.c:380
> #7  0x00007fae07afa811 in ldmprog_6 (rqstp=0x7fff030c11e0, transp=0x8426c0) 
> at ldm_svc.c:99
> #8  0x00007fae07b17249 in svc_getreqsock (sock=<value optimized out>) at 
> svc.c:541
> #9  0x00007fae07b06fd1 in one_svc_run (sock=1, timeout=<value optimized out>) 
> at one_svc_run.c:87
> #10 0x00000000004064dc in handle_connection (sock=0) at ldmd.c:737
> #11 sock_svc (sock=0) at ldmd.c:796
> #12 0x0000000000406ce9 in main (ac=<value optimized out>, av=<value optimized 
> out>) at ldmd.c:1085
> 
> 
> Art
> --
> Arthur A. Person
> Research Assistant, System Administrator
> Penn State Department of Meteorology
> email:  address@hidden, phone:  814-863-1563


Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: LWU-278473
Department: Support LDM
Priority: Normal
Status: Open