[LDM #JVF-237504]: 6.11.4 segfault on RHEL6.4 64bit

Daryl,

[Sorry for taking so long to respond. I was in an AWIPS-II administration 
workshop all week.]

Was the LDM version 6.11.4?

The core-dump indicates that the binary is newer than the core file. This is 
bad because the stack trace can't, therefore, be trusted.

Is there any chance I could log onto Iem21 as the LDM user?

> I attempted to use 6.11.x for the first time today.
> 
> It appeared to build
> just fine on my RHEL6.4 64bit system, but it immediately segfaulted after
> startup.  I've added the log file.
> 
> The server logged message:
> 
> ldmd[18269]: segfault at 0 ip 00002b1b6f067376 sp 00007fff43fed6d0 error 6
> in libldm.so.0.0.0[2b1b6f035000+53000]
> 
> abrtd captured a core dump of the process:
> 
> [root@iem21 ccpp-2013-04-08-13:44:28-14429]# gdb /home/ldm/bin/ldmd
> coredump
> GNU gdb (GDB) Red Hat Enterprise Linux (7.2-60.el6)
> Copyright (C) 2010 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-redhat-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /home/ldm/bin/ldmd...done.
> 
> warning: exec file is newer than core file.
> 
> [New Thread 14429]
> Missing separate debuginfo for /home/ldm/lib/libldm.so.0
> Try: yum --disablerepo='*' --enablerepo='*-debug*' install
> /usr/lib/debug/.build-id/f9/eb16e0c49470b9ad4b3549ec8a258d6d86238d
> Missing separate debuginfo for /home/ldm/lib/libxml2.so.2
> Try: yum --disablerepo='*' --enablerepo='*-debug*' install
> /usr/lib/debug/.build-id/c5/78910b23d719a098d7d2b181e1545052aeb58d
> Missing separate debuginfo for
> Try: yum --disablerepo='*' --enablerepo='*-debug*' install
> /usr/lib/debug/.build-id/a6/993d9af0d108bfc4a2bbfdb176ea3288f6fd5c
> Reading symbols from /home/ldm/lib/libldm.so.0...done.
> Loaded symbols for /home/ldm/lib/libldm.so.0
> Reading symbols from /home/ldm/lib/libxml2.so.2...done.
> Loaded symbols for /home/ldm/lib/libxml2.so.2
> Reading symbols from /lib64/libdl.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libdl.so.2
> Reading symbols from /lib64/libm.so.6...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libm.so.6
> Reading symbols from /lib64/libpthread.so.0...(no debugging symbols
> found)...done.
> [Thread debugging using libthread_db enabled]
> Loaded symbols for /lib64/libpthread.so.0
> Reading symbols from /lib64/libc.so.6...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libc.so.6
> Reading symbols from /lib64/ld-linux-x86-64.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/ld-linux-x86-64.so.2
> Reading symbols from /lib64/libnss_files.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libnss_files.so.2
> Reading symbols from /lib64/libnss_dns.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libnss_dns.so.2
> Reading symbols from /lib64/libresolv.so.2...(no debugging symbols
> found)...done.
> Loaded symbols for /lib64/libresolv.so.2
> Core was generated by `ldmd -I 0.0.0.0 -P 388 -M 256 -m 3600 -o 3600 -q
> /home/ldm/var/queues/ldm.pq /h'.
> Program terminated with signal 11, Segmentation fault.
> #0  sm_append (pid=14429, protoVers=5, sockAddr=0x2b8107f5c3a4,
> desired=<value optimized out>, allowed=0x20bd160, isNotifier=133541888,
> isPrimary=0)
> at uldb.c:1225
> 1225          eps->size = nbytes;
> Missing separate debuginfos, use: debuginfo-install
> glibc-2.12-1.107.el6.x86_64
> (gdb) bt full
> #0  sm_append (pid=14429, protoVers=5, sockAddr=0x2b8107f5c3a4,
> desired=<value optimized out>, allowed=0x20bd160, isNotifier=133541888,
> isPrimary=0)
> at uldb.c:1225
> prodSpec = 0x20bd1e0
> pattern = <value optimized out>
> nbytes = <value optimized out>
> entrySize = 88
> segment = 0x2b8107f5b000
> i = <value optimized out>
> entry = 0x2b8107f5c358
> epc = <value optimized out>
> eps = <value optimized out>
> numProdSpecs = 2
> prodSpecsSize = <value optimized out>
> #1  sm_addUpstreamLdm (pid=14429, protoVers=5, sockAddr=0x2b8107f5c3a4,
> desired=<value optimized out>, allowed=0x20bd160, isNotifier=133541888,
> isPrimary=0) at uldb.c:1278
> status = 0
> sm = 0x2b81078c59b0
> #2  sm_add (pid=14429, protoVers=5, sockAddr=0x2b8107f5c3a4,
> desired=<value optimized out>, allowed=0x20bd160, isNotifier=133541888,
> isPrimary=0)
> at uldb.c:1409
> sub = 0x20bd160
> status = <value optimized out>
> #3  uldb_addProcess (pid=14429, protoVers=5, sockAddr=0x2b8107f5c3a4,
> desired=<value optimized out>, allowed=0x20bd160, isNotifier=133541888,
> isPrimary=0)
> at uldb.c:1982
> sub = 0x0
> status = <value optimized out>
> #4  0x00002b8107680dcb in forn_5_svc (want=0x7fffcd45c100,
> rqstp=0x7fffcd45c6a0, ident=0x2b81076a2d1f "(feed)", doit=0x2b8107681960
> <feed5_sqf>)
> at forn5_svc.c:401
> theReply = {code = OK, ldm_replyt_u = {dpktdp = 0x0, signaturep =
> 0x0, alternatep = 0x0, newclssp = 0x0}}
> hc = {state = 749,
> remote =
> "\001\000\000\000H\300E\315\377\177\000\000\000\000\000\000\000\000\000\000
> \332e\a\201+\000\000@\300E\315\377\177\000\000h\300E\315\377\177\000\000\310\326e\a\201+\000\000\000\000\000\000\000\000\000\000\071\320>~",
> prog = 227547323962, vers = 0, prot = 0, addr = {sin_family = 0,
> sin_port = 0, sin_addr = {s_addr = 1}, sin_zero =
> "4\000\000\000\000\000\000"}, pmap_clnt = 0x1, rpcerr = {ru = {RE_errno =
> 124114632,
> RE_why = 124114632, RE_vers = {low = 47833174890184, high =
> 0}, RE_lb = {s1 = 47833174890184, s2 = 0}}, re_status = 124115488},
> port = 49104, clnt = 0x100000000, h_timeo = 0, begin = {tv_sec =
> 47833174891040, tv_usec = 140736637288432}, elapsed = {tv_sec = 34227772,
> tv_usec = 140736637288456},
> errmsg =
> "\206K(\373\001\000\000\000\300\353\t\002\000\000\000\000}Qf\a\201+\000\000\024\000\000\000\000\000\000\000þi\a\201+\000\000\300\353\t\---Type
> <return> to continue, or q <return> to quit---
> 002\000\000\000\000\024\000\000\000\000\000\000\000P\274\v\002\000\000\000\000\000\301E\315\377\177\000\000\240\306E\315\377\177\000\000\240\306E\315\377\177\000\000\220Vh\a\201+\000\000\355&j\a\201+\000\000\340\314h\a\201+\000\000\366\322i\a\201+\000\000\370;f\a\201+\000\000\310\326e\a\201+\000\000\020\300E\315\377\177\000\000\377\377\377\377\000\000\000\000P\274\v\002\000\000\000\000\001\000\000\000\000\000\000\000\330\065%",
> '\000' <repeats 13 times>"\355,
> &j\a\201+\000\000\340\314h\a\201+\000\000\000\340e\a\201+\000\000\320\337\340\372\064\000\000\000\001",
> '\000' <repeats 15 times>"\360,
> \351\t\002\000\000\000\000\370;f\a\201+\000\000(\301E\315\377\177\000"}
> rpc_stat = <value optimized out>
> status = <value optimized out>
> mt = TV_GT
> lastsent = {tv_sec = 47833175053968, tv_usec = 227547367061}
> now = {tv_sec = 140736637288704, tv_usec = 140736637290144}
> keepalive_interval = 300
> remote = 0x2b81078c5640
> uldbSub = 0x0
> downAddr = <value optimized out>
> #5  0x00002b8107684aa0 in ldmprog_5 (rqstp=0x7fffcd45c6a0,
> transp=0x209e980) at ldm5_svc.c:109
> argument = {hereis_5_arg = {info = {arrival = {tv_sec =
> 1365446433, tv_usec = 22227},
> signature =
> "\377\377\377\177\000\000\000\000?B\017\000\000\000\000", origin = 0x2
> <Address 0x2 out of bounds>, feedtype = 34323328,
> seqno = 0, ident = 0x0, sz = 0}, data = 0x0}, feedme_5_arg =
> {from = {tv_sec = 1365446433, tv_usec = 22227}, to = {tv_sec = 2147483647,
> tv_usec = 999999}, psa = {psa_len = 2, psa_val =
> 0x20bbb80}}, hiya_5_arg = {from = {tv_sec = 1365446433, tv_usec = 22227},
> to = {
> tv_sec = 2147483647, tv_usec = 999999}, psa = {psa_len = 2,
> psa_val = 0x20bbb80}}, notification_5_arg = {arrival = {tv_sec =
> 1365446433,
> tv_usec = 22227}, signature =
> "\377\377\377\177\000\000\000\000?B\017\000\000\000\000", origin = 0x2
> <Address 0x2 out of bounds>,
> feedtype = 34323328, seqno = 0, ident = 0x0, sz = 0},
> notifyme_5_arg = {from = {tv_sec = 1365446433, tv_usec = 22227}, to = {
> tv_sec = 2147483647, tv_usec = 999999}, psa = {psa_len = 2,
> psa_val = 0x20bbb80}}, comingsoon_5_arg = {infop = 0x51630f21, pktsz =
> 22227},
> blkdata_5_arg = {signaturep = 0x51630f21, pktnum = 22227, data =
> {dbuf_len = 2147483647, dbuf_val = 0xf423f <Address 0xf423f out of
> bounds>}}}
> result = <value optimized out>
> xdr_argument = 0x2b8107685690 <xdr_prod_class>
> xdr_result = 0x2b81076853b0 <xdr_ldm_replyt>
> local = 0x2b810768cce0 <feedme_5_svc>
> procName = 0x2b81076a26ed "FEEDME"
> #6  0x00002b8107699249 in svc_getreqsock (sock=<value optimized out>) at
> svc.c:541
> s = <value optimized out>
> why = <value optimized out>
> stat = <value optimized out>
> msg = {rm_xid = 1365666557, rm_direction = CALL, ru = {RM_cmb =
> {cb_rpcvers = 2, cb_prog = 300029, cb_vers = 5, cb_proc = 4, cb_cred = {
> oa_flavor = 0, oa_base = 0x7fffcd45c190 "\001", oa_length
> = 0}, cb_verf = {oa_flavor = 0, oa_base = 0x7fffcd45c320 "", oa_length =
> 0}},
> RM_rmb = {rp_stat = 2, ru = {RP_ar = {ar_verf = {oa_flavor =
> 300029, oa_base = 0x5 <Address 0x5 out of bounds>, oa_length = 4},
> ---Type <return> to continue, or q <return> to quit---
> ar_stat = SUCCESS, ru = {AR_versions = {low =
> 140736637288848, high = 0}, AR_results = {where = 0x7fffcd45c190 "\001",
> proc = 0}}},
> RP_dr = {rj_stat = 300029, ru = {RJ_versions = {low = 5,
> high = 4}, RJ_why = AUTH_TOOWEAK}}}}}}
> prog_found = <value optimized out>
> low_vers = <value optimized out>
> high_vers = <value optimized out>
> r = {rq_prog = 300029, rq_vers = 5, rq_proc = 4, rq_cred =
> {oa_flavor = 0, oa_base = 0x7fffcd45c190 "\001", oa_length = 0},
> rq_clntcred = 0x7fffcd45c4b0 "P)f\a\201+", rq_xprt = 0x209e980}
> xprt = 0x209e980
> cred_area =
> "\001\000\000\000\201+\000\000\000\000\000\000\000\000\000\000\006\000\000\000\000\000\000\000?
> \373\064\000\000\000\000\313\t\002\000\000\000\000\000\004\000\000\000\000\000\000
> \307E\315\377\177\000\000\000\000\000\000\000\000\000\000\002\000\000\000\000\000\000\000\225F\341\372\064\000\000\000\240\023Y\373\064\000\000\000@\026Y\373\064\000\000\000\002",
> '\000' <repeats 15 times>"\240,
> \023Y\373\064\000\000\000\200\300\005\002\000\000\000\000\000\004\000\000\000\000\000\000\000\313\t\002\000\000\000\000\000\004\000\000\000\000\000\000ux\325\a\201+\000\000\330\313E\315\377\177\000\000\023^\325\a\201+\000\000\000\000\000\000\064\000\000\000\065\345\340\372\064\000\000\000\270\002\000\000\000\000\000\000\001",
> '\000' <repeats 15 times>"\320,
> \302E\315\377\177\000\000\002\000\000\000\v\000\000\000
> \303E\315\377\177\000\000\000\313\t\002\000\000\000\000"...
> #7  0x00002b8107689281 in one_svc_run (sock=1, timeout=<value optimized
> out>) at one_svc_run.c:86
> readFds = {fds_bits = {2, 0 <repeats 15 times>}}
> before = {tv_sec = 1365446668, tv_usec = 444911}
> selectStatus = 1
> canonicalTimeout = {tv_sec = 60, tv_usec = 0}
> selectTimeout = {tv_sec = 59, tv_usec = 999997}
> fds = {fds_bits = {2, 0 <repeats 15 times>}}
> #8  0x0000000000406553 in handle_connection (sock=0) at ldmd.c:714
> TIMEOUT = 60
> raddr = {sin_family = 2, sin_port = 38356, sin_addr = {s_addr =
> 2276285068}, sin_zero = "\000\000\000\000\000\000\000"}
> len = 16
> xp_sock = 1
> pid = <value optimized out>
> xprt = 0x209e980
> status = 1
> remote = 0x2b81078c5640
> #9  sock_svc (sock=0) at ldmd.c:773
> ready = <value optimized out>
> readfds = {fds_bits = {1, 0 <repeats 15 times>}}
> stimeo = {tv_sec = 5, tv_usec = 712792}
> width = 1
> #10 0x0000000000406c86 in main (ac=<value optimized out>, av=<value
> optimized out>) at ldmd.c:1047
> ---Type <return> to continue, or q <return> to quit---
> pqfname = 0x7fffcd45df4c "/home/ldm/var/queues/ldm.pq"
> sock = <value optimized out>
> status = <value optimized out>
> doSomething = 1
> locIpAddr = 0
> ldmPort = 388
> 
> I reverted to 6.10.1 and it appears to be happily chugging along again.
> 
> daryl
> 
> --
> /**
> * Daryl Herzmann
> * Assistant Scientist -- Iowa Environmental Mesonet
> * http://mesonet.agron.iastate.edu
> */
> 
Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: JVF-237504
Department: Support LDM
Priority: High
Status: Closed