Re: 20020805: RPC Timed out error ldmping ldm problem

Mike Leuthold wrote:
> 
> >
> > It eventually stops?  Does the connection go down?  Can you give me more
> > details about this?
> >
> Here is the ldmd.log from ~9Z
> Aug 06 09:10:02 nimbus hailshaft(feed)[1354]: topo:  hailshaft.atmo.ttu.edu 
> DIFAX|FSL2|MCIDAS|IDS|DDPLUS
> Aug 06 09:10:19 nimbus striker[8371]: Timed out after 720 seconds inactivity
> Aug 06 09:10:19 nimbus striker[8371]: Disconnect
> Aug 06 09:10:34 nimbus allegan(feed)[1242]: FOUS51 KRNK 060804 /pRDFRNK: RPC: 
> Timed out (5)
> Aug 06 09:10:34 nimbus allegan(feed)[1242]: pq_sequence failed: Input/output 
> error (errno = 5)
> Aug 06 09:10:34 nimbus allegan(feed)[1242]: Exiting
> Aug 06 09:10:39 nimbus rpc.ldmd[8365]: child 1242 exited with status 1
> Aug 06 09:10:39 nimbus allegan[1468]: Connection from allegan.nr.usu.edu
> Aug 06 09:10:50 nimbus cirp[8370]: FEEDME(cirp.met.utah.edu): reclass: 
> 20020806080240.651 TS_ENDT {{NNEXRAD,  ".*"}}
> Aug 06 09:10:50 nimbus cirp[8370]: assertion "pIf(xdrs->x_op == XDR_ENCODE, 
> (tvp->tv_sec >= TS_ZERO.tv_sec && tvp->tv_usec >= TS_ZERO.tv_usec && 
> tvp->tv_sec <= TS_ENDT.tv_sec && tvp->tv_usec <= TS_ENDT.tv_usec))" failed: 
> file
> "timestamp.c", line 51
> Aug 06 09:10:56 nimbus rpc.ldmd[8365]: child 8370 terminated by signal 6
> Aug 06 09:10:56 nimbus rpc.ldmd[8365]: Killing (SIGINT) process group
> Aug 06 09:10:56 nimbus rpc.ldmd[8365]: Interrupt
> Aug 06 09:10:56 nimbus rpc.ldmd[8365]: Exiting
> Aug 06 09:10:56 nimbus hailshaft(feed)[1354]: Interrupt
> Aug 06 09:10:56 nimbus hailshaft(feed)[1354]: Exiting
> Aug 06 09:10:56 nimbus rpc.ldmd[8365]: Terminating process group
> Aug 06 09:10:56 nimbus suomildm1[8372]: Interrupt
> Aug 06 09:10:56 nimbus allegan[1468]: Interrupt
> Aug 06 09:10:56 nimbus allegan[1468]: Exiting
> Aug 06 09:10:56 nimbus suomildm1[8372]: Exiting
> Aug 06 09:10:56 nimbus hailshaft[32418]: Interrupt
> Aug 06 09:10:56 nimbus cyclone(feed)[29014]: Interrupt
> Aug 06 09:10:56 nimbus allegan[32416]: Interrupt
> Aug 06 09:10:56 nimbus striker[8371]: Interrupt
> Aug 06 09:10:56 nimbus cyclone(feed)[28884]: Interrupt
> Aug 06 09:10:56 nimbus hailshaft[32418]: Exiting
> Aug 06 09:10:56 nimbus pqbinstats[8366]: Interrupt
> Aug 06 09:10:56 nimbus sunny89[8368]: Interrupt
> Aug 06 09:10:56 nimbus pqact[8367]: Interrupt
> Aug 06 09:10:56 nimbus 128.95.89.38[8369]: Interrupt
> Aug 06 09:10:56 nimbus cyclone(feed)[29014]: Exiting
> Aug 06 09:10:56 nimbus allegan[32416]: Exiting
> Aug 06 09:10:56 nimbus striker[8371]: Exiting
> Aug 06 09:10:56 nimbus cyclone(feed)[28884]: Exiting
> Aug 06 09:10:56 nimbus pqbinstats[8366]: Exiting
> Aug 06 09:10:56 nimbus sunny89[8368]: Exiting
> Aug 06 09:10:57 nimbus 128.95.89.38[8369]: Exiting
> Aug 06 09:10:57 nimbus pqact[8367]: Exiting
> Aug 06 09:10:57 nimbus striker[8371]: mm_mtof: Couldn't riul_r_find 700006400
> 

Yuck!!  This doesn't appear to have to do with sunny89 per se.  Rather,
the assertion failure indicates that there's something wrong in the time
stamp of a product it received from cirp, causing the whole thing to
shut down.  Do you see this error much?


> >
> > I've been discussing this with Mike (our sys admin).  He was wondering
> > if a good old reboot might clear up some confusion.   Have you rebooted
> > recently?
> 
> First thing I tried of course!  Actually, both my machines that run ldm
> have this problem. (one linux, one IRIX) However, they do NOT have any
> problem talking to each other. One machine handles UNDATA, NNEXRAD, FSL2,
> the other does NMC2 from motherlode.  Thus my theory that it is a Telecom
> issue.
> 

That does make sense...

> >
> > And, if a reboot doesn't clear things up, may we log in to your machine
> > and take a look?
> Sure.
> The ldmfail in crontab is turned off since I am completely unable to feed
> from my primary and have hacked ldmd.conf to feed UNIDATA from motherlode
> for the short term.  Feel free to do whatever you with to ldmd.conf.
> 

I'm off to an appointment, but will log in with Mike ASAP, probably in
about 1.5 hours.


Anne

> 
> --
> Mike Leuthold
> Atmospheric Sciences/Institute of Atmospheric Physics
> University of Arizona
> address@hidden
> 520-621-2863

-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************