Re: sunset down again?

In a previous message to me, you wrote: 

 >Pete,
 >
 >Haven't been getting data from you since 1535Z.  ldmping shows SVC
 >UNAVAILABLE.  I've switched over, but thought you should know.  Is this
 >another kernel panic?
 >
 >Regards,
 >
 >Chris

Chris,

Yes, the ldm on sunset originally went down overnight at 03:28 CDT (0828
UTC). The machine was fine, but the ldm had crashed.  Somehow the
queue got corrupted I guess, from these log messages:

Aug 02 08:05:38 5Q:sunset pqexpire[421759]: > Recycled  12549.333 kb/hr ( 
12704.439 prods per hour)
Aug 02 08:18:09 3Q:sunset waldo(feed)[458926]: h_clnt_call: 
waldo.stcloudstate.edu: COMINGSOON: time elapsed  22.603882
Aug 02 08:19:25 3Q:sunset 144.92.109.209[421786]: Que corrupt: ftbl
Aug 02 08:19:31 3Q:sunset last message repeated 44 times
Aug 02 08:19:32 3Q:sunset unidata[411930]: Que corrupt: ftbl
Aug 02 08:19:32 3Q:sunset 144.92.109.209[421786]: Que corrupt: ftbl
Aug 02 08:19:32 3Q:sunset unidata[411930]: Que corrupt: ftbl
Aug 02 08:19:32 3Q:sunset 144.92.109.209[421786]: Que corrupt: ftbl
Aug 02 08:19:38 3Q:sunset last message repeated 58 times
....

 
I remade the queue and started it back up at 1343 UTC (8:43 AM CDT)
and it ran until 15:51 or so, when it again died with a queue problem:

Aug 02 15:51:33 3Q:sunset pqexpire[539999]: assertion "status != 0" failed: 
file "pq.c", line 3993
Aug 02 16:00:07 5Q:sunset 144.92.109.209[529125]: Connection reset by peer
Aug 02 16:00:07 5Q:sunset 144.92.109.209[529125]: Disconnect
Aug 02 16:00:07 5Q:sunset thelma[550505]: Connection reset by peer
Aug 02 16:00:07 5Q:sunset thelma[550505]: Disconnect
...

I just remade the queue again, and started it up. I'll try to keep
an eye on it, if it happens again I guess I'll need to increase the
queue size, or change the frequency at which pqexpire runs or
something..

Why do things like this always happen when I am on vacation? :O
Murphy's law I guess..

Sorry for the hassle.

Pete


--
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
^ Pete Pokrandt                    V 1447  AOSS Bldg  1225 W Dayton St^
^ Systems Programmer               V Madison,         WI     53706    ^
^                                  V      address@hidden       ^
^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (Phone/voicemail) ^
^ University of Wisconsin-Madison  V       262-0166 (Fax)             ^
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+

>From address@hidden Fri Aug  4 16:17:54 2000
>Subject: sunset's ldm crashed again 1550 UTC AM


Hi all,

Sunset's ldm took a dive again this morning.. It's running again
but who knows.. I am going to compile and install ldm 5.1.2 tonight
and see if that solves things.

Sorry for the hassles..

Pete

--
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
^ Pete Pokrandt                    V 1447  AOSS Bldg  1225 W Dayton St^
^ Systems Programmer               V Madison,         WI     53706    ^
^                                  V      address@hidden       ^
^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (Phone/voicemail) ^
^ University of Wisconsin-Madison  V       262-0166 (Fax)             ^
+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+