Re: sunset down again?

Pete Pokrandt wrote:

In a previous message to me, you wrote:

 >Pete,
 >
 >Haven't been getting data from you since 1535Z.  ldmping shows SVC
 >UNAVAILABLE.  I've switched over, but thought you should know.  Is this
 >another kernel panic?
 >
 >Regards,
 >
 >Chris

Chris,

Yes, the ldm on sunset originally went down overnight at 03:28 CDT (0828
UTC). The machine was fine, but the ldm had crashed.  Somehow the
queue got corrupted I guess, from these log messages:

Aug 02 08:05:38 5Q:sunset pqexpire[421759]: > Recycled  12549.333 kb/hr ( 12704.439 prods per hour)
Aug 02 08:18:09 3Q:sunset waldo(feed)[458926]: h_clnt_call: waldo.stcloudstate.edu: COMINGSOON: time elapsed  22.603882
Aug 02 08:19:25 3Q:sunset 144.92.109.209[421786]: Que corrupt: ftbl
Aug 02 08:19:31 3Q:sunset last message repeated 44 times
Aug 02 08:19:32 3Q:sunset unidata[411930]: Que corrupt: ftbl
Aug 02 08:19:32 3Q:sunset 144.92.109.209[421786]: Que corrupt: ftbl
Aug 02 08:19:32 3Q:sunset unidata[411930]: Que corrupt: ftbl
Aug 02 08:19:32 3Q:sunset 144.92.109.209[421786]: Que corrupt: ftbl
Aug 02 08:19:38 3Q:sunset last message repeated 58 times
....
 

I remade the queue and started it back up at 1343 UTC (8:43 AM CDT)
and it ran until 15:51 or so, when it again died with a queue problem:

Aug 02 15:51:33 3Q:sunset pqexpire[539999]: assertion "status != 0" failed: file "pq.c", line 3993
Aug 02 16:00:07 5Q:sunset 144.92.109.209[529125]: Connection reset by peer
Aug 02 16:00:07 5Q:sunset 144.92.109.209[529125]: Disconnect
Aug 02 16:00:07 5Q:sunset thelma[550505]: Connection reset by peer
Aug 02 16:00:07 5Q:sunset thelma[550505]: Disconnect
...

I just remade the queue again, and started it up. I'll try to keep
an eye on it, if it happens again I guess I'll need to increase the
queue size, or change the frequency at which pqexpire runs or
something..

Why do things like this always happen when I am on vacation? :O
Murphy's law I guess..

Sorry for the hassle.

Pete

--
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
^ Pete Pokrandt                    V 1447  AOSS Bldg  1225 W Dayton St^
^ Systems Programmer               V Madison,         WI     53706    ^
^                                  V      address@hidden       ^
^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (Phone/voicemail) ^
^ University of Wisconsin-Madison  V       262-0166 (Fax)             ^
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+

Hi Pete,

I'm just wondering if perhaps you've developed a bad block on your disk or some other disk problem.   Have things been working correctly for a while until now?   How old is your disk?  Most systems have a utility to scan for bad blocks - you might give that a try.

Although computers appear very dumb it's just a front.   They can be insidiously clever - they know when you're on vacation!

Anne

-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                  P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************