20020906: thelma not too good

>From:  anne <address@hidden>
>Organization:  UCAR/Unidata
>Keywords:  200209070333.g873XUj09291

Anne and Jeff,

>While thelma looked pretty good about 6:30 today, with a load average
>around 5, now it's not looking so good.  The load average was about 14,
>and it was sluggish in responding.  

Nuts.

>There are only 71 rpc.ldmds at the moment, less than the 72 that I
>thought we were able to handle easily before the reboot.  There are lots
>of reclasses to atm, plus some to sunset.aos.wisc.edu. 

>(What's 'aos'?).

This appears to be f5.aos.wisc.edu.  They are reporting realtime stats,
and their latencies don't look good.  Seems to me that they should
be feeding from SSEC, no?

>And connections are being dropped.  

So, when the load average goes above some level, data stops getting
delivered reliably and reclass messages ensue.

>I started a cron job to run uptime every minute to track the load
>average.  The resulting log is in ~logs/uptime.log.

The contents of this file are very interesting.  The load average comes
and goes.  We now need to correlate that with CONDUIT data volume (or
anything else).

It seems to me that we need to jump on getting 5.2.1 ready so we can
get both Washington and Penn State to upgrade to it and run rtstats.
This should help us understand what is happening at these sites.

The overnight rtstats from atm and f5.aos are really interesting.
atm looks OK except for NNEXRAD, and f5 looks bad.  I don't know
what to make of this!

Tom
--
+-----------------------------------------------------------------------------+
* Tom Yoksas                                             UCAR Unidata Program *
* (303) 497-8642 (last resort)                                  P.O. Box 3000 *
* address@hidden                                   Boulder, CO 80307 *
* Unidata WWW Service                             http://www.unidata.ucar.edu/*
+-----------------------------------------------------------------------------+