20030702: LSU LDM/network stress Test and test of email list

>From: Robert Leche <address@hidden>
>Organization: LSU
>Keywords: 200306161954.h5GJs2Ld016710 LDM-6 IDD

Bob,

re: continue to load seistan's LDM
>Initially loading was heavy, but loading has subsided some what.  Go 
>ahead with adding the additional load.

I added another machine to the list that seistan is feeding.  The full
list is now:

Machine                        Feeds from seistan
------------------------------+---------------------------------------------
emo.unidata.ucar.edu           HDS
chevy.unidata.ucar.edu         HDS, IDS|DDPLUS, UNIWISC
newshemp.unidata.ucar.edu      HDS
zasu.unidata.ucar.edu          HDS, IDS|DDPLUS, UNIWISC
zero.unidata.ucar.edu          HDS, IDS|DDPLUS, UNIWISC
imogene.unidata.ucar.edu       HDS
tornado.geos.ulm.edu           HDS, FSL2, IDS|DDPLUS, NNEXRAD, UNIWISC
hail.jsums.edu                 NNEXRAD, FNEXRAD, UNIWISC, IDS|DDPLUS
aqua.nsstc.uah.edu             UNIDATA (which is HDS, UNIWISC, IDS|DDPLUS)

I also installed a script that logs seistan's load averages (from
uptime), number of LDM connections (# seistan is feeding, # seistan is
requesting, total), time of the oldest product in the LDM queue (in
seconds), amount of free memory, and amount of swap used.

Here is a representative output:

CCYYMMDD.HHMM   ave1  ave5 ave15 nfeed nreceive nconnect nsec  memfree swapused
20030703.0116   2.35  2.66  2.79   27      8       35    19127  5096K    376K

Right at the moment, seistan is feeding 27 downstream LDM connections
and receiving 8 feeds from upstream LDMs, for a total of 35 LDM
connections.  As a comparison, thelma.ucar.edu has 120 downstream
feeds, 20 upstream feeds for a total of 140.  Again, thelma is the most
heavily loaded of all IDD relay nodes at the moment: it is relaying an
average of 40-50 Mbps of data.

The logging script runs once per minute, and the log output can be found in
~ldm/logs/seistan.uptime.  It will be rotated once per week and seven
files will be kept online.

We use the script, ~ldm/util/uptime.tcl, to monitor the top level IDD
nodes.  It helps us keep track of things that the real time stats plots
do not address.

Cheers,

Tom