Re: 20000924: statistics for data being handled by motherlode IDD? (fwd)

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================

---------- Forwarded message ----------
Date: Sun, 24 Sep 2000 20:53:37 -0600
From: Russ Rew <address@hidden>
To: Tom Yoksas <address@hidden>
Subject: Re: 20000924: statistics for data being handled by motherlode IDD? 

Tom,

> Has there been any workup of the amount of data that motherlode is
> handling currently?  Also, I am under the impression that motherlode is

I have some statistics on data from motherlode from running "pqutil
-w" last week on motherlode after Chiz included a few more feeds that
shemp had been getting.  The resulting 124 Mbyte log covers 15 hours
of time, product size, feed type, and product ID.  It's on buddy in
/buddy/russ/mother-prods.log.  An earlier 12-hour notifyme log is also
available as motherlode.log, but we decided that notifyme is not as
accurate for product arrival times as pqutil.

Here's some of what I got out of the log, covering 15 feeds going into
motherlodes product queue, including some that are not redistributed
(NLDN, SPARE, ACARS, ...):
   
 - For 15 hours, there were 14.47 Gbytes of data in 1138993 products.
   The min and max product sizes were 34 bytes (not counting a 12-byte
   spurious "product") and 14931306 bytes.  The average product size
   was 12708.5 bytes, but the product-size distribution is lumpy.

 - Extrapolating: 23.16 Gbytes/day in 1822390 products/day
   Averaging: 965 Mbytes/hour in 75933 products/hour, or 
              268054 bytes/sec in 21 products/second

 - Maximum products per second: 164.  The products/second distribution
   is pretty smooth, with a peak at about 18 products per second.
   There were only 449 seconds in 15 hours when no products were
   received.  The longest interval during which no products were
   received was 6 seconds.  These might be affected by occasional
   garbage-collection pauses in the LDM, so may not reflect what's
   happening on the data feeds.

 - Number of products per feedtype (extrapolated from 15 hours to 24
   hours):

   662890 WSI
   324736 NTEXT
   247482 NMC2
   210518 IDS|DDPLUS
   172115 NNEXRAD
   125970 HDS
    70346 NGRID
     3371 NIMAGE
     2557 NOGAPS
      790 SPARE
      640 NGRAPH
      326 MCIDAS
      264 FSL2
      240 NLDN
      144 PCWS

 - Number of Mbytes per feedtype (extrapolated from 15 hours to 24
   hours):

     7336 WSI
     6443 NMC2
     3231 NNEXRAD
     2977 NIMAGE
     1319 HDS
      938 NGRID
      262 NOGAPS
      210 NTEXT
      167 IDS|DDPLUS
       95 MCIDAS
       67 SPARE
       65 PCWS
       42 FSL2
        4 NGRAPH
        3 NLDN

--Russ