Re: [Fwd: 20010712: ldmprods]

On Thu, 12 Jul 2001, Anne Wilson wrote:

> 
> 
> -------- Original Message --------
> Subject: 20010712: ldmprods
> Date: Thu, 12 Jul 2001 10:27:54 -0600
> From: Unidata Support <address@hidden>
> Reply-To: Brendon Hoch <address@hidden>
> Organization: UCAR/Unidata
> To: address@hidden
> 
> 
> ------- Forwarded Message
> 
> >To: address@hidden
> >cc: address@hidden
> >From: Brendon Hoch <address@hidden>
> >Subject: ldmprods
> >Organization: UCAR/Unidata
> >Keywords: 200107121555.f6CFtJ119364
> 
> This is a multi-part message in MIME format.
> --------------363269955B1438D41CA6F678
> Content-Type: text/plain; charset=us-ascii
> Content-Transfer-Encoding: 7bit
> 
> Hello,
> 
> We are using ldmprods (running in the background) to monitor ldm and
> notify us if there are problems with the feed.  When there are problems,
> it works great and sends us an email notification message (WMO threshold
> of 20 minutes met).  But often, we get email notification with a blank
> message.  This happens anywhere from 2 to 15 times per day.  Inspection
> of the system upon receiving these blank email messages indicates things
> are working fine.   and everything is working fine.  It seems to happen
> only when ldm is receiving data from IDD.  I had two ldm boxes running
> for a few days, one receiving via both IDD and our local NOAAPORT, one
> receiving from NOAAPORT only.  The box receiving from IDD and NOAAPORT
> is the one that continued to send blank email messages, so I suspect
> it's something that is being sent over IDD that may be causing the
> problem.
> 
> The command we are using to run ldmprods is:
> /usr/local/ldm/bin/ldmprods -m address@hidden -l /dev/null -t 20
> > /dev/console

Brendon,

I'm not quite sure why ldmprods is sending an blank message but I did look
at the code.  I would comment line 122 out from the original script.  I
believe that the problem is one of the feeds/sources is not within the
threshold that's set. 
 
Also, I would omit using the -l flag since you are redirecting the output
to /dev/console.

One other speculations could be that ldmprods is getting a blank stats
file when it does a :

@FILES = split( /[ \t\n]+/, `ls -rx *.stats` ) ;

But no other users has mentioned this problem.

If you are still having a problem send me how ldmprods is run, ie crontab
or command line and a couple of the *.stats files so I can look at the
feeds, etc.  Also, check on the creation of the stats files at the
begining of the hour.


Robb...
> 
> Originally, I thought it had something to do with the bell function of
> ldmprods, so I modified ldmprods to pipe the ("echo No products received
> threshold of $shortout minutes met. line to pipe to /dev/null.  But the
> blank email messages continue.
> 
> Is there something else I'm missing in order to prevent these blank
> email messages from being sent?  I've attached the copy of ldmprods that
> we're running, any help is greatly appreciated.
> 
> Thanks,
> Brendon
> 
> _______________________________________________________
> Brendon Hoch
> International Research Institute for climate prediction
> Lamont Doherty Earth Observatory of Columbia University
> 141/208 Monell, 61 Route 9W, Palisades, NY 10964
> -------------------------------------------------------
> Phone: (845)680-4444 Fax:(845)680-4488
> mailto:address@hidden
> http://iri.columbia.edu/~bhoch
> -------------------------------------------------------
> --------------363269955B1438D41CA6F678
> Content-Type: text/plain; charset=us-ascii;
>  name="ldmprods"
> Content-Transfer-Encoding: 7bit
> Content-Disposition: inline;
>  filename="ldmprods"
> 
> #!/usr/bin/perl
> #
> # Name:       ldmprods
> # 
> # Author:     Robb Kambic
> # Date  :     Sept 29, 1994 Mods Dec 20, 1995
> # 
> # Purpose:    Displays products in the *.stats files in one minute
> intervals
> #    
> # Description:  
> #
> # Dependencies:  
> $ENV{ 'PATH' } = '/bin:/usr/ucb/:/sbin:/usr/local/bin:/usr/bsd';
> $ENV{ 'SHELL' } = '/bin/sh';
> $ENV{ 'IFS' } = ''if $ENV{ 'IFS' } ne '';
> $ENV{ 'TZ' } = 'UTC';
> 
> # process command line switches
> 
> while ($_ = $ARGV[0], /^-/) {
>        shift;
>        last if /^--$/;
>             /^-D(.*)/ && ($debug = $1);
>            /^(-v)/ && $verbose++;
>            /^(-T)/ && ( $host = shift ) ;
>            /^(-d)/ && ( $dir = shift ) ;
>            /^(-b)/ && ( $bell = shift ) ;
>            /^(-l)/ && ( $log = shift ) ;
>            /^(-m)/ && ( $mail = shift ) ;
>            /^(-t)/ && ( $shortout = shift ) ;
> }
> 
> # create defaults if no parameter given
> # get host
> if( !$host ) {
>       $host = `uname -n` ;
> }
> # goto log dir
> if( !$dir ) {
>       $dir = "/usr/local/ldm/logs" ;
> }
> chdir "$dir" || 
> die "Can't cd to log dir $dir: $!\nSet the -d flag to the logs
> directory.\n" ;
> #
> # get/set timeout parameters
> $shortout = 15 unless $shortout ;
> $longout = 3 * $shortout ;
> #
> #look for a mail or Mail program 
> 
> if( -e "/usr/ucb/mail" ) {
>       $mailprog = "/usr/ucb/mail" ;
> } elsif( -e "/usr/sbin/Mail" ) {
>       $mailprog = "/usr/sbin/Mail" ;
> } elsif( -e "/usr/bin/mail" ) {
>       $mailprog = "/usr/bin/mail" ;
> } elsif( -e "/bin/mail" ) {
>       $mailprog = "/bin/mail" ;
> } else {
>       select( STDOUT ) ;
>       print "Can't find a mail program for ldmprods\n" ;
> }
> # if output is diverted to a log file
> if( $log ) {
>       open( LOG, ">$log" ) || die "Could not open log file: $log : $!\n" ;
>       select( LOG ) ;
>       $~ = STDOUT ;
>       $| = 1 ;
>       print 
> "                         Product Monitor for $host
> Feedtype    Number     Bytes     Number   Bytes       Avg / Max Lat    
> Latest" ;
> } else {
>       select( STDOUT ) ;
> }
> 
> # get file names in order
> @FILES = split( /[ \t\n]+/, `ls -rx *.stats` ) ;
> 
> if( $#FILES == -1 ) {
>       print STDOUT "No *.stats files found, check logs directory for
> files.\n" ;
>       exit 0 ;
> }
> while( 1 ) {
>       # get current date/time
>       ( $weekday, $month, $day, $hms ) = split(' ', `date` ) ;
>       ( $hour, $minute, $second ) = split(':', $hms ) ;
> 
>       if( $minute == 1 ) {
>               # get file names in order
>               @FILES = split( /[ \t\n]+/, `ls -rx *.stats` ) ;
>               # clear hour accumulator
>               undef %lastcount ;
>               undef %lastbytes ;
>       }
>       open( STATS, "<$FILES[ 0 ]" ) || 
>               die "could not open stats file: $FILES[ 0 ] : $!\n" ;
>       if( $log ) {
>               print
> "\n\n$lmonth $lday          Minute $lminute              Cumulative For
> Hour $lhour\n\n" ;
>       } else {
> 
>               system( "clear" ) ;
>               print 
> "                                  Product Monitor for $host
> $lmonth $lday          Minute $lminute              Cumulative For Hour
> $lhour
> Feedtype    Number     Bytes     Number   Bytes       Avg / Max Lat    
> Latest    Source Site\n\n" ;
>       }
>       while( <STATS> ) {
>               ( $arrivaltime, $feedtype, $source, $products, $bytes, 
>                       $latency, $max, $lasttime ) = split( ' ', $_ ) ;
>               if( $minute == 1 ) {
>                       $minprods = $products ;
>                       $minbytes = $bytes ;
>               } else {
>                       $minprods = $products - 
>                               $lastcount{ "$feedtype $source" } ;
>                       $minbytes = $bytes - 
>                               $lastbytes{ "$feedtype $source" } ;
>               }
>               $lastcount{ "$feedtype $source" } = $products ;
>               $lastbytes{ "$feedtype $source" } = $bytes ;
>               &timecheck() unless $minprods ;
>               # collect number of products for warning
>               $totalminprods += $minprods ;
>               write ;
>       }
>       close STATS ;
> 
>       # print documentation
>       if( $verbose ) {
>               print "
> The ldmprods script shows the products arriving at the LDM server every 
> minute and it accumulates the data for the current hour in UTC time. 
> The 
> fields are:
> 
> Feed type     : All feed types shown, will not appear until a product
> arrives.
> Number                : Number of products that arrived the past minute.
> Bytes         : Total bytes that arrived the past minute.
> Number                : Number of products that arrived so far this hour.
> Bytes         : Total bytes that arrived so far this hour.
> Avg Lat               : Average latency for products arriving, calculated by: 
>                 latency = arrival time at site - injection time at source
>                 The average is derived from averaging the latencies.
> Max Lat               : Maximum latency seen this hour, ie. 8\@13:28 reads 8 
> seconds 
>                 at 13 minutes and 28 seconds after the hour.
> Latest                : Last time a product arrived in hh:mm:ss format.
> Source Site     : Originating site where the feed enters the system.
> 
> Bell          : Bell rings for every minute no data arrives.\n" ;
>       }
>       # ring bell & email if no products arrived in a minute
> # Remove  $mailprog $mail from system line to prevent overpaging, 6/8/01
> BH
> # Replace with /dev/null
> #
>       if( ! $totalminprods ) {
>               print STDOUT "\a" ;
>               system 
> ("echo No products received threshold of $shortout minutes met. |
> /dev/null")
>               if( ++$totalout > $shortout && $mail && ! $totalmailsent ) ;
>               $totalmailsent = 1 ;
>       } else {
>               # reset counters
>               $totalout = 0 ;
>               $totalmailsent = 0 ;
>               $totalminprods = 0 ;
>       }
>       $lmonth = $month ;
>       $lday = $day ;
>       $lhour = $hour ;
>       $lminute = $minute ;
> 
>       #print "\nSecond=$second\n" ;
>       # adjust to minute when off by ?? seconds
>       if( $second == 0 ) {
>               sleep( 60 ) ;
>       } elsif( $second == 1 ) {
>               sleep( 59 ) ;
>       } elsif( $second < 15 ) {
>               sleep( 58 ) ;
>       } elsif( $second < 30 ) {
>               sleep( 31 ) ;
>       } elsif( $second < 45 ) {
>               sleep( 16 ) ;
>       } else {
>               sleep( 61 ) ;
>       }
> }
> close( LOG ) if( $log ) ;
> 
> exit 0;
> 
> # timecheck registers feed failures and mails when thresholds are
> reached
> sub timecheck
> {
>       if( $pasttime{ "$feedtype $source" } ) {
>               ( $pasttime, $count, $mailsent ) = 
>                       split( ' ', $pasttime{ "$feedtype $source" } ) ;
>               $thetime = time() ;
>               # this is the check for consecutive occurrances
>               if( ( $thetime - $pasttime ) < 90 ) {
>                       $count++ ;
>                       if( $feedtype =~ /DIFAX|HRS|MCIDAS/ && $count > 
> $longout ) {
>                               system 
>               ("echo $feedtype threshold of $longout minutes met. | $mailprog
> $mail")
>                               if( ! $mailsent && $mail ) ;
>                               $mailsent = 1 ;
>                       # this handles all other feeds DDS PPS IDS NLDN etc
>                       } elsif( $feedtype =~ /DDS|PPS|IDS|NLDN|WMO/ &&
>                                       $count > $shortout ) {
>                               system 
>       ("echo $feedtype threshold of $shortout minutes met. | $mailprog
> $mail")
>                               if( ! $mailsent && $mail ) ;
>                               $mailsent = 1 ;
>                       }
>                       $pasttime{ "$feedtype $source" } = "$thetime $count 
> $mailsent" ;
>                       #print "$feedtype $thetime $count $mailsent\n" ;
>                       
>               } else {
>                       # old outage of data over, start again
>                       delete( $pasttime{ "$feedtype $source" } ) ;
>               }       
>       } else {
>               $thetime = time() ;
>               # set time of outage, minute, and mail sent to false
>               $pasttime{ "$feedtype $source" } = "$thetime 1 0" ;
>       }
> } #end timecheck
> 
> format STDOUT_TOP =
> .
> 
> format STDOUT =
> @<<<<<<<<< @#####  @#########   @#####   @######  @###.##  @>>>>>>:@>  
> @<:@<:@<  @<<<<<<<<<<<<<<<<<<<<<<<<<<
> $feedtype, $minprods, $minbytes, $products, $bytes, $latency, substr(
> $max, 0, length( $max ) -2), substr( $max,length( $max ) -2), substr(
> $lasttime, 8, 2 ), substr( $lasttime, 10, 2 ), substr( $lasttime, 12, 2
> ), $source
> .
> 
> --------------363269955B1438D41CA6F678--
> 
> 
> ------- End of Forwarded Message
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================