[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Was ldmConnect deprecated?



On Thu, 7 Oct 2010, Gregory Grosshans wrote:

Hi Robb,

Was it deprecates because the NWSTG/TOC/Gateway doesn't send out segmented 
products
anymore,

Hi Gregg,

Yes.


or does other software piece together the segmented products?

No other s/w pieces together segmented products.

Robb...


Thanks,
Gregg

Robb Kambic wrote:
      On Wed, 6 Oct 2010, Gregory Grosshans wrote:

            Can you tell me if ldmConnect was deprecated?


      Hi Gregory,

      Yes it was deprecated. But, I attached the documentation and decoder
      ldmConnect. It should work but it's not official supported.

      Robb...

      I wasn't able to easily find
            information about this, except it was in the decoders-4.1.7
            release and then
            was dropped in the 5.0 release.

            Thanks,
            Gregg



      
===============================================================================
      Robb Kambic                       Unidata Program Center
      Software Engineer III               Univ. Corp for Atmospheric Research
      address@hidden           WWW: http://www.unidata.ucar.edu/
      
===============================================================================

    __________________________________________________________________________

#!/usr/local/bin/perl

## Connect split LDM bulletins

#use strict ;

package Part ;

use File::Path ;

sub new { return bless {} ; }

sub getData {

my $self = shift;
my( $data ) = @_;
my( $count, $state, $wmohdr ) ;

return "SKIP", "", "" if( length( $data ) < 5 ) ;
if( ( $count = $data =~ tr/\n/\n/ ) < 7 ) { # not enough CR's
        $data =~ s#\cM+\n?#\cM\cM\n#g ; # convert \cM+ to CR's
}
$data =~ s#\cC## ;
$data =~ s#\cA\cM+\n## ;
# get header $data =~ s#(\d\d\d)\s*\cM+\n## ;
$self->{ 'number' } = $1 ;
$data =~ s#^\s*\cM+\n## ;  # reports with extra CR
return "SKIP", "", "" if( length( $data ) < 5 ) ;
$state = "" ;
$data =~ m#(.*?\cM+)\n# ;
$wmohdr = $1 ;
if( $wmohdr =~ m# (P\w\w)\cM| (P\w\w) (RTD|RTN)# ) {
        $state = ( $1 ? $1 : $2 ) ;
}
if( $data =~ s#^(\w{4} ?\d{0,2})\s+(\w{4})\s+(\d{4})(\d{2}).*\cM+\n##m ) {
        $self->{ 'bulletin' } = $1 ;
        $self->{ 'bulletin' } =~ s#\s+## ;
        $self->{ 'station' } = $2 ;
        $self->{ 'header' } = "$1 $2 $3$4" ;
        $self->{ 'ddhh' } = $3 ;
        $self->{ 'mm' } = $4 ;
} elsif( $data =~ s#(\w{4} ?\d{0,2})\s+(\w{4})\s+(DDHHMM).*\cM+\n##m ) {
        $self->{ 'bulletin' } = $1 ;
        $self->{ 'bulletin' } =~ s#\s+## ;
        $self->{ 'station' } = $2 ;
        $self->{ 'header' } = "$1 $2 $3" ;
        $self->{ 'ddhh' } = "DDHH" ;
        $self->{ 'mm' } = "MM" ;
} else {
        $self->{ 'ddhh' } = "DDHH" ;
        $self->{ 'mm' } = "MM" ;
}
if( ! $state  ) {
        $self->{ 'data' } = $data ;
        return "OK", "$self->{ 'header' }", "$state" ;
} elsif( $state =~ /^PZ/ ) {
        $self->{ "$state" } = $data ;
        return "DONE", "$self->{ 'header' }", "$state" ;
} elsif( $state =~ /^PAA/ ) {
        $self->{ 'parts' } = "$state " ;
        $self->{ 'btype' } = $1 if( $data =~ s#^(\w{5,7})\s*\cM*\n## ) ;
        $self->{ "$state" } = $data ;
        return "HEAD", "$self->{ 'header' }", "$state" ;
} elsif( $state =~ /^P/ ) {
        $self->{ "$state" } = $data ;
        return "PART", "$self->{ 'header' }", "$state" ;
} else {
        #if( $state =~ /AMD|COR|AAA|RTD/ )
        $self->{ 'data' } = $data ;
        return "OK", "$self->{ 'header' }", "$state" ;
}
} # end getData

# build filename
sub buildFN {

my $self = shift;
my( $dirStruct, $datadir, $yymm ) = @_ ;
my( $filename ) ;

if( $dirStruct ) {
        $filename = $datadir . "/" . $self->{ 'station' } .
                "/" . $self->{ 'bulletin' } ;
        mkpath( $filename, 0, 0775 ) ;
        $filename .= "/" . $yymm . $self->{ 'ddhh' } . $self->{ 'mm' } ;
} else {
        $filename = $datadir . "/" . $yymm . $self->{ 'ddhh' } ;
}
return $filename ;
} # End buildFN

sub print {

my $self = shift;
my( $cChars, $section ) = @_ ;

my( $end ) ;

if( $cChars ) {
        $end = "\n" ;
        $self->{ 'data' } =~ s#\cM##g ;
} else {
        $end = "\cM\cM\n" ;
        print "\cA\cM\cM\n" ;
}
print $self->{ 'number' }, " $end" ;
if( defined( $self->{ 'data' } ) ) {
        print "$self->{ 'header' }$end" ;
        print "$self->{ 'data' }$end" ;
} else {
        print "$self->{ 'header' } $section$end" ;
        print $self->{ $section } . $end ;
}
print "\cC" unless( $cChars ) ;

}

package Bulletin ;

#use vars qw( @ISA ) ;
@ISA = qw( Part ) ;

sub new {
        my $type = shift ;
        my( $self ) = @_  ;
        $self->{ 'Time' } = time() ; # Time of existence
        bless( $self, $type ) ;
        return $self ;
} # end new

sub time { # returns number of seconds in existence

my $self = shift;
return  time() - $self->{ 'Time' } ;
}

# add parts into bulletin object
sub addPart {

my $self = shift;
my( $section, $part ) = @_ ;

if( ! defined( $self->{ 'number' } ) ) {
        $self->{ 'number' } = $part->{ 'number' } ;
        $self->{ 'header' } = $part->{ 'header' } ;
        $self->{ 'ddhh' } = $part->{ 'ddhh' } ;
        $self->{ 'btype' } = $part->{ 'btype' } ;
}
$self->{ 'parts' } .= "$section " unless( $self->{ 'parts' } =~ /$section/ ) ;
$self->{ "$section" } = $part->{ "$section" } ;

} # end addPart

# PAA section is already collected
sub PAA {

my $self = shift;
if( $self->{ 'parts' } =~ /PAA/ ) {
        return 1 ;
} else {
        return 0 ;
}
} # end PAA

# combine sections together to make complete bulletin
sub combine {

my $self = shift;
my( $section, @sections ) ;

@sections = split( /[ \t\n]+/, $self->{ 'parts' } ) ;
foreach $section ( sort @sections ) {
        if( $self->{ 'btype' } && $section =~ /PAA/ ) {
                $self->{ "$section" } = "$self->{ 'btype' } \cM\n" .
                        $self->{ "$section" } ;
        } elsif( $self->{ 'btype' } ) {
                $self->{ "$section" } =~ s#^$self->{ 'btype' }\s*\cM+\n##  ;
        }
        # eat ending line feed in case bulletin split in middle of word
        $self->{ "$section" } =~ s/\cM+\n$//s ;
        $self->{ 'data' } .= $self->{ "$section" } ;
}
} # end combine

sub print {

my $self = shift;
my( $cChars ) = @_ ;

my( $end ) ;

if( $cChars ) {
        $end = "\n" ;
        $self->{ 'data' } =~ s#\cM##g ;
} else {
        $end = "\cM\cM\n" ;
        print "\cA\cM\cM\n" ;
}
print $self->{ 'number' }, " $end" ;
print "$self->{ 'header' }$end" ;
print "$self->{ 'data' }$end" ;
print "\cC" unless( $cChars ) ;
}

# fill bulletins parts from inProgress file
sub fill {

my $self = shift;
my( $data ) = @_ ;

my( $i, @parts, $section, @sections ) ;

$i = 0 ;
( @sections ) = split( /\t/, $data ) ;
$self->{ 'number' } = $sections[ $i++ ] ;
$self->{ 'bulletin' } = $sections[ $i++ ] ;
$self->{ 'station' } = $sections[ $i++ ] ;
$self->{ 'header' } = $sections[ $i++ ] ;
$self->{ 'ddhh' } = $sections[ $i++ ] ;
$self->{ 'mm' } = $sections[ $i++ ] ;
$self->{ 'parts' } = $sections[ $i++ ] ;
( @parts ) = split( /[ \t\n]+/, $self->{ 'parts' } ) ;
foreach $section ( @parts ) {
        $self->{ "$section" } = $sections[ $i++ ] ;
}
return $self->{ 'header' } ;
}

# dump bulletins parts to inProgress file
sub dump {

my $self = shift;
my( $section, @sections ) ;

print $self->{ 'number' }, "\t" ;
print $self->{ 'bulletin' }, "\t" ;
print $self->{ 'station' }, "\t" ;
print "$self->{ 'header' }\t" ;
print "$self->{ 'ddhh' }\t" ;
print "$self->{ 'mm' }\t" ;
print "$self->{ 'parts' }\t" ;
@sections = split( /[ \t\n]+/, $self->{ 'parts' } ) ;
foreach $section ( sort @sections ) {
        print $self->{ "$section" }, "\t" ;
}
print "\cC" ;
}

# sets, deletes, and stores file handles
package doIO ;

use FileHandle ;
#use vars qw( %handles %hTimes $logging ) ;

%handles = () ;
%hTimes = () ;
$logging = 0 ;

sub new { return bless {} ; }

sub setLog {

my $self = shift;
my( $datadir ) = @_ ;

# redirect STDOUT and STDERR
open( STDOUT, ">$datadir/connectLog.$$.log" ) ||
                die "could not open $datadir/connectLog.$$.log: $!\n" ;
open( STDERR, ">&STDOUT" ) ||
                die "could not dup stdout: $!\n" ;
select( STDERR ) ; $| = 1 ;
select( STDOUT ) ; $| = 1 ;
$logging = 1 ;

}

# sets a filehandle for the data
sub setFH {

my $self = shift;
my( $filename, $ext ) = @_ ;

my( $fh, $file, @files, $outfile, $opentime, $thetime, $Time ) ;

if( defined( $handles{ "$filename" } ) ) {
        $fh = $handles{ "$filename" } ;
        select( $fh ) ; $| = 1 ;
        return ;
}
$outfile =  $filename . $ext ;

# current time
$thetime = time() ;
( @files ) = keys %handles ;
#print STDERR "Number of files open is $#files.\n" ;
if( $#files < 15 ) {
        $opentime = 1200 ;
} elsif( $#files < 25 ) {
        $opentime = 600 ;
} elsif( $#files < 35 ) {
        $opentime = 300 ;
} elsif( $#files < 45 ) {
        $opentime = 60 ;
} else {
        $opentime = -1 ;
}
# close files with no activity for $opentime minutes
foreach $file ( keys %handles ) {
        $Time = $hTimes{ "$file" } ;
        if( $thetime - $Time > $opentime ) {
                print STDOUT
                        "Closing $file$ext, No write for > $opentime Sec\n"
                        if( $logging ) ;
                $fh = $handles{ "$file" } ;
                close( $fh ) ;
                delete( $handles{ $file } ) ;
        } else {
                $hTimes{ "$file" } = $thetime ;
        }
}
# open or create outfiles
$fh = new FileHandle ;
$fh->open( ">>$outfile" ) || print STDERR "Can't open $outfile: $!\n" ;
select( $fh ) ;
$fh->autoflush( 1 ) ;
$handles{ "$filename" } = $fh ;
$hTimes{ "$filename" } = $thetime ;
print STDOUT "Opening $outfile at $thetime\n" if( $logging ) ;

return ;
} # end setFH

sub Close {

my $self = shift;
my( $ext ) = @_ ;
my( $fh, $file ) ;

foreach $file ( keys %handles ) {
        $fh = $handles{ "$file" } ;
        close( $fh ) ;
        print STDOUT "Closing $file$ext\n" if( $logging ) ;
}
close( STDOUT ) ;
close( STDERR ) ;
}

sub inProgress {

my $self = shift;
my( $filename, $task ) = @_ ;

if( $task eq "data" ) {
        $_ = <INPROGRESS> ;
} elsif( $task eq "write" ) {
        open( INPROGRESS, ">$filename" ) ;
        select( INPROGRESS ) ; $| = 1 ;
} elsif( $task eq "read" ) {
        if( -e "$filename" && -s "$filename" ) {
                open( INPROGRESS, "$filename" ) ;
                return 1 ;
        } else {
                return 0 ;
        }
} else {
        close( INPROGRESS ) ;
}
} #end inProgress

# Connect and write bulletins here.
package main ;

use File::Path ;

#use vars qw( $cChars $ext $dirStruct $logging $yymm $datadir $filename ) ;
#use vars qw( $status $header $section $io $part $bulletin %inProgress ) ;
#use vars qw( $rin $timeout $nfound $rout ) ;

# process command line switches
while ($_ = $ARGV[0], /^-/) {
         shift;
       last if /^--$/;
             /^(-s)/ && $cChars++ ; # strip control chars
                # use this extension, default _for.wmo
             /^(-e)/ && do { # LDM doesn't pass a null, could get next arg
                                $ext = shift ;
                                if( $ext =~ m#^(-|\d|\s+)|/# ) {
                                        unshift( @ARGV, $ext )
                                                unless( $ext =~ m#^\s# ) ;
                                        $ext = "" ;
                                }
                        } ;
             /^(-a)/ && $dirStruct++ ; # write /stn/bulletin/report
             /^(-l)/ && $logging++ ; # turn logging on
}
$cChars = 0 if( ! defined( $cChars ) ) ;
$dirStruct = 0 if( ! defined( $dirStruct ) ) ;
$ext = "_for.wmo" if( ! defined( $ext ) ) ;
# process input parameters
if( $#ARGV == 0 ) {
        if( $ARGV[ 0 ] =~ /^[0-9]/ ) {
                $yymm = $ARGV[ 0 ] ;
        } else {
                $datadir = $ARGV[ 0 ] ;
        }
} elsif( $#ARGV == 1 ) {
        $yymm = $ARGV[ 0 ] ;
        $datadir = $ARGV[ 1 ] ;
} else {
        die "Usage with input on STDIN: ldmConnect [-a] [-s] [-l] [-e ext ] 
[yymm] [datadir
]\n";
}
# set interrupt handler
$SIG{ 'INT' }  = 'atexit' ;
$SIG{ 'KILL' }  = 'atexit' ;
$SIG{ 'TERM' }  = 'atexit' ;
$SIG{ 'QUIT' }  = 'atexit' ;
$SIG{ 'HUP' }  = 'atexit' ;

# the directory $datadir = "." if( ! $datadir ) ;
mkpath( $datadir, 0, 0775 ) if( ! -d $datadir ) ;
$io = new doIO() ;
$io->setLog( $datadir ) if( $logging ) ;

# year and month
if( ! $yymm ) {
        my( $theyear, $themonth ) ;
        $theyear = (gmtime())[ 5 ] ;
        $theyear = ( $theyear < 100 ? $theyear : $theyear - 100 ) ;
        $theyear = sprintf( "%02d", $theyear ) ;
        $themonth = (gmtime())[ 4 ] ;
        $themonth++ ;
        $yymm = $theyear . sprintf( "%02d", $themonth ) ;
}
# Now begin parsing file and decoding observations breaking on cntrl C
$/ = "\cC" ;

# read in inprogress data from last process, fill in bulletin
$filename = $datadir . "/inProgress" . "$ext" ;
if( $io->inProgress( $filename, "read" ) ) {
        while( 1 ) {
                $io->inProgress( $filename, "data" ) ;
                last if( !$_ ) ;
                $part = new Part() ;
                $bulletin = new Bulletin( $part ) ;
                $header = $bulletin->fill( $_ ) ;
                $inProgress{ "$header" } = $bulletin ;
                undef( $part ) ;
        }
        $io->inProgress( $filename, "close" ) ;
}
# main loop   set select processing here from STDIN
START:
while( 1 ) {
        open( STDIN, '-' ) ;
        vec( $rin, fileno( STDIN ), 1 ) = 1 ;
        $timeout = 1200 ; # 20 minutes
        $nfound = select( $rout = $rin, undef, undef, $timeout ) ;
        # timed out
        if( ! $nfound ) {
                print STDOUT "Shut down, time out 20 minutes\n"
                        if( $logging ) ;
                atexit() ;
        }
        atexit( "eof" ) if( eof( STDIN ) ) ;

        $part = new Part() ;
        # Process data first as part of a bulletin
        $_ = <STDIN> ;
        ( $status, $header, $section ) = $part->getData( $_ ) ;
        if( $status eq "OK" ) { # complete bulletin, write it out
                $filename = $part->buildFN( $dirStruct, $datadir, $yymm ) ;
                $io->setFH( $filename, $ext ) ;
                $part->print( $cChars, $section ) ;
        } elsif( $status eq "HEAD" ) {  # create new bulletin
                $bulletin = new Bulletin( $part ) ;
                $inProgress{ "$header" } = $bulletin ;
        } elsif( $status eq "PART" || $status eq "DONE" ) {
                if( defined( $inProgress{ "$header" } ) ) {
                        $bulletin = $inProgress{ "$header" } ;
                        $bulletin->addPart( $section, $part ) ;
                        # combine parts and write
                        if( $status eq "DONE" || $bulletin->time() > 900 ) {
                                $bulletin->combine() ;
                                $filename = $bulletin->buildFN
                                        ( $dirStruct, $datadir, $yymm ) ;
                                $io->setFH( $filename, $ext ) ;
                                $bulletin->print( $cChars ) ;
                                delete( $inProgress{ "$header" } ) ;
                        }
                } else { # catches weird PRE, PMD, etc suffices
                        $filename = $part->buildFN
                                ( $dirStruct, $datadir, $yymm ) ;
                        $io->setFH( $filename, $ext ) ;
                        $part->print( $cChars, $section ) ;
                }
        }
        undef( $part ) ;
        atexit( "eof" ) if( eof( STDIN ) ) ;
} # end while( 1 )
atexit( "eof" );
exit( 0 ) ; # should never get here

# execute at exit
sub atexit
{
my( $sig ) = @_ ;

if( $sig eq "eof" ) {
        print STDOUT "eof on STDIN --shutting down\n" if( $logging ) ;
} elsif( defined( $sig )) {
        print STDOUT "Caught SIG$sig --shutting down\n" if( $logging ) ;
}
# check if bulletin is older than 15 minutes, if so write, else save in
# the inProgress file
$filename = $datadir . "/inProgress" . "$ext" ;
$io->inProgress( $filename, "write" ) ;
foreach $header ( %inProgress ) {
        $bulletin = $inProgress{ "$header" } ;
        if( defined( $bulletin ) ) {
                if( $bulletin->time() < 900 ) {
                        $bulletin->dump() ;
                } else {
                        $bulletin->combine() ;
                        $filename = $bulletin->buildFN
                                        ( $dirStruct, $datadir, $yymm ) ;
                        $io->setFH( $filename, $ext ) ;
                        $bulletin->print( $cChars ) ;
                }
        }
}
$io->inProgress( $filename, "close" ) ;
$io->Close( $ext ) ;
exit( 0 ) ;
}


    __________________________________________________________________________


Documentation for "ldmConnect"

The perl decoder "ldmConnect" pieces together segmented zone forecasts or
other segmented products with WMO segment identifiers PAA, PAB, etc. For more information about the segmentation of large products and the
use of bulletin segment heading lines utilizing the Pxx Indicator Group,
see

http://www.nws.noaa.gov/tg/bbb.html

The "ldmConnect" script orders and combines bulletin segments in memory,
strips out extra headers, and then writes the completed bulletin.  If
"ldmConnect" exits, it writes an "inProgress" file to maintain the current
state of processing. When restarted, "ldmConnect" reads the "inProgress"
file first. Non split reports are written directly.

Usage with input on STDIN:
   ldmConnect [-a] [-s] [-l] [-e ext ] [yymm] [datadir]

-a    locates bulletin in directory structure  "station ID"/"bulletin ID"/

-s    strips out control characters (other than newline)

-l    turns on logging to create log files of the form "connectLog.<PID>.wmo"

-e    sets the file extension, default is "_for.wmo".  File format is:
      yymmddhh + extension.

yymm   is the current year and month. Current yymm is default.

datadir is the output directory for data, inProgress, and log files.


Typically "ldmConnect" is located in the decoders directory.

Typical pqact.conf entry:

# Forecasts in yymmddhh_for.wmo
# perl script ldmConnect, joins split bulletins, writes all bulletins
DDS|PPS ^F[PTXEQWZ].... .... ([0-3][0-9])
        PIPE    /usr/local/ldm/decoders/ldmConnect
        (\1:yy)(\1:mm)
        data/forecasts







===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================