Hi Robb,
Was it deprecates because the NWSTG/TOC/Gateway doesn't send out
segmented products anymore, or does other software piece together the
segmented products?
Thanks,
Gregg
Robb Kambic wrote:
On Wed, 6 Oct 2010, Gregory Grosshans wrote:
Can you tell me if ldmConnect was deprecated?
Hi Gregory,
Yes it was deprecated. But, I attached the documentation and decoder
ldmConnect. It should work but it's not official supported.
Robb...
I wasn't able to easily find
information about this, except it was in the
decoders-4.1.7 release and then
was dropped in the 5.0 release.
Thanks,
Gregg
===============================================================================
Robb Kambic Unidata Program Center
Software Engineer III Univ. Corp for Atmospheric Research
address@hidden WWW: http://www.unidata.ucar.edu/
===============================================================================
#!/usr/local/bin/perl
## Connect split LDM bulletins
#use strict ;
package Part ;
use File::Path ;
sub new { return bless {} ; }
sub getData {
my $self = shift;
my( $data ) = @_;
my( $count, $state, $wmohdr ) ;
return "SKIP", "", "" if( length( $data ) < 5 ) ;
if( ( $count = $data =~ tr/\n/\n/ ) < 7 ) { # not enough CR's
$data =~ s#\cM+\n?#\cM\cM\n#g ; # convert \cM+ to CR's
}
$data =~ s#\cC## ;
$data =~ s#\cA\cM+\n## ;
# get header
$data =~ s#(\d\d\d)\s*\cM+\n## ;
$self->{ 'number' } = $1 ;
$data =~ s#^\s*\cM+\n## ; # reports with extra CR
return "SKIP", "", "" if( length( $data ) < 5 ) ;
$state = "" ;
$data =~ m#(.*?\cM+)\n# ;
$wmohdr = $1 ;
if( $wmohdr =~ m# (P\w\w)\cM| (P\w\w) (RTD|RTN)# ) {
$state = ( $1 ? $1 : $2 ) ;
}
if( $data =~ s#^(\w{4} ?\d{0,2})\s+(\w{4})\s+(\d{4})(\d{2}).*\cM+\n##m ) {
$self->{ 'bulletin' } = $1 ;
$self->{ 'bulletin' } =~ s#\s+## ;
$self->{ 'station' } = $2 ;
$self->{ 'header' } = "$1 $2 $3$4" ;
$self->{ 'ddhh' } = $3 ;
$self->{ 'mm' } = $4 ;
} elsif( $data =~ s#(\w{4} ?\d{0,2})\s+(\w{4})\s+(DDHHMM).*\cM+\n##m ) {
$self->{ 'bulletin' } = $1 ;
$self->{ 'bulletin' } =~ s#\s+## ;
$self->{ 'station' } = $2 ;
$self->{ 'header' } = "$1 $2 $3" ;
$self->{ 'ddhh' } = "DDHH" ;
$self->{ 'mm' } = "MM" ;
} else {
$self->{ 'ddhh' } = "DDHH" ;
$self->{ 'mm' } = "MM" ;
}
if( ! $state ) {
$self->{ 'data' } = $data ;
return "OK", "$self->{ 'header' }", "$state" ;
} elsif( $state =~ /^PZ/ ) {
$self->{ "$state" } = $data ;
return "DONE", "$self->{ 'header' }", "$state" ;
} elsif( $state =~ /^PAA/ ) {
$self->{ 'parts' } = "$state " ;
$self->{ 'btype' } = $1 if( $data =~ s#^(\w{5,7})\s*\cM*\n## ) ;
$self->{ "$state" } = $data ;
return "HEAD", "$self->{ 'header' }", "$state" ;
} elsif( $state =~ /^P/ ) {
$self->{ "$state" } = $data ;
return "PART", "$self->{ 'header' }", "$state" ;
} else {
#if( $state =~ /AMD|COR|AAA|RTD/ )
$self->{ 'data' } = $data ;
return "OK", "$self->{ 'header' }", "$state" ;
}
} # end getData
# build filename
sub buildFN {
my $self = shift;
my( $dirStruct, $datadir, $yymm ) = @_ ;
my( $filename ) ;
if( $dirStruct ) {
$filename = $datadir . "/" . $self->{ 'station' } .
"/" . $self->{ 'bulletin' } ;
mkpath( $filename, 0, 0775 ) ;
$filename .= "/" . $yymm . $self->{ 'ddhh' } . $self->{ 'mm' } ;
} else {
$filename = $datadir . "/" . $yymm . $self->{ 'ddhh' } ;
}
return $filename ;
} # End buildFN
sub print {
my $self = shift;
my( $cChars, $section ) = @_ ;
my( $end ) ;
if( $cChars ) {
$end = "\n" ;
$self->{ 'data' } =~ s#\cM##g ;
} else {
$end = "\cM\cM\n" ;
print "\cA\cM\cM\n" ;
}
print $self->{ 'number' }, " $end" ;
if( defined( $self->{ 'data' } ) ) {
print "$self->{ 'header' }$end" ;
print "$self->{ 'data' }$end" ;
} else {
print "$self->{ 'header' } $section$end" ;
print $self->{ $section } . $end ;
}
print "\cC" unless( $cChars ) ;
}
package Bulletin ;
#use vars qw( @ISA ) ;
@ISA = qw( Part ) ;
sub new {
my $type = shift ;
my( $self ) = @_ ;
$self->{ 'Time' } = time() ; # Time of existence
bless( $self, $type ) ;
return $self ;
} # end new
sub time { # returns number of seconds in existence
my $self = shift;
return time() - $self->{ 'Time' } ;
}
# add parts into bulletin object
sub addPart {
my $self = shift;
my( $section, $part ) = @_ ;
if( ! defined( $self->{ 'number' } ) ) {
$self->{ 'number' } = $part->{ 'number' } ;
$self->{ 'header' } = $part->{ 'header' } ;
$self->{ 'ddhh' } = $part->{ 'ddhh' } ;
$self->{ 'btype' } = $part->{ 'btype' } ;
}
$self->{ 'parts' } .= "$section " unless( $self->{ 'parts' } =~ /$section/ ) ;
$self->{ "$section" } = $part->{ "$section" } ;
} # end addPart
# PAA section is already collected
sub PAA {
my $self = shift;
if( $self->{ 'parts' } =~ /PAA/ ) {
return 1 ;
} else {
return 0 ;
}
} # end PAA
# combine sections together to make complete bulletin
sub combine {
my $self = shift;
my( $section, @sections ) ;
@sections = split( /[ \t\n]+/, $self->{ 'parts' } ) ;
foreach $section ( sort @sections ) {
if( $self->{ 'btype' } && $section =~ /PAA/ ) {
$self->{ "$section" } = "$self->{ 'btype' } \cM\n" .
$self->{ "$section" } ;
} elsif( $self->{ 'btype' } ) {
$self->{ "$section" } =~ s#^$self->{ 'btype' }\s*\cM+\n## ;
}
# eat ending line feed in case bulletin split in middle of word
$self->{ "$section" } =~ s/\cM+\n$//s ;
$self->{ 'data' } .= $self->{ "$section" } ;
}
} # end combine
sub print {
my $self = shift;
my( $cChars ) = @_ ;
my( $end ) ;
if( $cChars ) {
$end = "\n" ;
$self->{ 'data' } =~ s#\cM##g ;
} else {
$end = "\cM\cM\n" ;
print "\cA\cM\cM\n" ;
}
print $self->{ 'number' }, " $end" ;
print "$self->{ 'header' }$end" ;
print "$self->{ 'data' }$end" ;
print "\cC" unless( $cChars ) ;
}
# fill bulletins parts from inProgress file
sub fill {
my $self = shift;
my( $data ) = @_ ;
my( $i, @parts, $section, @sections ) ;
$i = 0 ;
( @sections ) = split( /\t/, $data ) ;
$self->{ 'number' } = $sections[ $i++ ] ;
$self->{ 'bulletin' } = $sections[ $i++ ] ;
$self->{ 'station' } = $sections[ $i++ ] ;
$self->{ 'header' } = $sections[ $i++ ] ;
$self->{ 'ddhh' } = $sections[ $i++ ] ;
$self->{ 'mm' } = $sections[ $i++ ] ;
$self->{ 'parts' } = $sections[ $i++ ] ;
( @parts ) = split( /[ \t\n]+/, $self->{ 'parts' } ) ;
foreach $section ( @parts ) {
$self->{ "$section" } = $sections[ $i++ ] ;
}
return $self->{ 'header' } ;
}
# dump bulletins parts to inProgress file
sub dump {
my $self = shift;
my( $section, @sections ) ;
print $self->{ 'number' }, "\t" ;
print $self->{ 'bulletin' }, "\t" ;
print $self->{ 'station' }, "\t" ;
print "$self->{ 'header' }\t" ;
print "$self->{ 'ddhh' }\t" ;
print "$self->{ 'mm' }\t" ;
print "$self->{ 'parts' }\t" ;
@sections = split( /[ \t\n]+/, $self->{ 'parts' } ) ;
foreach $section ( sort @sections ) {
print $self->{ "$section" }, "\t" ;
}
print "\cC" ;
}
# sets, deletes, and stores file handles
package doIO ;
use FileHandle ;
#use vars qw( %handles %hTimes $logging ) ;
%handles = () ;
%hTimes = () ;
$logging = 0 ;
sub new { return bless {} ; }
sub setLog {
my $self = shift;
my( $datadir ) = @_ ;
# redirect STDOUT and STDERR
open( STDOUT, ">$datadir/connectLog.$$.log" ) ||
die "could not open $datadir/connectLog.$$.log: $!\n" ;
open( STDERR, ">&STDOUT" ) ||
die "could not dup stdout: $!\n" ;
select( STDERR ) ; $| = 1 ;
select( STDOUT ) ; $| = 1 ;
$logging = 1 ;
}
# sets a filehandle for the data
sub setFH {
my $self = shift;
my( $filename, $ext ) = @_ ;
my( $fh, $file, @files, $outfile, $opentime, $thetime, $Time ) ;
if( defined( $handles{ "$filename" } ) ) {
$fh = $handles{ "$filename" } ;
select( $fh ) ; $| = 1 ;
return ;
}
$outfile = $filename . $ext ;
# current time
$thetime = time() ;
( @files ) = keys %handles ;
#print STDERR "Number of files open is $#files.\n" ;
if( $#files < 15 ) {
$opentime = 1200 ;
} elsif( $#files < 25 ) {
$opentime = 600 ;
} elsif( $#files < 35 ) {
$opentime = 300 ;
} elsif( $#files < 45 ) {
$opentime = 60 ;
} else {
$opentime = -1 ;
}
# close files with no activity for $opentime minutes
foreach $file ( keys %handles ) {
$Time = $hTimes{ "$file" } ;
if( $thetime - $Time > $opentime ) {
print STDOUT
"Closing $file$ext, No write for > $opentime Sec\n"
if( $logging ) ;
$fh = $handles{ "$file" } ;
close( $fh ) ;
delete( $handles{ $file } ) ;
} else {
$hTimes{ "$file" } = $thetime ;
}
}
# open or create outfiles
$fh = new FileHandle ;
$fh->open( ">>$outfile" ) || print STDERR "Can't open $outfile: $!\n" ;
select( $fh ) ;
$fh->autoflush( 1 ) ;
$handles{ "$filename" } = $fh ;
$hTimes{ "$filename" } = $thetime ;
print STDOUT "Opening $outfile at $thetime\n" if( $logging ) ;
return ;
} # end setFH
sub Close {
my $self = shift;
my( $ext ) = @_ ;
my( $fh, $file ) ;
foreach $file ( keys %handles ) {
$fh = $handles{ "$file" } ;
close( $fh ) ;
print STDOUT "Closing $file$ext\n" if( $logging ) ;
}
close( STDOUT ) ;
close( STDERR ) ;
}
sub inProgress {
my $self = shift;
my( $filename, $task ) = @_ ;
if( $task eq "data" ) {
$_ = <INPROGRESS> ;
} elsif( $task eq "write" ) {
open( INPROGRESS, ">$filename" ) ;
select( INPROGRESS ) ; $| = 1 ;
} elsif( $task eq "read" ) {
if( -e "$filename" && -s "$filename" ) {
open( INPROGRESS, "$filename" ) ;
return 1 ;
} else {
return 0 ;
}
} else {
close( INPROGRESS ) ;
}
} #end inProgress
# Connect and write bulletins here.
package main ;
use File::Path ;
#use vars qw( $cChars $ext $dirStruct $logging $yymm $datadir $filename ) ;
#use vars qw( $status $header $section $io $part $bulletin %inProgress ) ;
#use vars qw( $rin $timeout $nfound $rout ) ;
# process command line switches
while ($_ = $ARGV[0], /^-/) {
shift;
last if /^--$/;
/^(-s)/ && $cChars++ ; # strip control chars
# use this extension, default _for.wmo
/^(-e)/ && do { # LDM doesn't pass a null, could get next arg
$ext = shift ;
if( $ext =~ m#^(-|\d|\s+)|/# ) {
unshift( @ARGV, $ext )
unless( $ext =~ m#^\s# ) ;
$ext = "" ;
}
} ;
/^(-a)/ && $dirStruct++ ; # write /stn/bulletin/report
/^(-l)/ && $logging++ ; # turn logging on
}
$cChars = 0 if( ! defined( $cChars ) ) ;
$dirStruct = 0 if( ! defined( $dirStruct ) ) ;
$ext = "_for.wmo" if( ! defined( $ext ) ) ;
# process input parameters
if( $#ARGV == 0 ) {
if( $ARGV[ 0 ] =~ /^[0-9]/ ) {
$yymm = $ARGV[ 0 ] ;
} else {
$datadir = $ARGV[ 0 ] ;
}
} elsif( $#ARGV == 1 ) {
$yymm = $ARGV[ 0 ] ;
$datadir = $ARGV[ 1 ] ;
} else {
die "Usage with input on STDIN: ldmConnect [-a] [-s] [-l] [-e ext ] [yymm] [datadir]\n";
}
# set interrupt handler
$SIG{ 'INT' } = 'atexit' ;
$SIG{ 'KILL' } = 'atexit' ;
$SIG{ 'TERM' } = 'atexit' ;
$SIG{ 'QUIT' } = 'atexit' ;
$SIG{ 'HUP' } = 'atexit' ;
# the directory
$datadir = "." if( ! $datadir ) ;
mkpath( $datadir, 0, 0775 ) if( ! -d $datadir ) ;
$io = new doIO() ;
$io->setLog( $datadir ) if( $logging ) ;
# year and month
if( ! $yymm ) {
my( $theyear, $themonth ) ;
$theyear = (gmtime())[ 5 ] ;
$theyear = ( $theyear < 100 ? $theyear : $theyear - 100 ) ;
$theyear = sprintf( "%02d", $theyear ) ;
$themonth = (gmtime())[ 4 ] ;
$themonth++ ;
$yymm = $theyear . sprintf( "%02d", $themonth ) ;
}
# Now begin parsing file and decoding observations breaking on cntrl C
$/ = "\cC" ;
# read in inprogress data from last process, fill in bulletin
$filename = $datadir . "/inProgress" . "$ext" ;
if( $io->inProgress( $filename, "read" ) ) {
while( 1 ) {
$io->inProgress( $filename, "data" ) ;
last if( !$_ ) ;
$part = new Part() ;
$bulletin = new Bulletin( $part ) ;
$header = $bulletin->fill( $_ ) ;
$inProgress{ "$header" } = $bulletin ;
undef( $part ) ;
}
$io->inProgress( $filename, "close" ) ;
}
# main loop set select processing here from STDIN
START:
while( 1 ) {
open( STDIN, '-' ) ;
vec( $rin, fileno( STDIN ), 1 ) = 1 ;
$timeout = 1200 ; # 20 minutes
$nfound = select( $rout = $rin, undef, undef, $timeout ) ;
# timed out
if( ! $nfound ) {
print STDOUT "Shut down, time out 20 minutes\n"
if( $logging ) ;
atexit() ;
}
atexit( "eof" ) if( eof( STDIN ) ) ;
$part = new Part() ;
# Process data first as part of a bulletin
$_ = <STDIN> ;
( $status, $header, $section ) = $part->getData( $_ ) ;
if( $status eq "OK" ) { # complete bulletin, write it out
$filename = $part->buildFN( $dirStruct, $datadir, $yymm ) ;
$io->setFH( $filename, $ext ) ;
$part->print( $cChars, $section ) ;
} elsif( $status eq "HEAD" ) { # create new bulletin
$bulletin = new Bulletin( $part ) ;
$inProgress{ "$header" } = $bulletin ;
} elsif( $status eq "PART" || $status eq "DONE" ) {
if( defined( $inProgress{ "$header" } ) ) {
$bulletin = $inProgress{ "$header" } ;
$bulletin->addPart( $section, $part ) ;
# combine parts and write
if( $status eq "DONE" || $bulletin->time() > 900 ) {
$bulletin->combine() ;
$filename = $bulletin->buildFN
( $dirStruct, $datadir, $yymm ) ;
$io->setFH( $filename, $ext ) ;
$bulletin->print( $cChars ) ;
delete( $inProgress{ "$header" } ) ;
}
} else { # catches weird PRE, PMD, etc suffices
$filename = $part->buildFN
( $dirStruct, $datadir, $yymm ) ;
$io->setFH( $filename, $ext ) ;
$part->print( $cChars, $section ) ;
}
}
undef( $part ) ;
atexit( "eof" ) if( eof( STDIN ) ) ;
} # end while( 1 )
atexit( "eof" );
exit( 0 ) ; # should never get here
# execute at exit
sub atexit
{
my( $sig ) = @_ ;
if( $sig eq "eof" ) {
print STDOUT "eof on STDIN --shutting down\n" if( $logging ) ;
} elsif( defined( $sig )) {
print STDOUT "Caught SIG$sig --shutting down\n" if( $logging ) ;
}
# check if bulletin is older than 15 minutes, if so write, else save in
# the inProgress file
$filename = $datadir . "/inProgress" . "$ext" ;
$io->inProgress( $filename, "write" ) ;
foreach $header ( %inProgress ) {
$bulletin = $inProgress{ "$header" } ;
if( defined( $bulletin ) ) {
if( $bulletin->time() < 900 ) {
$bulletin->dump() ;
} else {
$bulletin->combine() ;
$filename = $bulletin->buildFN
( $dirStruct, $datadir, $yymm ) ;
$io->setFH( $filename, $ext ) ;
$bulletin->print( $cChars ) ;
}
}
}
$io->inProgress( $filename, "close" ) ;
$io->Close( $ext ) ;
exit( 0 ) ;
}
Documentation for "ldmConnect"
The perl decoder "ldmConnect" pieces together segmented zone forecasts or
other segmented products with WMO segment identifiers PAA, PAB, etc.
For more information about the segmentation of large products and the
use of bulletin segment heading lines utilizing the Pxx Indicator Group,
see
http://www.nws.noaa.gov/tg/bbb.html
The "ldmConnect" script orders and combines bulletin segments in memory,
strips out extra headers, and then writes the completed bulletin. If
"ldmConnect" exits, it writes an "inProgress" file to maintain the current
state of processing. When restarted, "ldmConnect" reads the "inProgress"
file first. Non split reports are written directly.
Usage with input on STDIN:
ldmConnect [-a] [-s] [-l] [-e ext ] [yymm] [datadir]
-a locates bulletin in directory structure "station ID"/"bulletin ID"/
-s strips out control characters (other than newline)
-l turns on logging to create log files of the form "connectLog.<PID>.wmo"
-e sets the file extension, default is "_for.wmo". File format is:
yymmddhh + extension.
yymm is the current year and month. Current yymm is default.
datadir is the output directory for data, inProgress, and log files.
Typically "ldmConnect" is located in the decoders directory.
Typical pqact.conf entry:
# Forecasts in yymmddhh_for.wmo
# perl script ldmConnect, joins split bulletins, writes all bulletins
DDS|PPS ^F[PTXEQWZ].... .... ([0-3][0-9])
PIPE /usr/local/ldm/decoders/ldmConnect
(\1:yy)(\1:mm)
data/forecasts
|