Re: pqsurf.conf

Alan,

Good news, just what I expected. Your headers are a little different from
the standard NOAAport headers.  I wrote a perl script to modify the
headers to the correct format, that can be used in pqact.conf.  The script
is an attachment, called modHdrs. This is an example pqact entry, the -d
flag is for a directory

IDS|DDPLUS      ^(......) (....) (......)
        PIPE    modHdrs -d data/sao \1\2\3

modHdrs writes out a file with the correct header format.

Robb...





On Thu, 2 Dec 1999 address@hidden wrote:

> 
> 
> This is an excerpt from the ldmd.log:
> 
> Dec 02 15:25:18 npfos pqact[10850]: Successfully read pattern file
> "/usr/local/ldm/etc/pqsurf.conf"
> Dec 02 15:59:43 npfos pqsurf[10849]:     2325 19991202155928.888     IDS
> 000  SAUS70 KWBC 021600 1999336155812572
> Dec 02 15:59:43 npfos pqsurf[10849]: surface_split: Can't handle
> MESSAGE_TYPE_UNKNOWN
> 
> >From your previous message, this looks like what pqsurf is expecting.  I'm
> still not finding why pqsurf chokes with
> MESSAGE_TYPE_UNKNOWN.
> 
> I know I'm your worst nightmare...Alan.
> 
> 
> 
> 
> 
> Robb Kambic <address@hidden> on 11/29/99 04:01:49 PM
>                                                               
>                                                               
>                                                               
>  To:      Alan Hall/NCDC                                      
>                                                               
>  cc:      support-ldm <address@hidden>          
>                                                               
>                                                               
>                                                               
>  Subject: Re: pqsurf.conf                                     
>                                                               
> 
> 
> 
> 
> 
> 
> 
> 
> On Mon, 29 Nov 1999 address@hidden wrote:
> 
> >
> >
> > I have spent a frustrating two days trying to come up with a work around.
> > I have two possibilities:
> >
> > 1) file the products as they come in a file more like what pqsurf would
> be
> > expecting and then re-queue those files into the ldm.pq.  I can store the
> > files with a filename of: NWSTG TEXT SAUS70 KDEN 281536 1999333154000765
> > Getting that inserted into the queue is the next trick and I have
> succeeded
> > at that by using pqinsert as follows:
> >     pqinsert -f WMO 'NWSTG TEXT SAUS70 KDEN 281536 1999333154000765'
> > pqsurf still balks with Can't handle MESSAGE_TYPE_UNKNOWN.  I have also
> 
> Alan,
> 
> That's a good idea to re-queue the products but you got the header wrong,
> it should start with the "SAUS70 KDEN 281536 1999333154000765".  Here's
> the output from ldmadmin watch on the products:
> 
> Nov 29 20:48:02 pqutil:     1271 19991129204801.533 IDS|DDPLUS 420  SRUS58
> KWBC 292044 /pRRSMT
> Nov 29 20:48:02 pqutil:      147 19991129204801.542 IDS|DDPLUS 421  SRUS51
> KWBC 292044 /pRRSHOU
> Nov 29 20:48:02 pqutil:      826 19991129204801.553 IDS|DDPLUS 422  SRUS57
> KWBC 292044 /pRRSMS
> Nov 29 20:48:02 pqutil:      197 19991129204801.558 IDS|DDPLUS 423  SRUS60
> KWBC 292044 /pRRSMHX
> 
> I would also check surf_split.c to see what the header it's expecting.
> I think the routine to look at is: pqing/wmo_header.c:get_wmo_header
> 
> I think this will work with a little more investigation.
> 
> 
> Robb...
> 
> 
> 
> 
> > attempted to use the filename with the last 16 character date/time stamp
> > omitted at the end still get the same results.
> > I can get this work if I type in the pqinsert command manually, but if I
> > put into pqact.conf I get 'child exited with status 127' and I can't
> figure
> > out what that means and the product never gets inserted.
> >
> > 2) modify the ingest code to insert it the right way from the begining.
> I
> > am still investigating this, but with the tests from above I can't get
> > pqsurf to work even if I do have it inserted into the queue as it is
> > expecting it.
> >
> > If I could just get pqsurf to work so I can see what it's going to do, I
> > could make all the code changes needed.   I am completely blind about
> what
> > pqsurf is expecting, what it will do, what the pqsurf.conf file should
> look
> > like, what the 'standard WMO type of header' is supposed to look like?
> >
> > As you can see I'm getting very frustrated.
> >
> > Alan.
> >
> >
> >
> >
> >
> >
> > Robb Kambic <address@hidden> on 11/23/99 03:36:30 PM
> >
> >
> >
> >  To:      Alan Hall/NCDC
> >
> >  cc:      support-ldm <address@hidden>
> >
> >
> >
> >  Subject: Re: pqsurf.conf
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Alan,
> >
> > I looked into the pqsurf code, here's the scoop. The pqsurf code is
> > looking for the standard WMO type of header before it can split the
> > bulletin into individual reports. There are 2 solutions:
> >
> > - modify your ingest code to produce standard WMO type headers, ie
> > eliminate the NOAAPORT prefix and change the "." to spaces.
> >
> > - modify the surf_split.c code in the LDM pqsurf directory to accept your
> > current type headers.  At this time, I have many other top priority
> > projects so I could not get to this for months. Maybe you have the
> > resources to do the modifications.
> >
> > I know this is not what you wanted to hear but that's about it in a
> > nutshell. Maybe Leslie could help you out.
> >
> > Robb...
> >
> > On Mon, 22 Nov 1999 address@hidden wrote:
> >
> > >
> > >
> > > I think we are making progress.  I'm getting some entries in the
> ldmd.log
> > > that tell me we are getting things to pqsurf.  Here is a sample of the
> > log:
> > >
> > > Nov 22 13:30:15 npfos pqsurf[9283]: Starting Up (9279)
> > > Nov 22 13:30:15 npfos pqsurf[9283]: lastmatch:
> > > 6453cb112c44cf491c21769dee1f4138      987 19991122133008.749     WMO
> 000
> > > NOAAPORT.NW
> > > Nov 22 13:30:15 npfos pqsurf[9283]: 19991122123015.215 TS_ENDT {{WMO,
> > > "^NOAAPORT\.NWSTG\.TEXT\.S[AIMNP]"}}
> > > Nov 22 13:30:15 npfos pqact[9284]: Starting Up
> > > Nov 22 13:30:15 npfos pqact[9284]: Successfully read pattern file
> > > "/usr/local/ldm/etc/pqsurf.conf"
> > > Nov 22 13:30:15 npfos pqact[9284]: TS_ZERO TS_ENDT {{WMO,  ".*"}}
> > > Nov 22 13:30:15 npfos pqact[9284]:        0 19991122133015.238     ANY
> > 000
> > >   _BEGIN_
> > > Nov 22 13:30:17 npfos localhost[9291]: Connection from localhost
> > > Nov 22 13:30:17 npfos localhost[9291]: Connection reset by peer
> > > Nov 22 13:30:17 npfos localhost[9291]: Exiting
> > > Nov 22 13:30:30 npfos pqsurf[9283]:      103 19991122133016.828     WMO
> > 000
> > >   NOAAPORT.NWSTG.TEXT.SPCN51.CWAO.221327.1999326133016827
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]:      121 19991122133016.830     WMO
> > 000
> > >   NOAAPORT.NWSTG.TEXT.SPCN31.CWAO.221326.1999326133016830
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]:      139 19991122133016.832     WMO
> > 000
> > >   NOAAPORT.NWSTG.TEXT.SPCN41.CWAO.221327.1999326133016832
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]:      111 19991122133016.841     WMO
> > 000
> > >   NOAAPORT.NWSTG.TEXT.SPUS43.KWBC.221328.1999326133016840
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]:      586 19991122133016.843     WMO
> > 000
> > >   NOAAPORT.NWSTG.TEXT.SPUS70.KWBC.221328.1999326133016843
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]:      144 19991122133016.845     WMO
> > 000
> > >   NOAAPORT.NWSTG.TEXT.SPXX60.KWBC.221328.1999326133016844
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]:      115 19991122133019.927     WMO
> > 000
> > >   NOAAPORT.NWSTG.TEXT.SAPL61.KRAK.221330.1999326133019926
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]:      545 19991122133024.005     WMO
> > 000
> > >   NOAAPORT.NWSTG.TEXT.SNVB20.RJTD.221300.1999326133024005
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]:      107 19991122133026.080     WMO
> > 000
> > >   NOAAPORT.NWSTG.TEXT.SPCN48.CWAO.221326.1999326133026080
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > > Nov 22 13:30:30 npfos pqsurf[9283]:      133 19991122133026.083     WMO
> > 000
> > >   NOAAPORT.NWSTG.TEXT.SACN87.CWAO.221325.1999326133026083
> > > Nov 22 13:30:30 npfos pqsurf[9283]: surface_split: Can't handle
> > > MESSAGE_TYPE_UNKNOWN
> > >
> > > What is MESSAGE_TYPE_UNKNOWN and how can I correct it?
> > >
> > >
> > >
> > >
> > >
> > > Robb Kambic <address@hidden> on 11/19/99 04:11:06 PM
> > >
> > >
> > >
> > >  To:      Alan Hall/NCDC
> > >
> > >  cc:      support-ldm <address@hidden>
> > >
> > >
> > >
> > >  Subject: Re: pqsurf.conf
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Fri, 19 Nov 1999 address@hidden wrote:
> > >
> > > >
> > > >
> > > > I have the following entry in my pqsurf.conf file (this very line
> works
> > > > perfectly in pqact.conf):
> > > >
> > > > WMO
> > > >
> > >
> >
> ^NOAAPORT\.NWSTG\.(TEXT)\.([A-Z]...).[0-9]\.[A-Z]..[A-Z]|[0-9]\.([0-3][0-9]
> > > > )....\.
> > > >      FILE -strip
> > > >      /npraid/noaaport/data/NWSTG/\1/SURF/\2(\3:yyyy)(\3:mmm)(\3:dd)
> > >
> > > Alan,
> > >
> > > I reread the man page, it seems there are flags to set the feedtype and
> > > pattern. use the following flags on the pqsuf line in ldmd.conf.
> > >
> > > exec pqsurf -f WMO -p "^NOAAPORT\.NWSTG\.(TEXT)\.S[AIMNP]"
> > >
> > > Robb...
> > >
> > >
> > >
> > > >
> > > > tabs separate all the fields.  I can't get anything from pqsurf.  I
> am
> > > > completely lost on what this is supposed to do...
> > > > What should an entry in pqsurf look like?
> > > >
> > > > Alan.
> > > >
> > > >
> > >
> > >
> >
> ===========================================================================
> > > ====
> > > Robb Kambic                      Unidata Program Center
> > > Software Engineer III                 Univ. Corp for Atmospheric
> Research
> > > address@hidden         WWW: http://www.unidata.ucar.edu/
> > >
> >
> ===========================================================================
> > > ====
> > >
> > >
> > >
> > >
> >
> >
> ===========================================================================
> > ====
> > Robb Kambic                      Unidata Program Center
> > Software Engineer III                 Univ. Corp for Atmospheric Research
> > address@hidden         WWW: http://www.unidata.ucar.edu/
> >
> ===========================================================================
> > ====
> >
> >
> >
> >
> 
> ===========================================================================
> ====
> Robb Kambic                      Unidata Program Center
> Software Engineer III                 Univ. Corp for Atmospheric Research
> address@hidden         WWW: http://www.unidata.ucar.edu/
> ===========================================================================
> ====
> 
> 
> 
> 
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================

#!/usr/local/bin/perl
#
#  Program to modify headers and trailers of feedtype to look like the 
#  old FOS format.  
#
#  usage: 
#
#  IDS|DDPLUS   
#  written by Robb Kambic 12/3/99
#
# process command line switches, no valid switches at this time
while ($_ = $ARGV[0], /^-/) {
         shift;
       last if /^--$/;
             /^(-v)/ && $verbose++;
             /^(-d)/ && ($dir = shift) ;
}
$dir = "." if( ! $dir ) ;

# process input parameters
if( $#ARGV == 0 ) {
        open( STDOUT, ">$dir/$ARGV[ 0 ]" ) || die "could not open $ARGV[ 0 ] 
$!\n" ;
}
select( STDOUT ) ; $| = 1 ;

# set interrupt handler
$SIG{ 'INT' }  = 'atexit' ;
$SIG{ 'KILL' }  = 'atexit' ;
$SIG{ 'TERM' }  = 'atexit' ;
$SIG{ 'QUIT' }  = 'atexit' ;

# Now begin parsing stream and breaking on \cC
$/ = "\cC" ;
# main loop   set select processing here from STDIN
START:
while( 1 ) {
        open( STDIN, '-' ) ;
        vec($rin,fileno(STDIN),1) = 1;
        $timeout = 1800 ; # 30 minutes
        $nfound = select( $rout = $rin, undef, undef, $timeout );
        # timed out
        if( ! $nfound ) {
                print STDERR "Shut down, time out 30 minutes\n" ;
                atexit() ;
        }
        atexit( "eof" ) if( eof( STDIN ) ) ;

        $_ = <STDIN> ; # get product
        #
        # Check and modify NOAAport headers 
        #
        s#^.*\cA.*\cA....#\cA\cM\r\n999 \cM\r\n# ; 
        #
        print $_ ; # write product
        #
        atexit( "eof" ) if( eof( STDIN ) ) ;
} # end while( 1 )
atexit( "eof" );
exit( 0 ) ; #should never get here

# execute at exit
sub atexit
{
local( $sig ) = @_ ;

if( $sig eq "eof" ) {
        print STDERR "eof on STDIN -- shutting down\n" ;
} elsif( defined( $sig )) {
        print STDERR "Caught SIG$sig -- shutting down\n" ;
}
close( STDOUT ) ;
close( STDERR ) ;
exit( 0 ) ;

}