Re: Sequence number use in pqact.conf??

On Wed, 20 Sep 2000, Michael W Dross wrote:

> 
> 
> Rob,
> 
> I configured the ldm's pqing to use the -5 flag in the ldmd.conf file. This
> system is feeding the WMO feedtype via socket from our NOAAPORT machine.
> Its not feeding WMO data from any other sources.
> 
> What I have noticed is it appears that the pqing is flagging products as "
> Product already in queue" when it really isn't. It appears to be
> ignoring the bulletin size, or at least it would appear.  I have attached a
> couple of hours of the pqing -v   log. The bulletin I noticed most
> often flagged is the "SXUS70 KWAL" since it comes across often.  In particular
> SXUS70 KWAL 200437 was
> flagged and could find another one prior that was with the same time and/or 
> size
> (604 bytes).
> 
> Take a look and let me know what you think. Maybe I am not understanding
> something...
> 
Mike,

Even though you are getting products from a single source there will still
be duplicates. The NWS resends products all the time.  Now with the -5
flag the difference of sequence numbers, ie 259 and 260 doesn't matter.

I checked the log, the SNPS40 product was flagged as a duplicate but it's
size is 97 bytes so it's a duplicate.  I checked about 10 products it was
the same scenario.

@Sep 20 05:16:19 pqing[26561]:       97 20000920051619.499     WMO 259
SNPS40
AMMC 200400 RRF
^@Sep 20 05:16:19 pqing[26561]:       97 20000920051619.504     WMO 260
SNPS40
AMMC 200400 RRF
^@Sep 20 05:16:19 pqing[26561]: Product already in queue    

I did a test with pqing by using the same rawfile twice and it flagged all
the duplicates correctly:

pqing -vl - -q data/ldm.pq -f HDS -5 <ingest source or rawfile>


You can also use pqcat to dump the LDM queue if you are thinking that a
product is missing. Look at the pqcat man page.

At this point I think the -5 flag is working correctly.

Robb...




> Thanks,
> 
> (See attached file: pqing.log)
> 
> Mike
> 
> 
> |--------+-------------------------->
> |        |          Robb Kambic     |
> |        |          <rkambic@unidata|
> |        |          .ucar.edu>      |
> |        |                          |
> |        |          09/18/2000 12:07|
> |        |          PM              |
> |        |                          |
> |--------+-------------------------->
>   
> >----------------------------------------------------------------------------|
>   |                                                                           
>  |
>   |       To:     Michael W Dross/Cust/DukePower@DukePower                    
>  |
>   |       cc:     address@hidden, support-ldm                      |
>   |       <address@hidden>                                       |
>   |       bcc:                                                                
>  |
>   |       Subject:     Re: Sequence  number use in pqact.conf??               
>  |
>   
> >----------------------------------------------------------------------------|
> 
> 
> 
> 
> 
> 
> On Sun, 17 Sep 2000, Michael W Dross wrote:
> 
> >
> >
> > I have searched and could not find the answer... So I turn to you guys  :-)
> >
> > Is there a way in defining a pqact.conf entry that will use the sequence
> > number?? I have tried with no luck. Consider the following example.
> >
> >
> > DDPLUS  ^WHXX04 (....) ([0-3][0-9])([0-2][0-9])([0-5][0-9])
> >         FILE    -overwrite 
> > data/tropical/GFDL_Hurricane_Guidance.(\2:mm)\2\3\4
> >
> Mike,
> 
> I looked at the pqact code about the feedtype and pattern matches, here's
> a comment that states the sequence number is NOT included in the match.
> 
> * If the feedtype matches AND ((the product ID matches the
> * regular expression) OR (the pattern is "_ELSE_" AND nothing
> * has been done to this product yet AND the first char of
> * the ident isn't '_'))
> 
> This is a problem because products coming from different NOAAport ingest
> systems will look differnt because of the different sequence number.  In
> the ldm-5.1.2 release we added a -5 flag to pqing to NOT include the
> sequence number in the md5 checksum.  Therefore, the checksum will just be
> on the data section of the product and it can designate these type of
> products as duplicates.  At this time, we have not implemented this
> because it needs some synchronization between all the NOAAport ingest
> source site.  I suspect it will happen in the next couple of months.
> 
> Robb...
> 
> 
> 
> >
> > Occasionally multiple bulletins with the same time/date heading are sent and
> it
> > gets appended to the file (-overwrite doesn't prevent it
> > for some reason) the only difference is the sequence number. I thought this
> > might work, but doesn't.
> >
> > DDPLUS  ^(...)  WHXX04 (....) ([0-3][0-9])([0-2][0-9])([0-5][0-9])
> >         FILE    -overwrite
> data/tropical/GFDL_Hurricane_Guidance.\1(\3:mm)\3\4\5
> >
> >
> > It seems the entry starts with WHXX04 and the sequence number cannot be
> > referenced??
> >
> > Any ideas/suggestions would be greatly appreciated!
> >
> > Mike Dross
> >
> >
> 
> ===============================================================================
> Robb Kambic                      Unidata Program Center
> Software Engineer III                 Univ. Corp for Atmospheric Research
> address@hidden         WWW: http://www.unidata.ucar.edu/
> ===============================================================================
> 
> 
> 
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================