[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: ldm dedup rule

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: RE: ldm dedup rule
  • Date: Mon, 25 Jun 2007 14:43:42 -0600
Ketan,

The MD5 checksum will provide duplicate detection for the duration
that the original product resides in your queue. The "pqmon" utility
will show the age of the oldest product in your queue. We try to advise
users to create a product queue that will hold at least 1 hours worth of
data depending
on how much data they are ingesting (the product queue size is
determined in 
the ldmadmin-pl.conf file when runing "ldmadmin mkqueue").

Steve Chiswell
Unidata User Support


On Mon, 2007-06-25 at 16:40 -0400, Rout, Minaketan wrote:
> Steve, My sincere than you for quick reply. 
> On the same note, what is the duration product queue holds the checksum
> value.
> 
> Example : Let's say I received a bulletin  now. After 10 min I received
> the same bulletin. Does LDM will dedup the last bulletin not going out? 
> 
> Thanks in advance.
> Ketan
> 
> -----Original Message-----
> From: Steve Chiswell [mailto:address@hidden] 
> Sent: Monday, June 25, 2007 4:36 PM
> To: Rout, Minaketan
> Cc: address@hidden; address@hidden
> Subject: Re: ldm dedup rule
> 
> 
> 
> 
> On Mon, 2007-06-25 at 15:28 -0400, Rout, Minaketan wrote:
> >  
> > 
> >  
> > 
> > We are receiving NOAAPort data via LDM. We also ingest WMO formatted
> > bulletin  manually using pqing. We like to apply dedup rule to make
> > sure if there is an same bulletin processed before. If so then the new
> > bulletin should get rejected.
> > 
> >  
> > 
> > How can I know LDM dedup rules? I would appreciate if you point me in
> > right direction.
> > 
> >  
> > 
> > 
> 
> Ketan,
> 
> Pqing by default for WMO bulletins will compute the MD5 checksum only on
> the data portion of the WMO bulletin (eg following the FOS start of
> product sequence ans sequence number). This is disabled using the "-5"
> option to allow the full product
> including the sequence number to be used in the MD5.
> 
> Ensure that your MD5 from your LDM source is calculating the MD5
> similarly 
> depending on the source of your LDM data stream. The Unidata IDD stream
> will match the default behavior of pqing. 
> 
> Steve Chiswell
> Unidata User Support
> 
> 
> 
-- 
Steve Chiswell <address@hidden>
Unidata