[LDM #TUN-623089]: LDM Duplicate Products

Robert,

> I have a question regarding duplicate products...
> 
> Let me first describe our LDM processing...  we have one "gateway"
> server running pdinrs (Planetary Data) and LDM (6.10.1) that processes
> the NOAAPort stream received via satellite/Novra receiver.  This LDM
> instnace essentially acts as a relay to other downstream LDM servers.
> It builds an LDM queue (~2GB, product age is about 15-45 minutes),
> but does not run pqact.  Downstream, our primary LDM server (6.12.3)
> is responsible for pulling the products and writing them to file, etc.
> This also also has a 2GB queue with a product age of 90-120 minutes.
> 
> My understanding is that incoming product signatures are compared with
> the signature of products sitting in the queue.  That said, we seem to
> commonly experience duplicate products written to file -- a good example
> being the LAMP output for a particular hour may show up multiple times in
> an appended file.  Another example might be the temperature grid from the
> RTMA shows up 3 times in a particular model run.  This is particularly
> bad during times of high retransmission.  Using a tool like notifyme
> on the downstream server, I can see the same product sitting in queue,
> only differing by sequence number.  When I extract those 3 instances
> of the same product with pqcat, strip off the first 31 bytes, I am left
> with 3 identical files, proving (I believe) that the products themselves
> are identical.  So I am wondering if the product signatures are calculated
> with the sequence number?  If so, how can I avoid excessive duplication?
> Or am I looking at this wrong and is my problem the size of the upstream
> queue - should it be larger?

There are two problems: 1) the maximum latency parameter and the queue size 
parameters are inconsistent and need to be reconciled; and 2) the same 
data-product is being injected into the LDM network multiple times.

The first problem is this: when a data-product arrives, it is discarded if its 
latency (arrival-time minus creation-time) exceeds the value of the registry 
parameter "/server/max latency"; otherwise, an attempt is made to insert the 
product into the product-queue. The product will be rejected as a duplicate if 
a data-product with the same signature (i.e., MD5 checksum) already exists in 
the queue (the checksum is computed using the product's data: no metadata is 
involved). For this to be a valid test, the minimum residence time of a product 
in the queue *must* be greater than the maximum latency time; otherwise, a 
duplicate product could arrive that would, nevertheless, pass both tests.

The second problem is likely due to retransmission requests, which appear to be 
more numerous with the higher NOAAPORT bandwidth. The fact that the sequence 
numbers are different in the duplicate products indicates that the same product 
was transmitted twice.

The minimum residence time of a product in the queue can be seen with the 
command "pqmon -S" (see the manual page on this utility for an explanation).

The relationship between the queue size parameters and the maximum latency 
parameter can be seen or reconciled with the command "ldmadmin vetqueuesize". 
If the registry parameter "/reconciliation-mode" is set to "do nothing", then 
the command will inform you of any inconsistency. If, however, this parameter 
is set to "increase queue" or "decrease maximum latency", then the size of the 
queue will be increased or the maximum latency decreased, respectively. Be 
advised, however, that, due to uncertainties in the computation, such an 
automatic adjustment can be wildly off (resulting in an excessively large 
queue, for example) if the queue is grossly undersized to begin with.

I hope this helps.

> Best Regards,
> Bob

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: TUN-623089
Department: Support LDM
Priority: Normal
Status: Closed