[LDM #UEF-429372]: LDM - pqinsert error

> Institution: National Weather Service
> Package Version: 6.4.4
> Operating System: RHEL WS Ver. 4u2
> Hardware Information: Dual Xeon 3.0GHZ with 3GB DDR
> Inquiry: I am attempting to ldm a series of images to multiple sites (GIF).  
> My image sizes vary 1KB to 35KB and I receive the following error from some 
> of the smaller gif images (Seems to be an MD5 Sum Error)
> 
> pqinsert ERROR: Product already in queue: 8a9f53fd4c568dcda50a353adf929f84    
>  1798 20060202191503.218     EXP 000  
> /data2/www/html/ridge/Warnings/Long/FWS/FWS_20060202_1805_Long_Warnings.gif
> 
> I get this error even though the files are physically different.

The files might be physically different, but the data they contain are 
identical.  This is why the product-queue is rejecting the insertion.  
pqinsert(1) computes an MD5 checksum for the data-portion of every 
data-product.  If two data-products have the same MD5 checksum, then the odds 
of them having different data is about 1:2^128, against.

If the data-products are distinct, then they should have different MD5 
checksums.  For example, you could add a timestamp to the data to indicate when 
it was observed.  A simple counter or filename would also suffice -- anything 
that would make the data different.

Having different product-identifiers (i.e., filename) isn't enough because 
product-identifiers are used by the LDM system but not, in general, by decoders 
of the data.  They must be able to recognize data-products from the data itself 
and not rely on the product-identifier.

> Is there a way to force pqinsert to create a new value so that the image can 
> be sent through the ldm stream.  If I purge the queue and send this file 
> first it works, but when I send the next file it says it is already in the 
> queueu.

You could run pqexpire(1) on the queue, but that really won't help because the 
the downstream LDM will also reject data-products that have identical MD5 
checksums.

> Thank you for your time,
> Paul Kirkwood
Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: UEF-429372
Department: Support LDM
Priority: Normal
Status: Closed