Re: MD5 checksum, deduplication, etc.

Stonie,

The FOS header check flag for sequence number difference is
not an option for NIMAGE. The MD5 is not based on feedtype
or other product queue structure information so the
same product inserted more than once will be a duplicate.
One work around if you have control over this with your
stream is to append either a blank line or something
like a unique date stamp following the data of the
product that won't interfere with decoding/display of the image, but
will generate a unique MD5. This is what was done for the floater NIDS
images to make them distinct from the same products being carried in the
NNEXRAD feed.


Steve Chiswell
Unidata User Support


On Thu, 2004-06-10 at 11:54, Stonie Cooper wrote:
> Steve,
> 
> Have a quick question.  We have a config for streaming some data to LDM 
> (6.0.14) that sends the same product twice to the queue - but applies a 
> different feedtype and sequence number.
> 
> Unfortunately, the MD5 dedup catches the duplicate and throws it away.
> 
> Is there a way to turn deduplication off . . . or use the feedtype and/or 
> sequence number in the MD5 to get a unique product?  It happens to be GINI 
> imagery . . . so applying a faux sequence number in a faux FOS header is not 
> an option.
> 
> Thanks for any advise.