[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: MD5 checksum, deduplication, etc.

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: MD5 checksum, deduplication, etc.
  • Date: 10 Jun 2004 12:27:14 -0600
Stonie,

The FOS header check flag for sequence number difference is
not an option for NIMAGE. The MD5 is not based on feedtype
or other product queue structure information so the
same product inserted more than once will be a duplicate.
One work around if you have control over this with your
stream is to append either a blank line or something
like a unique date stamp following the data of the
product that won't interfere with decoding/display of the image, but
will generate a unique MD5. This is what was done for the floater NIDS
images to make them distinct from the same products being carried in the
NNEXRAD feed.


Steve Chiswell
Unidata User Support


On Thu, 2004-06-10 at 11:54, Stonie Cooper wrote:
> Steve,
> 
> Have a quick question.  We have a config for streaming some data to LDM 
> (6.0.14) that sends the same product twice to the queue - but applies a 
> different feedtype and sequence number.
> 
> Unfortunately, the MD5 dedup catches the duplicate and throws it away.
> 
> Is there a way to turn deduplication off . . . or use the feedtype and/or 
> sequence number in the MD5 to get a unique product?  It happens to be GINI 
> imagery . . . so applying a faux sequence number in a faux FOS header is not 
> an option.
> 
> Thanks for any advise.