[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: MD5 checksum, deduplication, etc.

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: MD5 checksum, deduplication, etc.
  • Date: Thu, 10 Jun 2004 12:27:33 -0600
Stonie,

>Date: Thu, 10 Jun 2004 17:54:52 +0000
>From: Stonie Cooper <address@hidden>
>Organization: Planetary Data, Incorporated
>To: Steve Chiswell <address@hidden>,
>To: address@hidden
>Subject: MD5 checksum, deduplication, etc.

The above message contained the following:

> Have a quick question.  We have a config for streaming some data to LDM 
> (6.0.14) that sends the same product twice to the queue - but applies a 
> different feedtype and sequence number.
> 
> Unfortunately, the MD5 dedup catches the duplicate and throws it away.
> 
> Is there a way to turn deduplication off . . .

There's no way to disable the check of the MD5 checksum on the
downstream LDM without modifying the code.

> or use the feedtype and/or 
> sequence number in the MD5 to get a unique product?

Threre's no way for an LDM to change the MD5 checksum without modifying
the code.  Of course, the data-product ingester can do whatever it wants.

> It happens to be GINI 
> imagery . . . so applying a faux sequence number in a faux FOS header is not 
> an option.

Sounds like the ingester isn't under your control.

There might be another way to accomplish the larger task.  What are you
trying to do for which duplicate data-products appears to be a
solution?

> Thanks for any advise.
> -- 
> Stonie R. Cooper
> Planetary Data, Incorporated
> (402) 727-6599

Regards,
Steve Emmerson