[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20040610: MD5 checksum, deduplication, etc.

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: 20040610: MD5 checksum, deduplication, etc.
  • Date: Fri, 11 Jun 2004 11:58:47 -0600
Stonie,

The LDM does not calculate the signature in transmission. It is
only calculated by the host origin at insertion.

It is somewhat like a contract, the downstream sites take it on faith that
the MD5 is valid and unique. So long as all sites concerned understand 
what the limitations on the data stream/MD5 are, you are ok.
If you send the unaltered data with an MD5 that was arrived at from a modified 
product, the only problem would be that someone couldn't arrive at the same 
signature later if reinserting, without modifying the calculation scheme 
to match the original.

Steve Chiswell
Unidata User Support




>From: Stonie Cooper <address@hidden>
>Organization: Planetary Data, Incorporated
>Keywords: 200406101852.i5AIqntK009428

>Steve C.,
>
>To expand on this a little more . . . does the original MD5 checksum 
>calculated follow the product, or is recalculated with each queue insertion?  
>My thought is that I could do a MD5Update() by adding feedtype/sequence 
>number to the end, do the MD5Final, but actually only pass the product 
>buffered, without the extra string.
>
>If the MD5 is calculated at each server, though, it would only be good for the
>  
>first node.
>
>Stonie
>
>On Thursday 10 June 2004 18:27 pm, Steve Chiswell wrote:
>> Stonie,
>>
>> The FOS header check flag for sequence number difference is
>> not an option for NIMAGE. The MD5 is not based on feedtype
>> or other product queue structure information so the
>> same product inserted more than once will be a duplicate.
>> One work around if you have control over this with your
>> stream is to append either a blank line or something
>> like a unique date stamp following the data of the
>> product that won't interfere with decoding/display of the image, but
>> will generate a unique MD5. This is what was done for the floater NIDS
>> images to make them distinct from the same products being carried in the
>> NNEXRAD feed.
>>
>>
>> Steve Chiswell
>> Unidata User Support
>-- 
>Stonie R. Cooper
>Planetary Data, Incorporated
>(402) 727-6599
>
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.