[LDM #BMK-187963]: LDM Redundancy and Duplicated Files

Hi,

re:
> Thank you so much Tom and Steve, for your replies and clarification.
> I appreciate your prompt responses!  And I totally understand the typo.
> I do it all the time ????

:-)

re:
> I ran the Linux checksum command on both LDM A & LDM B on the same
> file.  The checksum outputs are identical

OK, but this does not guarantee that the MD5 signatures used by the
LDM will be the same.

re:
> (the 2 data servers are
> the same model & run the same version of Linux).

I still don't understand the setup there, sorry for being
a little dense.

Are you saying that there is a model that runs and its output
is sent to both LDM1 and LDM2 where each product is then
inserted into the LDM queue so it can be relayed to downstream
machines?  Or, are you saying that each machine is running
a model, and the output from each is being inserted into
the LDM queue for relay to downstreams?

In either case, what are you using to insert the model products
into the LDM queues?  For instance,, the LDM has a utility
'pqinsert' that is used for inserting products into an LDM
queue, but it can be told to calculate the MD5 signature in
two different ways, and those two ways will have different
MD5 signatures for the same product.  The first, and default,
way to calculate the MD5 signature is to use all of the bytes
in the product, and the second way is to use just the bytes
in the Product ID to calculate the MD5 signature.  What we
need to know is which method is being used on each machine.

re:
> Under which scenario would you think they will be different?

If one machine uses the default method for calculating the MD5
signature and the other uses the other method, then the signatures
will not match, so a downstream LDM that is receiving the products
from two upstreams will treat the products as being different.
I suspect this because of your original comment that you are getting
duplicates for products in the LDM queues on the downstream machines.

re:
> On the timing of the file arrival on LDM A & B, the files are
> transferred simultaneously from LDM 1 & 2 (i.e. within seconds -
> worst case).  I checked the LDM Max Latency value is set at 3600.

OK, this is good/important to know.  In most (approaching all) cases,
this should mean that the first copy of a received product should
still be in the downstream machine's LDM queue when the second
copy is received.  Nonetheless, it is still possible that this is
not the case.  Knowing the time history of the maximum queue residency
time would tell us if this could be happening or not.  This is why
I included the comments about the 'ldmadmin addmetrics' crontab
action in my first email.

If it would help to talk about your situation, we can do a Google
Meet with you.  I am available most times except today when I will
be involved in a GOES-R/HRIT/EMWIN user's group vMeeting from noon
until about 2:30 today.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: BMK-187963
Department: Support LDM
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.