[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #VTV-967500]: 'Duplicate' data from two LDM servers

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [LDM #VTV-967500]: 'Duplicate' data from two LDM servers
  • Date: Thu, 29 May 2014 13:52:50 -0600
Brice,

> We are running 6.11.6.

Good.

> I think I understand your answer about latency and queue size, but I have a 
> question about implementation.  If the upstream feeders are running only 
> small amounts of data through their queue, i.e. not full NOAAPort for 
> instance just some specific local implementation, how am I going to know how 
> to balance the size?

I'm not sure what you're asking. The upstream LDM-s will need a queue size that 
ensures that none of their data-products resides in their queue longer than 
their LDM's maximum latency setting -- so that they (the upstream LDM-s) can 
reject incoming duplicates. If the upstream LDM-s don't make any REQUEST-s for 
data-products, then their queues can be as small as the amount of time that 
they could be offline. Our NOAAPORT ingest LDM-s are in this latter category 
and their queues can typically guarantee only the last 15 minutes of data. (We 
run redundant NOAAPORT ingest LDM-s, so we don't care if their queues are 
small.)

If, instead, your talking about the size of the downstream LDM's queue, then it 
needs to be large enough so that the minimum residence time of a data-product 
in the queue is greater than the maximum latency parameter of that LDM (which 
is one hour by default). This needs to be the case regardless of the number or 
configuration of the downstream LDM's REQUEST entries.

> And, could the LDM settings and checks you propose handle that sort of 
> asymmetry?

Asymmetry of feeds is irrelevant: the only things that matter are the minimum 
residence time in the queue (regardless of feed) and the maximum latency 
parameter. If the former is greater than the latter, then duplicates 
(regardless of feed) will always be rejected.

> Thanks,
> 
> Brice

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: VTV-967500
Department: Support LDM
Priority: Normal
Status: Closed