[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #KAM-474743]: LDM catching up

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [LDM #KAM-474743]: LDM catching up
  • Date: Fri, 20 Apr 2007 16:07:32 -0600
Carl,

> I've noticed that since we've upgraded from the 6.4.x to 6.5.1, LDM now
> does it's catch up differently.  For example, if the LDM server goes off
> the network for an hour and then comes back online, the client has to
> catch up to the latest data in the queue.  In 6.4.x, the oldest file in
> the queue (not yet received by the client) would be sent to the client
> first; once that file was complete the next file in line would be sent
> and so-on until the queue was sending current data.  In 6.5.1, when the
> server comes back online, the client appears to receive several files
> all at once (in parallel) but if there are bandwidth limitations, then
> the files trickle in a little bit at a time.  The problem with the
> parallel method is this: a partial file cannot be fully processed until
> it arrives; also, we've seen newer files arrive completed before older
> files.  When algorithms are dependent on data that arrives in a
> time-sequence, this can cause problems.
> 
> My question is: can we turn this "feature" off?  We'd like to have the
> files arrive one at a time in the proper time-sequence...

An upstream LDM 6.5 process uses a different algorithm
than an upstream LDM 6.4 process to decide when to flush
the connection to a downstream LDM process.  The algorithm
can result in a burst of data-products.  The products will
still be in the same order, however.  The algorithm was
further modified in LDM 6.6.3 to make the flushing less
bursty.  I suggest that you upgrade to that version.

In all versions of the LDM, data-products are sent in
the order in which they were inserted into the product-
queue.  It's possible that you are receiving the same
data-products via more than one upstream site.  This 
kind of multiple-path delivery can result in data-
products that appear to be received out of order.

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: KAM-474743
Department: Support LDM
Priority: Urgent
Status: Closed