[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #ZZW-504319]: LDM catching up

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [LDM #ZZW-504319]: LDM catching up
  • Date: Thu, 12 Apr 2007 12:22:59 -0600
Carl,

> I've noticed that since we've upgraded from the 6.4.x to 6.5.1, LDM now
> does it's catch up differently.  For example, if the LDM server goes off
> the network for an hour and then comes back online, the client has to
> catch up to the latest data in the queue.  In 6.4.x, the oldest file in
> the queue (not yet received by the client) would be sent to the client
> first; once that file was complete the next file in line would be sent
> and so-on until the queue was sending current data.  In 6.5.1, when the
> server comes back online, the client appears to receive several files
> all at once (in parallel) but if there are bandwidth limitations, then
> the files trickle in a little bit at a time.  The problem with the
> parallel method is this: a partial file cannot be fully processed until
> it arrives; also, we've seen newer files arrive completed before older
> files.  When algorithms are dependent on data that arrives in a
> time-sequence, this can cause problems.
> 
> My question is: can we turn this "feature" off?  We'd like to have the
> files arrive one at a time in the proper time-sequence...

I suggest upgrading to LDM 6.6.3.  A new feature of 6.6 is the
addition of a persistent "state" file for each downstream
LDM.  The purpose of the file is to contain information
on the last, successfully-received data-product for that
particular request.

By the way, if an LDM receives the same data-products via
multiple REQUEST entries to different upstream sites, then
there's no way to guarantee that the data-products are
received in the order in which they're created at the
source site.

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: ZZW-504319
Department: Support LDM
Priority: Normal
Status: Closed