[LDM #EAN-986598]: Relationship between Upstream and Downstream queues

Leonard,

> I'd like to understand the queues better.  I'm getting the idea about
> sizing the Upstream queue according the size of individual
> data-products, the rate they are inserted and the bandwidth heading
> downstream.  On the Downstream size, I think it should be sized
> according to Upstream and the rate for which they can be disposed of by
> pqact and whatever is triggered by that.

The product-queue should be sized so that any given data-product will reside in 
the queue for at least as long as the "maximum latency" parameter, which sets 
the maximum amount of time from data-product creation to data-product arrival 
that a downstream LDM will accept for a data-product (a downstream LDM will 
reject data-products -- and not put them into the product-queue -- if their 
latency is greater than the maximum latency parameter (which is, typically, 
3600 seconds).

Beginning with LDM 6.9, the ldmadmin(1) script can be set to automatically 
reconcile the size of the product-queue and the maximum latency parameter.

> I'm timing the insert according to the bandwidth, so that the need for
> queuing is minimized.  I worry that the network could be down, and the
> queue fills, and older data-products are deleted before being sent.
> I've seen that happen, indicated in the logs.  That's when I decided to
> time the inserts instead of inserting everything as quickly as possible
> (for one instrument we have, the data are sent overnight, and 100GB is
> online to be sent at the start, but I didn't want to create a queue of
> 100-200GB).

The LDM system is designed for a network that doesn't go down that often -- and 
not for long if it does. That is why the default maximum latency parameter is 
one hour.

If your network is down for longer periods, then you have a problem.

> I'm starting to understand the FEEDME and HEREIS, for example.  I see
> how Downstream requests data-products from Upstream.  But, shouldn't
> Downstream notify Upstream that it has received a particular
> data-product?  How do the two queues work together so as to not send the
> same data-product?

Each data-product has a unique MD5 checksum. The product-queue will reject any 
attempt to insert a data-product with an MD5 checksum identical to an existing 
data-product.

> Thanks.
> 
> --
> ==Leonard E. Sitongia
> High Altitude Observatory
> National Center for Atmospheric Research
> P.O. Box 3000 Boulder CO 80307  USA
> address@hidden  voice: (303)497-2454  fax: (303)497-1589

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: EAN-986598
Department: Support LDM
Priority: Normal
Status: Closed