[LDM #TWA-232236]: Question regarding LDM request in ldmd.conf and bandwith.

Hi Mike,

re: 
> After doing some testing I think I have discovered the answer to my question
> but wanted to run it by you guys and make sure this is a correct conclusion.
> 
> When configuring the LDM's ldmd.conf file for multiple sources for primary
> and alternate feeds, such as the example listed on the Unidatas'  web site.
> 
> REQUEST ANY ".*" host1.site1.edu # initially primary-mode
> REQUEST ANY ".*" host2.site2.edu # initially alternate-mode
> 
> 
> My assumption was that the LDM compared only the MD5 checksums from the
> alternate site against the data flowing in from the primary and any product
> that primary did not get sent from the primary site would be sent from the
> alternate feed site. Also, I had assumed that the data sent to/from the
> alternate site, was low bandwidth data, MD5 checksum data only.

The following comments pertain to LDM 6.4+ distributions:

When the regular expressions in the replicated feed requests are identical, then
the slower feed process will fall back to being an alternate while the faster
stays as a primary.  The decision for this partitioning is made 
semi-continuously
so when the two upstream sites are able to feed the downstream equally fast,
the connections will likely change from alternate to primary and back (because
over the time period in which the decision is made, it is likely that products
come from one site).

When an upstream is feeding in alternate mode, it offers products to the
downstream one-by-one.  If the downstream has not already received the product
being offered, it will accept it.  If the number of products accepted
in one feed exceeds the number of products accepted by another feed, the
downstream (receiver) will force a disconnection from the upstream and request
a new feed that operates as a primary.  If the number of products received
by the downstream by a primary upstream feeder is less than another feed,
then the connection will be broken and a new, alternate feed request will
be sent to the upstream.  Please remember that this assumes that the set of
possible products being sent from each upstream is identical.
 
> My testing of bandwidth usage with Cacti SNMP graphs and iptraf, indicates
> that when you include an alternate site in the ldmd.conf file, you are
> *doubling* the incoming data volume, since both sites are sending the entire
> requested data sets, not just the MD5 comparison data from alternate site.

This should not be the case _unless_ the regular expressions for the data
being requested are identical.  For instance, if your example requests
were:

REQUEST ANY ".*" host1.site1.edu # initially primary-mode
REQUEST ANY "(.*)" host2.site2.edu # initially alternate-mode

the two requests would be different, and each feed would be established as
a primary.

> So my question is...Is it true by adding a alternate feed site you effective
> get two sets of the same data stream, ie double the bandwidth consumption?

It will be true if the regular expressions for the requested data products
are different.  If they are not, you should not see double the number of
products being sent.

> If this is true, I may not be the only LDM user that unaware of this and
> implications of bandwidth usage.

I believe that the LDM documentation describes how replicated feed requests
work.

> Thanks for your help and clarification on this issue.

No worries.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TWA-232236
Department: Support LDM
Priority: Normal
Status: Closed