[LDM #GUQ-669331]: Both Radar and Satellite feeds stopped downloading abruptly

Hi John,

re:
> When you say smaller or larger products, can you tell me if that means a 
> different
> transfer method via TCP?  Or, simply just a matter of shear volume?

There is no difference in the LDM transfer method.  To me, at least, the product
size could be important if there are a lot of re-transmits (for whatever
reason) as it would take longer for a product to be received, and the transfer
of products for each/every ldmd connection are serialized.

Speaking of serialization, one approach to mitigating the receipt latency
of high volume feeds is to split a single ".*" REQUEST into multiple, disjoint
REQUESTs the union of which is the full set of data.  For feeds like
NIMAGE, this is very easy as there are subsets that are easily identifiable
(e.g., GOES16 is part of each NIMAGE Product ID for GOES-16 imagery
and GOES17 is part of each NIMAGE Product ID for GOES-17 imagery; there
are 4 distinct coverages of CONUS, FullDisk, Mesoscale-1, and Mesoscale-2;
etc.; etc).  Feeds like NGRID, on the other hand are difficult to split
into disjoint subsets as their Product IDs don't easily lend themselves
to splitting.  The best example for a feed that is easy to split is
CONDUIT since each product sent in the CONDUIT feed has a sequence number
as the last element of the Product ID.  This allows for easy split into
2, 5, 10, etc.

The same notion of splitting high volume feeds into subsets also works
for the case where there is some form of bandwidth limiting on a per
connection basis but the overall available bandwidth is not limited.

re:
> When I watch the "ldmadmin watch" I see numerous product entries for NEXRAD 
> and
> IDS|DDSPLUS and very few entries for NGRID, NIMAGE, and NOTHER.

Correct.  The reason for this is that the latencies for the NGRID, NIMAGE
and NOTHER products is large enough that they exceed the 
<max-latency></max-latency>
setting in the LDM registry, ~ldm/etc/registry.xml.  One can increase the
<max-latency> value over its 3600s (1 hour) default, but this does not
guarantee that the upstream that is sending the product will have a large
enough LDM queue so that the product will still be in the queue after
whatever value the <max-latency> is increased to.

re:
> Note, when I run "iptraf-ng" or a wireshark capture, I do see about 1/3 to 1/4
> the amount of traffic reaching a server during the "traffic shaping" period
> compared to normal function times.

OK, this is consistent with what my thinking was to begin with.  I say this
because other sites being fed the same feed(s) as the ones that are being
REQUESTed by mammatus were showing near-zero latencies at the same time
that mammatus was reporting latencies that were greater than an hour.
If you recall, I noted that in one of my previous emails.

By the way, you never did answer my question about why rime only has a
100 Mbps connection: is it the Ethernet interface(s) on rime, or is
it that you are connected to a 100 Mbps port on a switch?

One last comment:

We recommend keeping a REQUEST for IDS|DDPLUS in the LDM configurations
on both mammatus and meso-file1.  The reason is that low latencies seen
for products in this feed when compared to latencies seen in higher
volume feeds is a good rule-of-thumb indicator for intentional/unintentional
shaping of feeds, and its volume is low enough that it will not have
much, if any, affect on the max residency time of products in the local
LDM queue.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: GUQ-669331
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.