20030805: differences between LDM 5 and 6

Karen,

> From: "Karen Cooper" <address@hidden>
> To: <address@hidden>
> Sent: Tuesday, August 05, 2003 8:52 AM
> Subject: LDM question

The above message contained the following:

> About the differences between LDM 5 and 6.  If I understand correctly 
> according to your webpage
> 
> http://my.unidata.ucar.edu/content/software/ldm/ldm5vs6.html
> 
> when the upstream machine sends a message it does not wait for a reply 
> from the downstream machine prior to sending the next available 
> message.

That's correct.

> Therefore, if there is a problem with the first message and it 
> has to be resent, then the downstream machine will receive them out of 
> order.

That's not correct.  The data-products are sent over a TCP connection.
The TCP layer is responsible for retransmission of data packets and
guarantees that bytes are received in the same order in which they were
sent.

If the TCP connection is closed somehow, then the downstream LDM will
reconnect to the upstream LDM and ask for the same set of data-products
-- starting with the last data-product that it received as given by the
timestamp on that last data-product.

> If this is true, then it may be causing us problems with WSR-88D Level 
> II data.  Because the Level II data is sent in 100 radial packets which 
> are accumulated into a volume scan file, if they are out of order the 
> "late/resent" packet never gets concatenated to the file and is 
> therefore missing. 

Are these data outages coincident with LDM reconnections?  If not (i.e.,
if the connection between the two LDM-s was good during the data loss)
then it is EXTREMELY unlikely (verging on inconceivable) that the LDM
is responsible.  It is also very unlikely that a disconnection and
reconnection would be responsible -- although I could possibly contrive
a scenario involving bad data-product timestamps that might accomplish
this.

[ASIDE: It's never made sense to us that the radar products are packaged
into (at most) 100-radial data-products.  It would be more efficient
and less risky to package an entire sweep into a data-product (i.e.
all radials from one 360 degree scan).  The larger size would use the
bandwidth more efficiently and the obvious integrity of the data-product
should simplify the processing algorithms.  It could be just this
complexity that is the root cause of the problem you're encountering.
Please take this as well-intentioned puzzlement.]

> I think this is happening with some of our data, since our ingest 
> machine has complete files but downstream machines sometimes have large 
> chunks of data missing.

This should be investigated further.  Can you identify an example?  Can
you send me the logfile entries surrounding that incident?  Or, better
yet, can I log onto the system(s) in question?

> If it's true, is there some kind of flag that 
> could be added which would allow us to delay the sending of packets 
> until a reply is seen so that we can keep things in order?

You could declare the connection to the upstream LDM to be an ALTERNATE
one rather than a default PRIMARY one in the LDM configuration-file
(etc/ldmd.conf).  This will cause the upstream LDM to ask the downstream
LDM if it wants a given data-product -- for every data-product -- before
sending it.  This will, effectively, make the connection synchronous at
the cost of greatly reduced throughput and with no conceivable benefit.

This assumes that the upstream LDM is version 6.  If, instead, the
upstream LDM is version 5, then nothing will change because a downstream
LDM 6 uses the LDM 5 protocol when receiving from an upstream LDM 5.

I suspect, however, that the problem lies outside the LDM and that if
you use this "solution", then you might only hide the real problem and
risk having it rise-up and bite you again at a later time.

> address@hidden
> Phone:405-366-0434 
> Cell:405-834-8559
> CIMMS/University of Oklahoma
> Affiliate of National Severe Storms Laboratory

Regards,
Steve Emmerson
LDM Developer