20030217: LDM 6 release candidate tests at OU/CAPS (cont.)

>From: Carl Sinclair <address@hidden>
>Organization: CAPS
>Keywords: 200302172141.h1HLfK425607 LDM 6 CRAFT

Carl,

>I'm the contact for CRAFT LDM technical issues.

OK, this is good to know.  I was going to try and talk with Kevin
Kelleher at the end of the FOS meeting at the AMS in Long Beach, but
when I went to look for him, he was gone (smart move :-).

>As it turns out, I
>installed 6.0.0.9.  I've since installed 6.0.0.17.  It did have some
>issues during compile time, but the data is flowing.

The only problem with 17 was that configure was not setup to create
a Makefile in the src/regex directory.  Since this is a new addition
to LDM 6 and is not needed for its operation (it is used for tuning
regular expressions), it is not a problem.

>I'll update to 18
>when I see it available.  Since I installed an early version, I'll
>assume that was the problem and we'll see what happens with the newer
>one.  

18 will likely just have the configure fix for regex, so you can
probably skip it.  I will let you know if more mods are rolled into
it.

>As for the compression of 100 radials, I believe that was determined to
>be the most efficient size for the quickest transfer of the data.

I understand, but I am still under the impression (undoubtedly, through
lack of information) that it was chosen so that the LDM5 could get the
block of data out to the client as soon as possible, not so that the
product could be created as fast as possible at the inject location.

The LDM5 approach was to send products smaller than 16384 in one
blocked RPC transaction.  Products larger than 16384 bytes were broken
up into chunks of 16384 bytes each and the chunks were each sent with
blocking RPC transactions.  If a 100 radial compressed product happens
to be smaller than the LDM5 magic number of 16384, then the product
would be sent in one shot with a blocking RPC call.

The major difference between LDM5 and 6 is that LDM5 used blocking RPCs
to send products or chunks of products.  LDM 6, on the other hand,
sends an entire product with a non-blocking RPC.  For products bigger
than a user-specified size, the upstream host asks the requestor if it
wants the product.  If the answer is yes, the entire product is sent.
The net effect of the change is that the LDM 6 does not spend much if
any time waiting to receive notification of receipt of a product by a
downstream requestor.  As you might expect, this greatly speeds up the
movement of data especially when the sending and receiving machines are
electronically distant from one another.  We are using this capability
to move realtime meteorological data (obs, satellite imagery, etc.) to
a site at the Universidade Federal do Para in Belem, Brazil with great
success.

>Most
>clients we've had discussions with (some here in CAPS) want the data
>asap, and if part of an elevation scan can get here faster than the
>whole elevation scan, then that's what they want.

How much longer does it take to get a full scan from a NEXRAD than 100
radials?  I realize that it will take 3.65 times the amount of time to
accumulate the data, but I don't know if this is a fraction of a
second, several seconds, minutes, etc.  If the answer is a fraction of
a second to a couple of seconds, then I would venture to guess that the
decrease in the latency by moving from LDM5 to LDM 6 would more than
make up for the time spent waiting for a scan to finish.  The real
benefit would then be that an entire scan would be in hand and usable.
Of course, if the time to finish a scan at one elevation is several
seconds, sending smaller chunks would still be the best way to get some
of the data there ASAP.

>It is certainly
>possible to compress a whole elevation scan at one time, but the issue
>was waiting for the scan to complete.  Kelvin may be able to elaborate
>further on this issue.

OK.  I just wanted to make sure that you folks were aware of the
new possibilities with LDM 6.

>I'll keep everyone updated on how the newer versions operate.

Excellent.  It is always good to have multiple people testing a
new release (candidate) in different ways.

Tom

>From address@hidden Mon Feb 17 16:16:34 2003
>To: address@hidden
>From: Kelvin Droegemeier <address@hidden>
>Subject: Re: 20030217: LDM 6 release candidate tests at OU/CAPS
>Cc: address@hidden, Kelvin Droegemeier <address@hidden>,
>   Kevin Kelleher <address@hidden>

Carl and Tom,

With regard to the 100 radial compression, Carl is right....numerous
tests were done, and based on them and the latency issue, 100 radial
packages were viewed as the way to go.

Kelvin