20021116: using the LDM to feed a remote site in Hong Kong (cont.)

>From: David YEUNG <address@hidden>
>Organization: CCAR HKUST
>Keywords: 200211142212.gAEMCPL03300 IDD LDM

Hi David,

>This is the information regarding our LDM:
> fully qualified hostname
>
>rcz006.ust.hk
>
> LDM version
>
>ldm-5.1.4
>
> fully qualified name of the upstream host
>
>iita.rap.ucar.edu
>
> datastreams you are trying to feed
>
>WMO
>
> > o copies of your ldmd.conf and pqact.conf files
>
>See the attached files.
>
> > o what operating system (vendor, release, etc.) you are running on
> >   your LDM machine
>
>RedHat 7.2
>
>[ldm@rcz006 ldm]$ uname -a
>Linux rcz006.ust.hk 2.4.9-13 #1 Tue Oct 30 20:11:04 EST 2001 i686 unknown

Thanks for this information.  I see right off a problem you will see
with the current incarnation of the LDM.  Your feed request line:

# LDM5 servers request data
#
#       request <feedset> <pattern> <hostname pattern>
#
request WMO ".*" iita.rap.ucar.edu

will cause all of the products in the 'WMO' feed to be delivered on one
rpc.ldmd connection.  This works fine for machines that are
electronically close to their upstream feed site(s), but this is not
the case for you.

A little bit of background by way of explanation is in order:  the
current LDM (including versions up to and including 5.2.2) uses a
blocking RPC to deliver data products that are less than 16384 bytes in
size and multiple blocking RPC calls for products that are larger than
16384 bytes.

A blocking RPC call is one where the upstream LDM waits until it
receives acknowledgement of receipt of a product/piece of product
before sending the next product/piece of product.  Traceroutes to your
machine, rcz006.ust.hk show that the round trip time from Boulder is on
the order of a quarter of a second:

                           Matt's traceroute  [v0.44]
laraine.unidata.ucar.edu                               Mon Nov 18 10:13:43 2002
Keys:  D - Display mode    R - Restart statistics    Q - Quit
                                           Packets               Pings
Hostname                                %Loss  Rcv  Snt  Last Best  Avg  Worst
 1. flra-n140.unidata.ucar.edu             0%  540  540     0    0    0     10
 2. gin-n243-80.ucar.edu                   0%  539  539     0    0    0     10
 3. frgp-gw-1.ucar.edu                     0%  539  539     1    1    2     14
 4. 198.32.11.105                          0%  539  539     1    1    3     17
 5. 198.32.11.111                          0%  539  539     2    1    2     11
 6. kscy-dnvr.abilene.ucaid.edu            0%  539  539    14   12   13     60
 7. kscyng-kscy.abilene.ucaid.edu          0%  539  539    12   12   13     32
 8. iplsng-kscyng.abilene.ucaid.edu        2%  533  539    47   21   86    332
 9. ipls-iplsng.abilene.ucaid.edu          0%  539  539    21   21   22     52
10. 192.245.196.109                        0%  539  539   262  261  264    471
11. 192.245.196.81                         0%  539  539   263  263  264    322
12. 202.40.138.137                         0%  539  539   264  263  269    551
13. 202.40.138.250                         0%  538  539   269  263  281    649
14. internet-gw1.ust.hk                    0%  538  539   272  265  286    630
15. rcz006.ust.hk                          0%  538  539   267  265  285    585

This means that at most 4 products per second can be delivered to your
machine on any one rpc.ldmd connection.  The 'WMO' feed, which is the
combination of IDS, DDS, PPS, and HRS (DDS + PPS == DDPLUS) has a LOT of
products in it.  There is virtually no way that all of these products
can be delivered to your machine using a single rpc.ldmd connection.
The number of products available in the IDS|DDPLUS feed alone will make
it impossible for you to have a reliable feed unless you take steps to
split your feed request into multiple pieces.

The objective is to change your system and LDM configurations to cause
use of multiple rpc.ldmd connections between your machine and your
upstream feed site.  The current LDM gathers all feed requests to
individual machines into a single rpc.ldmd.  To get around this "feature",
you have to play some games.  Here's how:

<as 'root'>

o edit your /etc/hosts file and create an entry for your upstream feed
  site that contains several aliases.  Here is what I recommend:

128.117.200.211 iita.rap.ucar.edu iita1.rap.ucar.edu iita2.rap.ucar.edu 
iita3.rap.ucar.edu iita4.rap.ucar.edu iita5.rap.ucar.edu

-- be mindful that a tab is required as white space between the machine IP
   and the first name; all other white space can be spaces --

This creates 4 aliases for the machine iita.rap.ucar.edu.  We will use
these aliases to split your data request into smaller pieces.

Next, check your /etc/nsswitch.conf file to make sure that the /etc/hosts
file will be used first when looking-up machine names.  Here is how we have
one of our RedHat Linux 7.3 systems setup:

hosts:      files dns nisplus nis

This specifies the search order when doing host look-ups to:

1st) files   -> /etc/hosts
2nd) dns     -> use DNS lookup
3rd) nisplus -> use nisplus
4th) nis     -> use nis

You may choose to specify nisplus and/or nis before dns, but you should
set 'files' to be first in the list.

<as 'ldm'>

Now that you have several aliases for your feed host, you can split
your data requests.  How this can be done varies.  I will show you how
I setup a machine in Belem, Brazil that has traceroutes virtually
identical to yours to receive data:

change:

request        WMO ".*" iita.rap.ucar.edu

to:

#request        WMO ".*" iita.rap.ucar.edu

request DDPLUS|IDS      "^[^S]" iita.rap.ucar.edu
request DDPLUS|IDS      "^S[AR]" iita1.rap.ucar.edu
request DDPLUS|IDS      "^S[^AR]" iita2.rap.ucar.edu
request HDS             "^H.[I-P]... KWB[^K]" iita3.rap.ucar.edu
request HDS             "^H[A-Z][ABCD][A-Z][0-9][0-9] KWB." iita4.rap.ucar.edu
request HDS             ".*(ECMWF|EGRR)" iita5.rap.ucar.edu

What this does is split the request for DDPLUS|IDS data (global
observational data) roughly into thirds by number of products.  The HDS
request lines request just the global model output from the HDS feed.
I imagine that this would be all that you will need, since the majority
of model data in HDS covers only North America.

After you make the changes to ~ldm/etc/ldmd.conf, you will need to stop
and restart your LDM:

ldmadmin stop
<wait until all LDM processes have exited>
ldmadmin start

At this point (assuming that you didn't have any typos in /etc/hosts,
/etc/nsswitch.conf, and ~ldm/etc/ldmd.conf), you should see 7 rpc.ldmds
running on your machine where previously you would have seen only 2.
This same technique for feed splitting has worked incredibly well
for the machine in Brazil that we are feeding.

OK, so the procedures listed above are UGLY.  We are working on a new
version of the LDM that gets rid of all of this ugliness.  Since
this version will not be generally available for some unknown time,
the feed splitting technique is what you will need to do.

With all of the above, I would like to encourage you to upgrade your
LDM installation to 5.2.2.  The main reason for this is that 5.2+
allows sites to report realtime statistics back to the Unidata Program
Center (UPC).  5.2.2 is only available as source, so you would need to
build it locally.  LDM versions 5.2.0 and 5.2.1 have known memory leaks
in the real time statistics reporting facility, so please install 5.2.2.

Here is the quick and dirty for doing an LDM installation:

<login as 'ldm'>
cd ~ldm
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> your_full_email_address
  cd pub/ldm5
  binary
  get ldm-5.2.2.tar.Z
  quit

zcat ldm-5.2.2.tar.Z | tar xvf -

cd ldm-5.2.2/src
./configure
make
make install
sudo make install_setuids

cd ~/ldm-5.2.2/bin
<edit ldmadmin and make sure that:

o '$hostname = "@HOSTNAME@"' is replaced by '$hostname = "rcz006.ust.hk"'
o the location for Perl on your system is correct
o the LDM product queue size is what you want

Once you have finished the build, you will need to stop the LDM if it
is running; switch your runtime link; deleted your old LDM queue; remake
a new LDM queue; add a line to the LDM ldmd.conf file to report
real time statistics; and restart your LDM:

cd ~
ldmadmin stop
rm runtime
ln -s ldm-5.2.2 runtime
ldmadmin delqueue
ldmadmin mkqueue               <- will take varying amounts of time; be patient
cd etc
<edit ldmd.conf and add the line:

exec    "rtstats -h rtstats.unidata.ucar.edu"

put this after the line that starts pqact.

ldmadmin start

Once we start receiving realtime statistics from your, you will be able
to look at how well you are receiving data online:

http://www.unidata.ucar.edu/staff/chiz/latency/stats

Click on the 'Statistics by Host' link from the above page and look for
your host, rcz006.ust.hk.

Please let us know if you have any questions on the above procedures.

Cheers,

Tom Yoksas

>From: address@hidden
>Date: Tue, 19 Nov 2002 01:56:34 +0800 (CST)
>Subject: Away from my email [Re: 20021116: using the LDM to feed a remote site 
>in Hong Kong (cont.)]

>I will be out of town until 20th Nov.
>Your email regarding

>20021116: using the LDM to feed a remote site in Hong Kong (cont.)

>will be read when I return.