20030125: LDM 6 release candidate available for testing

>From: Unidata Support <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200212271733.gBRHXAt25394 LDM 6 test

Hi Harry,

We have a test version of LDM 6 available that we love to see you test
on marzipan.  LDM 6 offers a number of new ideas that we are excited
about.  Here are a few:

o request lines to the same host in ldmd.conf are no longer accumulated into
  a single rpc.ldmd invocation.  This allows a user to split feeds without
  having to create machine name aliases in /etc/hosts.

o the size of the HEREIS - COMINGSOON "fence" is now user configurable.
  In LDM 5, all products smaller than 16384 bytes would be sent using
  the HEREIS protocol.  In LDM 6, this "fence" is settable on a per-request
  line basis.  The default for the "fence" size is now essentially
  infinity (actually, it is size of a uint).  You specify the "fence"
  size as the 4th parameter on an ldmd.conf request line.  LDM 6
  understands a couple of mnemonics for minimim and maximum "fence"
  sizes:

request IDS|DDPLUS ".*" my.upstream.site MAX        (or MAXIMIM)
request IDS|DDPLUS ".*" my.upstream.site MIN        (or MINIMUM)

  We think that the MAX (uint (or is it ulong??)) and MIN (zero)
  values are basically all that a site will ever need/use, but
  the values are configurable nonetheless.

o products sent with the COMINGSOON/BLKDATA protocol are sent in one
  chunk after a transaction is made between the client and server
  in which the client says that it wants the product.  In LDM 5, the
  same client/server transaction was made and then the product was
  sent in 16384 BLKDATA chunks.

o LDM 6 is backward/forward compatible with LDM 5.  LDM 5 clients
  can receive data from LDM 6 servers, and LDM 6 clients can receive
  data from LDM 5 servers.  When LDM 5 talks to LDM 6, it does so
  using LDM 5 protocols.  LDM 6 running LDM 5 protocols is more
  efficient than LDM 5.

o LDM 6 can use the same product queue that LDM 5 uses.  This allows
  a site to install the new version and start running without
  remaking the product queue.
  
o LDM 6 is much more capable of delivering data to electronically
  remote sites.  One site we have been using for testing is located
  on the edge of the Amazon in Belem, Brazil.

o LDM 6's ldmadmin now does several things for you that you had
  to do by hand in LDM 5:

  o put you in the ~ldm directory at LDM startup
  o create the ~ldm/logs/ldmd.log file if it does not already exist
  o determine your machine's hostname using a 'uname -n'.  If
    this does not result in a fully qualified hostname, ldmadmin
    will complain and not start the LDM.  In this case, you
    need to edit ldmadmin and set the hostname:

change:

chop($hostname = `uname -n`);
# $hostname = "your.hostname.here";

to:

# chop($hostname = `uname -n`);
$hostname = "marzipan.atmos.washington.edu";

o LDM 6's ldmadmin now waits until the rpc.ldmd processes have all
  exited before returning you to the Unix command prompt

o LDM 6 now understands CONDUIT and CRAFT as the NMC2 and NEXRD2
  streams.

o Some early tests using synthesized data showed that LDM 6 was capable
  of moving 8.4 times more data than LDM 5.  The increase in "efficiency"
  using real data has, however, not yet been quantified.  For reference,
  the early tests that showed the 8.4 times increase was conducted between
  a machine here at the UPC, and a Linux PC running at the Universidade
  Federal do Para in Belem, Brazil using synthetic products that were\
  each 200KB in size.  Also, during the test, the client LDM did not
  write into a product queue.

If you decide that you want to help us test out LDM 6, then you should
do the following:

1) make backup copies of your ~ldm/etc/ldmd.conf file(s).  While
   you are at it, you could make a backup copy of your ~ldm/etc/pqact.conf
   file, but this is optional since it will not be affected by moving
   to LDM 6.

2) FTP LDM 6 release candidate 6:

<login as 'ldm'>
cd ~ldm
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> address@hidden
  cd pub/ldm5/test
  binary
  get ldm-6.0,0.6.tar.Z
  quit
zcat ldm-6.0,0.6.tar.Z | tar xvf -

3) build LDM 6.  Note that LDM 6 will build using c89 and '-g' for
   CFLAGS unless you override these.  You may well want to set
   CFLAGS to -O.

setenv LDMHOME your_ldm_home_directory
setenv CFLAGS -O

cd ~ldm/ldm-6.0.0.6/src
./configure
make && make install

4) finish the LDM install as 'root':

sudo make install_setuids

5) adjust entries in ldmadmin to match those from your existing ldmadin.
   This should boil down to you changing the product queue size and
   number of log files that you want to keep online (we keep 14 on
   our IDD machines).  I already addressed the possible need to set
   the hostname in ldmadmin above.

6) shutdown your current LDM waiting for all LDM processes to exit.

7) change the runtime link to point at ldm-6.0.0.6

cd ~ldm
rm runtime && ln -s ldm-6.0,0.6 runtime

8) verify that your operating system has finished flushing the LDM
   queue from memory to disk.  I found that this was an important
   step in getting things to work smoothly on thelma.  I did
   the verification by watching the iostat listing of top.  As soon
   as it quites down, you are ready to go.  Our experience is that
   the larger the queue, the longer this step takes.

9) start LDM 6:

rehash
ldmadmin start

I think I have covered all bases in this note, but you never know.
The good news is that you already know all of the upgrade procedure,
so most of the above was unneeded.

So, if you are game, LDM 6 is out there waiting.  If you would like me to
help in the upgrade, just let me know.

Tom

>From address@hidden Mon Jan 27 14:42:48 2003
>Subject: Re: 20030125: LDM 6 release candidate available for testing

Will do in the next day or two.

-- 
 Dr. Harry Edmon                        E-MAIL: address@hidden
 206-543-0547                                   address@hidden
 Dept of Atmospheric Sciences           FAX:    206-543-0308
 University of Washington, Box 351640, Seattle, WA 98195-1640