20030418: change UNIWISC feed from gold to unidata2; upgrade to LDM-6 (cont.)

>From: Bill Noon <address@hidden>
>Organization: Unidata Program Center/UCAR
>Keywords: 200304162320.h3GNK07U012403 IDD UNIWISC/MCIDAS LDM-6

Hi Bill,

re: switching UNIWISC/MCIDAS from gold to unidata2
>Oops, forgot that I was getting MCIDAS from gold.  Switched now.

Wonderful, thanks for the quick change.

re: upgrade to LDM-6
>I was out of town last week and didn't want to make the move before  
>leaving.

I understand completely.

>Will get the download and compile things up today.  Will probably  
>switch over this weekend.

Super!  Your data reception AND relay will significantly improve
after you move up to LDM-6.  Please don't forget to add the ldmd.conf
entry for real time statistics reporting:

exec    "rtstats -h rtstats.unidata.ucar.edu"

If you are using any LDM feed types in a "non-standard" manner
for internal routing of data (i.e., you are using a feed type name
to move some data that never makes it into the IDD), then please
tailor your rtstats entry in ldmd.conf to only report for those feeds
being used in the IDD.  We are trying to get our real time statistics
pages to reflect the IDD's health, and external, private feeds tend
to cloud the picture.

>Are there ldm-6 specific tunings that must be done?

There are no LDM-6 specific tunings that _must_ be done, but there are
some that should be done.  In particular:

1) if any of your feed requests use host IP addresses, please switch
   them to use of fully qualified host names.  This is for statistics
   reporting, nothing else

2) if you split feeds to single upstream hosts by creating /etc/hosts
   aliases for the upstream host's name, you should change the requests
   with the aliases to the machine's real name.  LDM-6 no longer
   accumulates requests to the same upstream host into single rpc.ldmds,
   so the process of splitting feed requests is much easier

3) since the ldm no longer accumulates feed requests to the same upstream
   host into single rpc.ldmds, it is up to the site to accumulate
   (and split) feeds where appropriate.  Depending on who a site is
   feeding from (i.e., how far away they are electronically; how solid
   the connection is; and how fast both the upstream and downstream
   machines are), one can ususally combine the UNIWISC, FSL2, and FNEXRAD
   feeds:

request UNIWISC|FSL2|FNEXRAD ".*"       upstream_IDD_host

   The goal with feed request management is to combine as many feeds
   as possible into single requests _without_ putting too many together
   so that the reception of data in one inteferes with the reception
   in another.  By intefering I mean that if there are lots of products
   in the combined request, then the queue that the rpc.ldmd has to work
   through to send the data gets to be long.  Products at the end of
   the queue have to wait for products at the front of the queue, so
   there is a slowing down of delivery.

4) removal of problematic regular expressions in ldmd.conf requests
   and pqact.conf actions.  A problematic regular expression is one
   that has patterns like '.*' where they are not needed.  The typical
   example of this is a '.*' at the beginning or end of a pattern.  Here
   is the email that Steve Emmerson sent to ldm-users on this topic:

   From: Steve Emmerson <address@hidden>
   To: address@hidden
   Subject: pathological LDM regular-expressions
   Date: Fri, 14 Feb 2003 10:12:41 -0700
   
   Hi,
   
   In getting the latest LDM ready for release, we've learned that certain,
   seemingly inocent regular-expressions (RE-s) can cause the upstream LDM
   to spend over 99% of its user-time executing the RE -- leaving little
   time for sending you the data. This email is to let you know about these
   RE-s and to ask you to replace them in your LDM configuration-file.
   
   The pathological RE-s have the form
   
       .*<rest of RE>
   
   i.e., they have "dot-star" as a prefix. The ".*" adds nothing to the
   meaning of the RE but, on a Solaris system at least, it's execution is
   about 1300 times slower than the corresponding unprefixed RE.
   
   I'm not making this up.
   
   Would you please go through your LDM configuration file and replace all
   RE-s of the above form with the unprefixed version.  For example, the
   entry
   
       request HDS  ".*foo/bar" ...
   
   should become
   
       request HDS  "foo/bar" ...
   
   i.e., remove the ".*" prefix.  Note that this is only applicable to RE-s
   with ".*" AS A PREFIX.  RE-s that consist solely of ".*" (as many, if
   not most, due) should be left unaltered.
   
   Then, restart your LDM if you made any changes.
   
   Email me or this list if you have any questions.
   
   Regards,
   Steve Emmerson

   Note that these comments also apply to pqact.conf entries!

5) change any/all allow ANY lines to allow ANY-WSI-NLDN-PCWS-GEM to
   eliminate the possibility of relaying data that is not allowed
   to be relayed

Off of the top of my head, that is the top 5 things to look for.  I
have to say, however, that sometimes we can see things that need
changing when we look at a site's ldmd.conf file.  If loggin onto your
system is not easily done, you could send us your ldmd.conf and
possibly pqact.conf files so we could take a look.

Once again, thanks for the effort!

Cheers,

Tom