[IDD #TOP-610401]: 20120927: need to change LDM/IDD feed REQUEST ASAP

Hi Brendon,

re: need to switch feed REQUESTs from idd.cise-nsf.gov

> Thanks.  Most of our ldmd.conf entries for the NSF site were secondary
> in nature.

We knew that, but we wanted to make sure that folks knew of viable
alternates to idd.cise-nsf.gov.

re:
> With regards to the email Steve recently sent out, I was concerned about
> overlapping feed types, and wanted to have a fresh pair of eyes look at
> our config.

OK.

re:
> I originally thought that ldm.conf was supposed to be
> ordered in a top down config, meaning that the order in which feed types
> were listed would determine where data flows from.

No, the ordering of REQUEST lines in the ~ldm/etc/ldmd.conf file do
not matter.  The ordering of actions in a pattern-action file, on
the other hand, do matter as the actions are processed sequentially.

re:
> I tried to structure
> it such that our in-house noaaport feed is primary, with everything else
> IDD as backup.  Here is the request portion for our current ldmd.conf
> for typhoon.plymouth.edu:
> 
> request         ANY     ".*"    noaaport0.plymouth.edu
> request         WMO     ".*"    idd.unidata.ucar.edu
> request         WMO     ".*"    flood.atmos.uiuc.edu
> #request                WMO     ".*"    ldm.engin.umich.edu
> 
> 
> request         UNIWISC ".*"    flood.atmos.uiuc.edu
> request         UNIWISC ".*"    idd.ssec.wisc.edu
> #request                UNIWISC ".*"    ldm.engin.umich.edu
> 
> request         GPS     ".*"    suomildm1.cosmic.ucar.edu
> request         GPS     ".*"    wx-ipw.plymouth.edu
> 
> request         GPSSRC  ".*"    suomildm1.cosmic.ucar.edu
> request         GPSSRC  ".*"    wx-ipw.plymouth.edu
> 
> request FNEXRAD "^(rad/|pnga2area)"     idd.ssec.wisc.edu
> request FNEXRAD "^(rad/|pnga2area)"     idd.unidata.ucar.edu
> 
> request NIMAGE  ".*"    idd.unidata.ucar.edu
> 
> request NGRID   ".*"    idd.unidata.ucar.edu
> 
> request NNEXRAD ".*"    idd.unidata.ucar.edu
> request NNEXRAD ".*"    idd.ssec.wisc.edu
> request NNEXRAD ".*"    flood.atmos.uiuc.edu
> 
> request NLDN    ".*"    striker2.atmos.albany.edu
> #request        NLDN    ".*"    216.236.239.157
> #request NLDN    ".*"    216.236.239.158
> request NLDN    ".*"    96.8.93.15
> request NLDN    ".*"    96.8.93.16
> request NLDN    ".*"    96.8.94.15
> request NLDN    ".*"    96.8.94.16
> 
> request EXP     ".*"    199.192.6.37
> 
> Please make any recommendations you feel are appropriate.

Using the same pattern for REQUEST lines in recent versions of the
LDM (I think since v6.4.x) will result in automatic promotion of
the connection that is delivering the data the fastest to a 
primary mode and the demotion of the other requests in the matching
set to secondary.  This insures that the data gets to the host
as fast as possible.

re:
> Woefully
> behind on ldm upgrades, but the potential DOS implications may raise the
> level of priority on this.

The changes Steve has talked about are in the newest version of the LDM,
v 6.11.x.  They do not affect older versions.  Also, the overlapping
request elimination feature only is pertinent when the REQUESTs are
to the same machine.  From the ldmd.conf listing you provided, I can
see no overlapping REQUESTs since the duplicates are to different
upstream machines.

Here is a very simple example of overlapping feed REQUESTs:

request WMO ".*" noaaport0.plymouth.edu
request HDS ".*" noaaport0.plymouth.edu

The reason that they are overlapping is that WMO is a compound feed
type:

WMO == HDS|IDS|DDPLUS

REQUESTing HDS in a separate line will, in LDM versions previous to
v6.11.x, result in the same data being sent from the upstream twice.
The logic in LDM v6.11.1 is designed to eliminate the duplicate
flow of data and thus save bandwidth.

Steve made a comment about a data product being processed twice
when there are overlapping REQUESTs.  This is not likely to be
the case ** unless one feed is much, much slower than the other **
since the receiving LDM will throw away the duplicate product
received.  This feature has not been turned off or eliminated
in the new LDM.

One other comment about your ldmd.conf REQUEST lines:

If you want to force each REQUEST to work in primary mode,
you need to change the patterns to not be identical.  This
is accomplished easily as follows:

change:

request WMO ".*" idd.unidata.ucar.edu
request WMO ".*" flood.atmos.uiuc.edu

to:

request WMO ".*" idd.unidata.ucar.edu
request WMO "(.*)" flood.atmos.uiuc.edu

The patterns are different as far as the regular expression
interpreter is concerned, but are identical in what they
really are requesting.

re:
> Thanks,

No worries.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TOP-610401
Department: Support IDD
Priority: Normal
Status: Closed