20030715: IDS|DDPLUS latencies and building an LDM with large file support (cont.)

>From: Celia Chen <address@hidden>
>Organization: NCAR/RAP
>Keywords: 200307151639.h6FGdSLd007355 LDM IDD rtstats largefile

Celia,

>I had
>exe     "rtstats -q /home/ldm/data/ldm.pq -d /home/ldm/logs -h 
>rtstats.unidata.ucar.edu" 
>in chisel's ldmd.conf file as the last exec line. Is there a reason for the
>above line not working properly?

If you really did have 'exe' instead of 'exec', and the ldmd.conf
listing below shows that you did, then yes, this was the problem.  'exe'
is not a recognized abbreviation for 'exec'.

The -q, -d, and -h entries were OK, but not needed _if_ your LDM setup is
such that the queue is in ~ldm/data and the log flie directory is
~ldm/logs.

>However, I just replaced it with 
> exec  "rtstats -h rtstats.unidata.ucar.edu"
>hope this will send the statistics to you.

We are now getting real time stats from chisel as can be seen from:

http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+chisel.rap.ucar.edu

From the latency plot for IDS|DDPLUS, I can say that chisel is getting
the data from thelma with no delay.  This tells me that if some METARs
are coming in late, they were late in the NOAAPORT broadcast; and
there is nothing we can do about that.

>PS - Below is chisel's ldmd.conf file:

OK, here are some suggestions for tuning your ldmd.conf file:

1) change the Exec Entries section to clean things up:

exec    "pqbinstats"
exec    "pqact -l /home/ldm/logs/pqact.log"
exec    "pqsurf -p ^(SACN43|S[APX]|FXUS43) -Q /home/ldm/data/pqsurf.pq -l 
logs/pqsurf.log etc/pqsurf.conf"
exec    "rtstats -h rtstats.unidata.ucar.edu"

    The suggested changes assume that the LDM queue can be found in
    ~ldm/data, and the log files are written into ~ldm/logs.

2) The NLDN feed is supposed to be available only by point-to-point feed
   from SUNY Albany.  This is the agreement the Unidata community has
   with SUNY Albany.  Breaching it could result in the community losing
   this feed.  We should, therefore, not be relaying this data to you from
   the atm.geo.nsf.gov machine.

request NLDN    ".*"    striker.atmos.albany.edu        PRIMARY
request NLDN    ".*"    atm.geo.nsf.gov ALTERNATE       

   On top of this, striker.atmos.albany.edu is the injection machine
   for NLDN data.  If it goes down, there will be no NLDN data from
   any machine.

   Given these two observations, please remove your NLDN request line that
   points to atm.geo.nsf.gov.  I will be removing the allow for NLDN
   data from the ldmd.conf file on atm.

   Also, you are allowing a number of machines to feed 'ANY' from chisel.
   This is OK for RAP machines and mostly OK for other .ucar.edu machines,
   but is not OK for others like the .gov machines.  The same comment can
   be made for WSI data.  You should, therefore, change your allow lines
   for non-RAP machines from:

allow   ANY     ...

   to:

allow   ANY-WSI-NLDN    ...

3) The following suggestion will help cut down on the number of rpc.ldmd
   connections to upstream feeds sites; they should have no effect
   on your reception of data:

   - combine your DIFAX, FSL2, and MCIDAS data requests to thelma and atm:

     change the three lines:

request FSL2    ".*"    thelma.ucar.edu PRIMARY
request DIFAX   ".*"    thelma.ucar.edu PRIMARY
request MCIDAS  ".*"    thelma.ucar.edu PRIMARY

     and:

request FSL2    ".*"    atm.geo.nsf.gov ALTERNATE
request DIFAX  ".*"    atm.geo.nsf.gov  ALTERNATE
request MCIDAS  ".*"    atm.geo.nsf.gov ALTERNATE

     to the single lines: 

request DIFAX|FSL2|UNIWISC    ".*"    thelma.ucar.edu PRIMARY
request DIFAX|FSL2|UNIWISC    ".*"    atm.geo.nsf.gov ALTERNATE

     (UNIWISC is the accepted name for Unidata-Wisconsin datasteam
     products.  MCIDAS still works, but it has been deprecated).

     Cutting down on the number of rpc.ldmds running on both
     your machine and the upstream feed host(s) will help both
     machines run more efficiently.

4) remove the request for NEXRD2 data to atm.geo.nsf.gov.  atm does not
   receive this feed, so requesting the data from it does nothing more
   than add a load to your machine AND to atm.  Unfortunately, there
   is no current failover machine for this feed.

5) I would normally ask you to remove the NMC2 (CONDUIT) feed to
   atm since atm doesn't have this data, but we are trying to get
   the weather service to feed atm.  If/when this does happen, thelma
   will ingest CONDUIT from atm.  Having chisel ingest from thelma
   and atm does not give you any real redundancy.  Because of this,
   I think you would do better to request the CONDUIT stream from
   the University of Wisconsin/AOS machine.

   change:

request NMC2  ".*"    thelma.ucar.edu   PRIMARY
request NMC2  ".*"    atm.geo.nsf.gov   ALTERNATE

   to:

request CONDUIT  ".*"    thelma.ucar.edu   PRIMARY
request CONDUIT  ".*"    f5.aos.wisc.edu   ALTERNATE

   At some point in the near future (not now), you might be asked to
   change this to:

request CONDUIT "[09]$" f5.aos.wisc.edu PRIMARY
request CONDUIT "[18]$" f5.aos.wisc.edu PRIMARY
request CONDUIT "[27]$" f5.aos.wisc.edu PRIMARY
request CONDUIT "[36]$" f5.aos.wisc.edu PRIMARY
request CONDUIT "[45]$" f5.aos.wisc.edu PRIMARY
request CONDUIT  ".*"    atm.geo.nsf.gov   ALTERNATE

   You will need to contact Pete Pokrandt <address@hidden>
   at UW/AOS to get him to add an allow for CONDUIT for chisel.


Combining the above recommendations, your cleaned-up ldmd.conf file would
look like:


###############################################################################
# Exec Entries
###############################################################################
#
exec    "pqbinstats"
exec    "pqact -l /home/ldm/logs/pqact.log"
exec    "pqsurf -p ^(SACN43|S[APX]|FXUS43) -Q /home/ldm/data/pqsurf.pq -l 
/home/ldm/logs/pqsurf.log etc/pqsurf.conf"
#
# rtstats uses the LDM to send product latency statistics to the UPC.
# We are currently evolving from pqbinstats to rtstats, but in the
# mean time we hope sites will run both.  For more info on rtstats
# please see the man page.
exec    "rtstats -h rtstats.unidata.ucar.edu"
#
###############################################################################
# Request Entries
###############################################################################
#
# LDM5 servers request data from Data Sources
#
#       request <feedset> <pattern> <hostname pattern>
#
# 1/3/2003 Request lightning data from Albany:
# address@hidden (David Knight)
#
request NLDN    ".*"    striker.atmos.albany.edu        PRIMARY

#
# Request data from FSL - ACARS UATurbulence data 
# Request data from FSL - AMDAR data 
#
request PCWS    "^FSL\.NetCDF\.ACARS\.QC\."     eldm.fsl.noaa.gov
request FSL3    "^FILTER\.ACARS\.UATurbulence\."        eldm.fsl.noaa.gov 

#
# DIFAX   - DIFAX-like products from University of Wisconsin/AOS
# FSL2    - FSL wind profiler
# UNIWISC - GOES satellite image sectors and products from UW/SSEC

request DIFAX|FSL2|UNIWISC    ".*"    thelma.ucar.edu PRIMARY
request DIFAX|FSL2|UNIWISC    ".*"    atm.geo.nsf.gov ALTERNATE

#
# IDS|DDPLUS - NOAAPORT global observations

request WMO     ".*"    thelma.ucar.edu PRIMARY
request WMO     ".*"    atm.geo.nsf.gov ALTERNATE       
 
#
# NNEXRAD - NOAAPORT NEXRAD Level III data
 
request NNEXRAD  ".*"   thelma.ucar.edu PRIMARY
request NNEXRAD  ".*"   atm.geo.nsf.gov ALTERNATE       

#
# CONDUIT - High resolution model output from NCEP

request CONDUIT ".*"    thelma.ucar.edu PRIMARY
request CONDUIT ".*"    f5.aos.wisc.edu ALTERNATE

#
# NEXRD2 - NEXRAD Level-2 data  NEXRD2 / NMC3

request NEXRD2  ".*"    thelma.ucar.edu PRIMARY

#
# EXP

request EXP     ".*"    level.rap.ucar.edu
request EXP     ".*"    turb.rap.ucar.edu
 
#
# New WSI data feed (10/21/1998 - Celia Chen RAP/NCAR)

request WSI     ".*"    iita.rap.ucar.edu
#request WSI    ".*"    wsihcsn.unidata.ucar.edu

###############################################################################
# Begin Access control
###############################################################################
 
###############################################################################
# ALLOW: Who we are willing to feed
# Allow Entries
# allow <feedset> <hostname pattern>
###############################################################################
#
# Giving permission for a Data Sink to perform a request to your LDM
#
# allow <feedset> <hostname pattern>
#
# Giving permission to your own machine and Unidata
#
# Under no circumstances comment out the next allow entry to localhost
# The LDM will NOT start if the lines are commented out.
#
allow   ANY
    ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$
 ))
allow   ANY
    ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.rap\.ucar\.edu\.?$))
###############################################################################
 
# send anything to your own machine (and other RAP LDM servers)

allow   ANY     chisel
allow   ANY     level
allow   ANY     iita
allow   ANY     ofour
allow   ANY     front
allow   ANY     turb

allow   ANY     moonbow.rap.ucar.edu
allow   ANY     serac.rap.ucar.edu
allow   ANY     saint.rap.ucar.edu
allow   ANY     
^((localhost|loopback)|(127\.0\.0\.1\.?$)|(^[a-z].*\.rap\.ucar\.edu\.?$))

# ATEC machines (128.117.200.84 atec-ingest)
# ATEC machines (128.117.200.11 mm5-dev)
# ATEC machines (128.117.200.101 uae-c1)
allow   ANY     uae-c1.rap.ucar.edu
allow   ANY     atec-ingest
allow   ANY     mm5-dev
#
# Ceiling and Visibilities Project:
allow   ANY     ceil
allow   ANY     vis
allow   ANY     shaggy
allow   ANY     scrappy
allow   ANY     scooter
allow   ANY     speeder
allow   ANY     wan
allow   ANY     scooby

# AWC machines
allow   ANY-WSI-NLDN     adds.aviationweather.gov
allow   ANY-WSI-NLDN     amelia.kc.noaa.gov
allow   ANY-WSI-NLDN     orville.kc.noaa.gov
allow   ANY-WSI-NLDN     wilbur.kc.noaa.gov
allow   ANY-WSI-NLDN     204.227.127.34
allow   ANY-WSI-NLDN     204.227.127.38
allow   ANY-WSI-NLDN     204.227.127.39
allow   ANY-WSI-NLDN     204.227.127.40
allow   EXP     adds.awc-kc.noaa.gov
allow   EXP     amelia.kc.noaa.gov
allow   EXP     orville.kc.noaa.gov
allow   EXP     wilbur.kc.noaa.gov
#
# UCAR LDM servers
allow   ANY     thelma.ucar.edu
allow   ANY     ldm.comet.ucar.edu
allow   ANY     gizmo.comet.ucar.edu
allow   ANY     zero.unidata.ucar.edu
allow   ANY     box.mmm.ucar.edu
allow   ANY     tea.mmm.ucar.edu
allow   ANY     128.117.88.210
#
# FSL LDM servers
allow   ANY-WSI-NLDN     eldm.fsl.noaa.gov
allow   ANY-WSI-NLDN     137.75.1.51
# WSI LDM servers
allow   ANY-WSI-NLDN     wsi-204-115.wsi.com

# Other outside Users
allow   ANY-WSI atm.geo.nsf.gov 
allow   EXP     sirocco.srcc.lsu.edu
allow   EXP     rainbow.eas.purdue.edu
allow   EXP     sunset.meteor.wisc.edu
allow   EXP     unidata.ssec.wisc.edu
allow   EXP     grayskies.atmos.washington.edu
allow   EXP     drier.atmos.washington.edu
allow   EXP     dry.atmos.washington.edu
allow   EXP     redwood.atmos.albany.edu
allow   WMO     wnnh001.wind.ttu.edu
#
###############################################################################
# Accept Entries
###############################################################################
# ACCEPT: Who can feed us, currently this action is ONLY needed for WSI data
#
# accept <feedset> <pattern> <hostname pattern>
#
# accept anything from yourself
#
#accept ANY ".*" ^((localhost|loopback)|(127\.0\.0\.1\.?$))
#
# accept from your upstream site
#
# WSI is using ldm4 protocol so the accept is still required
#accept WSI
#    .*
#    ^[a-z].*\.uni\.wsicorp\.com$
#
###############################################################################
# End 
###############################################################################


If you have any questions on the above changes, or if you would like
us to tune up your ldmd.conf file please let me know.

Tom 

--- chisel.rap.ucar.edu's ldmd.conf file ---

>#####
># $Id: ldmd.conf,v 1.16 2002/06/17 21:12:43 anne Exp $
># Sample ldmd.conf for ldm5
>####
>#
># This is the main configuration file for the LDM server. All lines that start
># with a "#" sign are comments.
>#
># To debug an LDM that hangs on start up, run the following from LDM home:
># % bin/rpc.ldmd -vl - -q data/ldm.pq etc/ldmd.conf
>#
># If the LDM still hangs, comment out all lines in this file except noted
># allow line entry below, try again.
>#
>##############################################################################
> #
># Exec Entries
>##############################################################################
> #
>#
># Programs that can share a queue with rpc.ldmd server, these programs
># are started by the "exec" command and are in the same process group.
>#
># Regarding pqbinstats, while pqbinstats will generate .stats files in
># your logs directory, it is necessary to also run 'ldmadmin dostats'
># once per hour via cron in order to email the stats to the UPC and
># remove the .stats files that are no longer needed.  See
># http://www.unidata.ucar.edu/packages/ldm/ldmConfigInstallList.html#s7
># for more information.
>#exec  "pqbinstats"
>exec   "pqbinstats -d /home/ldm/logs"
>exec   "pqact -l /home/ldm/logs/pqact.log -d /home/ldm /home/ldm/etc/pqact.con
> f"
>exec   "pqsurf -p ^(SACN43|S[APX]|FXUS43) -Q /home/ldm/data/pqsurf.pq -l /home
> /ldm/logs/pqsurf.log -d /home/ldm /home/ldm/etc/pqsurf.conf"
>#
>#
># rtstats uses the LDM to send product latency statistics to the UPC.
># We are currently evolving from pqbinstats to rtstats, but in the
># mean time we hope sites will run both.  For more info on rtstats
># please see the man page.
># exe  "rtstats -q /home/ldm/data/ldm.pq -d /home/ldm/logs -h rtstats.unidata.
> ucar.edu"
>exec    "rtstats -h rtstats.unidata.ucar.edu"
># send info back to local disk - NOT working:
>#exec  "rtstats -q /home/ldm/data/ldm.pq -d /home/ldm/logs -h "
>#
>##############################################################################
> #
># Request Entries
>##############################################################################
> #
>#
># LDM5 servers request data from Data Sources
>#
>#      request <feedset> <pattern> <hostname pattern>
>#
>#request WMO ".*" uni0.unidata.ucar.edu
>#
># 1/3/2003 Request lightning data from Albany:
># address@hidden (David Knight)
>#
>request NLDN   ".*"    striker.atmos.albany.edu        PRIMARY
>request NLDN   ".*"    atm.geo.nsf.gov ALTERNATE       
>#
># Request data from FSL - ACARS UATurbulence data 
># Request data from FSL - AMDAR data 
>#
>request PCWS
>        "^FSL\.NetCDF\.ACARS\.QC\."
>        eldm.fsl.noaa.gov
>
>request FSL3
>        "^FILTER\.ACARS\.UATurbulence\."
>       eldm.fsl.noaa.gov 
>#
># Requesting data from thelma 
># FSL2 - Profiler data
> 
>request FSL2   ".*"    thelma.ucar.edu PRIMARY
>request FSL2   ".*"    atm.geo.nsf.gov ALTERNATE       
> 
>#request        IDS|DDPLUS
>request WMO    ".*"    thelma.ucar.edu PRIMARY
>request WMO    ".*"    atm.geo.nsf.gov ALTERNATE       
> 
>request EXP
>        ".*"
>        level.rap.ucar.edu
>request EXP
>        ".*"
>        turb.rap.ucar.edu
> 
># Greg Thompson's (1/2003 Celia Chen)
>#request EXP
>#        ".*"
>#        129.118.105.15
> 
># DIFAX feed
>request DIFAX   ".*"    thelma.ucar.edu        PRIMARY
>request DIFAX  ".*"    atm.geo.nsf.gov ALTERNATE       
> 
>#
># get NEXRAD data
> 
>request NEXRAD  ".*"    thelma.ucar.edu        PRIMARY
>request NEXRAD  ".*"    atm.geo.nsf.gov        ALTERNATE       
>
>request NMC2  ".*"    thelma.ucar.edu  PRIMARY
>request NMC2  ".*"    atm.geo.nsf.gov  ALTERNATE       
>
># Level-2 data:  NEXRD2 / NMC3
>
>request NEXRD2  ".*"    thelma.ucar.edu        PRIMARY
>request NEXRD2  ".*"    atm.geo.nsf.gov        ALTERNATE       
>
>#
>request MCIDAS ".*"    thelma.ucar.edu PRIMARY
>request MCIDAS  ".*"    atm.geo.nsf.gov        ALTERNATE       
> 
># New WSI data feed (10/21/1998 - Celia Chen RAP/NCAR)
> 
>request WSI    ".*"    iita.rap.ucar.edu
>#request WSI   ".*"    wsihcsn.unidata.ucar.edu
>##############################################################################
> #
># Begin Access control
>##############################################################################
> #
> 
>##############################################################################
> #
># ALLOW: Who we are willing to feed
># Allow Entries
># allow <feedset> <hostname pattern>
>##############################################################################
> #
>#
># Giving permission for a Data Sink to perform a request to your LDM
>#
># allow        <feedset> <hostname pattern>
>#
># Giving permission to your own machine and Unidata
>#
># Under no circumstances comment out the next allow entry to localhost
># The LDM will NOT start if the lines are commented out.
>#
>allow  ANY
>    ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$
> ))
>allow  ANY
>    ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.rap\.ucar\.edu\.?$))
>##############################################################################
> #
> 
># send anything to your own machine (and other RAP LDM servers)
>
>allow   ANY     chisel
>allow   ANY     level
>allow   ANY     iita
>allow   ANY     ofour
>allow   ANY     front
>allow   ANY     turb
>
>allow   ANY     moonbow.rap.ucar.edu
>allow   ANY     serac.rap.ucar.edu
>allow   ANY     saint.rap.ucar.edu
>allow   ANY     ^((localhost|loopback)|(127\.0\.0\.1\.?$)|(^[a-z].*\.rap\.ucar
> \.edu\.?$))
>
># ATEC machines (128.117.200.84 atec-ingest)
># ATEC machines (128.117.200.11 mm5-dev)
># ATEC machines (128.117.200.101 uae-c1)
>allow   ANY     uae-c1.rap.ucar.edu
>allow  ANY     atec-ingest
>allow  ANY     mm5-dev
>#
># Ceiling and Visibilities Project:
>allow  ANY     ceil
>allow  ANY     vis
>allow  ANY     shaggy
>allow  ANY     scrappy
>allow  ANY     scooter
>allow  ANY     speeder
>allow  ANY     wan
>allow  ANY     scooby
>
># AWC machines
>allow   ANY     adds.aviationweather.gov
>allow   ANY     amelia.kc.noaa.gov
>allow   ANY     orville.kc.noaa.gov
>allow   ANY     wilbur.kc.noaa.gov
>allow   ANY     204.227.127.34
>allow   ANY     204.227.127.38
>allow   ANY     204.227.127.39
>allow   ANY     204.227.127.40
>allow   EXP     adds.awc-kc.noaa.gov
>allow   EXP     amelia.kc.noaa.gov
>allow   EXP     orville.kc.noaa.gov
>allow   EXP     wilbur.kc.noaa.gov
>#
># UCAR LDM servers
>allow   ANY     thelma.ucar.edu
>allow   ANY     ldm.comet.ucar.edu
>allow   ANY     gizmo.comet.ucar.edu
>allow   ANY     zero.unidata.ucar.edu
>allow   ANY     box.mmm.ucar.edu
>allow   ANY     tea.mmm.ucar.edu
>allow   ANY     128.117.88.210
>#
># FSL LDM servers
>allow   ANY     eldm.fsl.noaa.gov
>allow   ANY     137.75.1.51
># WSI LDM servers
>allow   ANY     wsi-204-115.wsi.com
>
># Other outside Users
>allow  ANY-WSI atm.geo.nsf.gov 
>allow   EXP     sirocco.srcc.lsu.edu
>allow   EXP     rainbow.eas.purdue.edu
>allow   EXP     sunset.meteor.wisc.edu
>allow   EXP     unidata.ssec.wisc.edu
>allow   EXP     grayskies.atmos.washington.edu
>allow   EXP     drier.atmos.washington.edu
>allow   EXP     dry.atmos.washington.edu
>allow   EXP     redwood.atmos.albany.edu
>allow  WMO     wnnh001.wind.ttu.edu
>#
>##############################################################################
> #
># Accept Entries
>##############################################################################
> #
># ACCEPT: Who can feed us, currently this action is ONLY needed for WSI data
>#
># accept <feedset> <pattern> <hostname pattern>
>#
># accept anything from yourself
>#
>#accept ANY ".*" ^((localhost|loopback)|(127\.0\.0\.1\.?$))
>#
># accept from your upstream site
>#
># WSI is using ldm4 protocol so the accept is still required
>#accept        WSI
>#    .*
>#    ^[a-z].*\.uni\.wsicorp\.com$
>#
>##############################################################################
> #
># End 
>##############################################################################
> #


>From address@hidden Tue Jul 15 14:05:10 2003

Tom,

I want to thank you so very much for cleaning up my ldmd.conf file, which I 
adapted many years ago and has some pretty old stuff in it. - like my old
garage at home. ;)

I also appreciate your suggestion in combining some request lines.  I have
noticed that there are a lot of rpc.ldmd lines when I do "ldmadmin ps". I
just assumed that it was normal. Silly me!

Thank you also for suggesting to request the CONDUIT stream from
the University of Wisconsin/AOS machine. I will send Pete an email
asking him to add an allow for chisel.  I guess I didn't understand
fully Jeff's message when he mentioned something about these special
data feeds. 


Many thanks again.

Celia