This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
>From: Celia Chen <address@hidden> >Organization: NCAR/RAP >Keywords: 200307151639.h6FGdSLd007355 LDM IDD rtstats largefile Celia, >I had >exe "rtstats -q /home/ldm/data/ldm.pq -d /home/ldm/logs -h >rtstats.unidata.ucar.edu" >in chisel's ldmd.conf file as the last exec line. Is there a reason for the >above line not working properly? If you really did have 'exe' instead of 'exec', and the ldmd.conf listing below shows that you did, then yes, this was the problem. 'exe' is not a recognized abbreviation for 'exec'. The -q, -d, and -h entries were OK, but not needed _if_ your LDM setup is such that the queue is in ~ldm/data and the log flie directory is ~ldm/logs. >However, I just replaced it with > exec "rtstats -h rtstats.unidata.ucar.edu" >hope this will send the statistics to you. We are now getting real time stats from chisel as can be seen from: http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+chisel.rap.ucar.edu From the latency plot for IDS|DDPLUS, I can say that chisel is getting the data from thelma with no delay. This tells me that if some METARs are coming in late, they were late in the NOAAPORT broadcast; and there is nothing we can do about that. >PS - Below is chisel's ldmd.conf file: OK, here are some suggestions for tuning your ldmd.conf file: 1) change the Exec Entries section to clean things up: exec "pqbinstats" exec "pqact -l /home/ldm/logs/pqact.log" exec "pqsurf -p ^(SACN43|S[APX]|FXUS43) -Q /home/ldm/data/pqsurf.pq -l logs/pqsurf.log etc/pqsurf.conf" exec "rtstats -h rtstats.unidata.ucar.edu" The suggested changes assume that the LDM queue can be found in ~ldm/data, and the log files are written into ~ldm/logs. 2) The NLDN feed is supposed to be available only by point-to-point feed from SUNY Albany. This is the agreement the Unidata community has with SUNY Albany. Breaching it could result in the community losing this feed. We should, therefore, not be relaying this data to you from the atm.geo.nsf.gov machine. request NLDN ".*" striker.atmos.albany.edu PRIMARY request NLDN ".*" atm.geo.nsf.gov ALTERNATE On top of this, striker.atmos.albany.edu is the injection machine for NLDN data. If it goes down, there will be no NLDN data from any machine. Given these two observations, please remove your NLDN request line that points to atm.geo.nsf.gov. I will be removing the allow for NLDN data from the ldmd.conf file on atm. Also, you are allowing a number of machines to feed 'ANY' from chisel. This is OK for RAP machines and mostly OK for other .ucar.edu machines, but is not OK for others like the .gov machines. The same comment can be made for WSI data. You should, therefore, change your allow lines for non-RAP machines from: allow ANY ... to: allow ANY-WSI-NLDN ... 3) The following suggestion will help cut down on the number of rpc.ldmd connections to upstream feeds sites; they should have no effect on your reception of data: - combine your DIFAX, FSL2, and MCIDAS data requests to thelma and atm: change the three lines: request FSL2 ".*" thelma.ucar.edu PRIMARY request DIFAX ".*" thelma.ucar.edu PRIMARY request MCIDAS ".*" thelma.ucar.edu PRIMARY and: request FSL2 ".*" atm.geo.nsf.gov ALTERNATE request DIFAX ".*" atm.geo.nsf.gov ALTERNATE request MCIDAS ".*" atm.geo.nsf.gov ALTERNATE to the single lines: request DIFAX|FSL2|UNIWISC ".*" thelma.ucar.edu PRIMARY request DIFAX|FSL2|UNIWISC ".*" atm.geo.nsf.gov ALTERNATE (UNIWISC is the accepted name for Unidata-Wisconsin datasteam products. MCIDAS still works, but it has been deprecated). Cutting down on the number of rpc.ldmds running on both your machine and the upstream feed host(s) will help both machines run more efficiently. 4) remove the request for NEXRD2 data to atm.geo.nsf.gov. atm does not receive this feed, so requesting the data from it does nothing more than add a load to your machine AND to atm. Unfortunately, there is no current failover machine for this feed. 5) I would normally ask you to remove the NMC2 (CONDUIT) feed to atm since atm doesn't have this data, but we are trying to get the weather service to feed atm. If/when this does happen, thelma will ingest CONDUIT from atm. Having chisel ingest from thelma and atm does not give you any real redundancy. Because of this, I think you would do better to request the CONDUIT stream from the University of Wisconsin/AOS machine. change: request NMC2 ".*" thelma.ucar.edu PRIMARY request NMC2 ".*" atm.geo.nsf.gov ALTERNATE to: request CONDUIT ".*" thelma.ucar.edu PRIMARY request CONDUIT ".*" f5.aos.wisc.edu ALTERNATE At some point in the near future (not now), you might be asked to change this to: request CONDUIT "[09]$" f5.aos.wisc.edu PRIMARY request CONDUIT "[18]$" f5.aos.wisc.edu PRIMARY request CONDUIT "[27]$" f5.aos.wisc.edu PRIMARY request CONDUIT "[36]$" f5.aos.wisc.edu PRIMARY request CONDUIT "[45]$" f5.aos.wisc.edu PRIMARY request CONDUIT ".*" atm.geo.nsf.gov ALTERNATE You will need to contact Pete Pokrandt <address@hidden> at UW/AOS to get him to add an allow for CONDUIT for chisel. Combining the above recommendations, your cleaned-up ldmd.conf file would look like: ############################################################################### # Exec Entries ############################################################################### # exec "pqbinstats" exec "pqact -l /home/ldm/logs/pqact.log" exec "pqsurf -p ^(SACN43|S[APX]|FXUS43) -Q /home/ldm/data/pqsurf.pq -l /home/ldm/logs/pqsurf.log etc/pqsurf.conf" # # rtstats uses the LDM to send product latency statistics to the UPC. # We are currently evolving from pqbinstats to rtstats, but in the # mean time we hope sites will run both. For more info on rtstats # please see the man page. exec "rtstats -h rtstats.unidata.ucar.edu" # ############################################################################### # Request Entries ############################################################################### # # LDM5 servers request data from Data Sources # # request <feedset> <pattern> <hostname pattern> # # 1/3/2003 Request lightning data from Albany: # address@hidden (David Knight) # request NLDN ".*" striker.atmos.albany.edu PRIMARY # # Request data from FSL - ACARS UATurbulence data # Request data from FSL - AMDAR data # request PCWS "^FSL\.NetCDF\.ACARS\.QC\." eldm.fsl.noaa.gov request FSL3 "^FILTER\.ACARS\.UATurbulence\." eldm.fsl.noaa.gov # # DIFAX - DIFAX-like products from University of Wisconsin/AOS # FSL2 - FSL wind profiler # UNIWISC - GOES satellite image sectors and products from UW/SSEC request DIFAX|FSL2|UNIWISC ".*" thelma.ucar.edu PRIMARY request DIFAX|FSL2|UNIWISC ".*" atm.geo.nsf.gov ALTERNATE # # IDS|DDPLUS - NOAAPORT global observations request WMO ".*" thelma.ucar.edu PRIMARY request WMO ".*" atm.geo.nsf.gov ALTERNATE # # NNEXRAD - NOAAPORT NEXRAD Level III data request NNEXRAD ".*" thelma.ucar.edu PRIMARY request NNEXRAD ".*" atm.geo.nsf.gov ALTERNATE # # CONDUIT - High resolution model output from NCEP request CONDUIT ".*" thelma.ucar.edu PRIMARY request CONDUIT ".*" f5.aos.wisc.edu ALTERNATE # # NEXRD2 - NEXRAD Level-2 data NEXRD2 / NMC3 request NEXRD2 ".*" thelma.ucar.edu PRIMARY # # EXP request EXP ".*" level.rap.ucar.edu request EXP ".*" turb.rap.ucar.edu # # New WSI data feed (10/21/1998 - Celia Chen RAP/NCAR) request WSI ".*" iita.rap.ucar.edu #request WSI ".*" wsihcsn.unidata.ucar.edu ############################################################################### # Begin Access control ############################################################################### ############################################################################### # ALLOW: Who we are willing to feed # Allow Entries # allow <feedset> <hostname pattern> ############################################################################### # # Giving permission for a Data Sink to perform a request to your LDM # # allow <feedset> <hostname pattern> # # Giving permission to your own machine and Unidata # # Under no circumstances comment out the next allow entry to localhost # The LDM will NOT start if the lines are commented out. # allow ANY ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$ )) allow ANY ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.rap\.ucar\.edu\.?$)) ############################################################################### # send anything to your own machine (and other RAP LDM servers) allow ANY chisel allow ANY level allow ANY iita allow ANY ofour allow ANY front allow ANY turb allow ANY moonbow.rap.ucar.edu allow ANY serac.rap.ucar.edu allow ANY saint.rap.ucar.edu allow ANY ^((localhost|loopback)|(127\.0\.0\.1\.?$)|(^[a-z].*\.rap\.ucar\.edu\.?$)) # ATEC machines (128.117.200.84 atec-ingest) # ATEC machines (128.117.200.11 mm5-dev) # ATEC machines (128.117.200.101 uae-c1) allow ANY uae-c1.rap.ucar.edu allow ANY atec-ingest allow ANY mm5-dev # # Ceiling and Visibilities Project: allow ANY ceil allow ANY vis allow ANY shaggy allow ANY scrappy allow ANY scooter allow ANY speeder allow ANY wan allow ANY scooby # AWC machines allow ANY-WSI-NLDN adds.aviationweather.gov allow ANY-WSI-NLDN amelia.kc.noaa.gov allow ANY-WSI-NLDN orville.kc.noaa.gov allow ANY-WSI-NLDN wilbur.kc.noaa.gov allow ANY-WSI-NLDN 204.227.127.34 allow ANY-WSI-NLDN 204.227.127.38 allow ANY-WSI-NLDN 204.227.127.39 allow ANY-WSI-NLDN 204.227.127.40 allow EXP adds.awc-kc.noaa.gov allow EXP amelia.kc.noaa.gov allow EXP orville.kc.noaa.gov allow EXP wilbur.kc.noaa.gov # # UCAR LDM servers allow ANY thelma.ucar.edu allow ANY ldm.comet.ucar.edu allow ANY gizmo.comet.ucar.edu allow ANY zero.unidata.ucar.edu allow ANY box.mmm.ucar.edu allow ANY tea.mmm.ucar.edu allow ANY 128.117.88.210 # # FSL LDM servers allow ANY-WSI-NLDN eldm.fsl.noaa.gov allow ANY-WSI-NLDN 137.75.1.51 # WSI LDM servers allow ANY-WSI-NLDN wsi-204-115.wsi.com # Other outside Users allow ANY-WSI atm.geo.nsf.gov allow EXP sirocco.srcc.lsu.edu allow EXP rainbow.eas.purdue.edu allow EXP sunset.meteor.wisc.edu allow EXP unidata.ssec.wisc.edu allow EXP grayskies.atmos.washington.edu allow EXP drier.atmos.washington.edu allow EXP dry.atmos.washington.edu allow EXP redwood.atmos.albany.edu allow WMO wnnh001.wind.ttu.edu # ############################################################################### # Accept Entries ############################################################################### # ACCEPT: Who can feed us, currently this action is ONLY needed for WSI data # # accept <feedset> <pattern> <hostname pattern> # # accept anything from yourself # #accept ANY ".*" ^((localhost|loopback)|(127\.0\.0\.1\.?$)) # # accept from your upstream site # # WSI is using ldm4 protocol so the accept is still required #accept WSI # .* # ^[a-z].*\.uni\.wsicorp\.com$ # ############################################################################### # End ############################################################################### If you have any questions on the above changes, or if you would like us to tune up your ldmd.conf file please let me know. Tom --- chisel.rap.ucar.edu's ldmd.conf file --- >##### ># $Id: ldmd.conf,v 1.16 2002/06/17 21:12:43 anne Exp $ ># Sample ldmd.conf for ldm5 >#### ># ># This is the main configuration file for the LDM server. All lines that start ># with a "#" sign are comments. ># ># To debug an LDM that hangs on start up, run the following from LDM home: ># % bin/rpc.ldmd -vl - -q data/ldm.pq etc/ldmd.conf ># ># If the LDM still hangs, comment out all lines in this file except noted ># allow line entry below, try again. ># >############################################################################## > # ># Exec Entries >############################################################################## > # ># ># Programs that can share a queue with rpc.ldmd server, these programs ># are started by the "exec" command and are in the same process group. ># ># Regarding pqbinstats, while pqbinstats will generate .stats files in ># your logs directory, it is necessary to also run 'ldmadmin dostats' ># once per hour via cron in order to email the stats to the UPC and ># remove the .stats files that are no longer needed. See ># http://www.unidata.ucar.edu/packages/ldm/ldmConfigInstallList.html#s7 ># for more information. >#exec "pqbinstats" >exec "pqbinstats -d /home/ldm/logs" >exec "pqact -l /home/ldm/logs/pqact.log -d /home/ldm /home/ldm/etc/pqact.con > f" >exec "pqsurf -p ^(SACN43|S[APX]|FXUS43) -Q /home/ldm/data/pqsurf.pq -l /home > /ldm/logs/pqsurf.log -d /home/ldm /home/ldm/etc/pqsurf.conf" ># ># ># rtstats uses the LDM to send product latency statistics to the UPC. ># We are currently evolving from pqbinstats to rtstats, but in the ># mean time we hope sites will run both. For more info on rtstats ># please see the man page. ># exe "rtstats -q /home/ldm/data/ldm.pq -d /home/ldm/logs -h rtstats.unidata. > ucar.edu" >exec "rtstats -h rtstats.unidata.ucar.edu" ># send info back to local disk - NOT working: >#exec "rtstats -q /home/ldm/data/ldm.pq -d /home/ldm/logs -h " ># >############################################################################## > # ># Request Entries >############################################################################## > # ># ># LDM5 servers request data from Data Sources ># ># request <feedset> <pattern> <hostname pattern> ># >#request WMO ".*" uni0.unidata.ucar.edu ># ># 1/3/2003 Request lightning data from Albany: ># address@hidden (David Knight) ># >request NLDN ".*" striker.atmos.albany.edu PRIMARY >request NLDN ".*" atm.geo.nsf.gov ALTERNATE ># ># Request data from FSL - ACARS UATurbulence data ># Request data from FSL - AMDAR data ># >request PCWS > "^FSL\.NetCDF\.ACARS\.QC\." > eldm.fsl.noaa.gov > >request FSL3 > "^FILTER\.ACARS\.UATurbulence\." > eldm.fsl.noaa.gov ># ># Requesting data from thelma ># FSL2 - Profiler data > >request FSL2 ".*" thelma.ucar.edu PRIMARY >request FSL2 ".*" atm.geo.nsf.gov ALTERNATE > >#request IDS|DDPLUS >request WMO ".*" thelma.ucar.edu PRIMARY >request WMO ".*" atm.geo.nsf.gov ALTERNATE > >request EXP > ".*" > level.rap.ucar.edu >request EXP > ".*" > turb.rap.ucar.edu > ># Greg Thompson's (1/2003 Celia Chen) >#request EXP ># ".*" ># 129.118.105.15 > ># DIFAX feed >request DIFAX ".*" thelma.ucar.edu PRIMARY >request DIFAX ".*" atm.geo.nsf.gov ALTERNATE > ># ># get NEXRAD data > >request NEXRAD ".*" thelma.ucar.edu PRIMARY >request NEXRAD ".*" atm.geo.nsf.gov ALTERNATE > >request NMC2 ".*" thelma.ucar.edu PRIMARY >request NMC2 ".*" atm.geo.nsf.gov ALTERNATE > ># Level-2 data: NEXRD2 / NMC3 > >request NEXRD2 ".*" thelma.ucar.edu PRIMARY >request NEXRD2 ".*" atm.geo.nsf.gov ALTERNATE > ># >request MCIDAS ".*" thelma.ucar.edu PRIMARY >request MCIDAS ".*" atm.geo.nsf.gov ALTERNATE > ># New WSI data feed (10/21/1998 - Celia Chen RAP/NCAR) > >request WSI ".*" iita.rap.ucar.edu >#request WSI ".*" wsihcsn.unidata.ucar.edu >############################################################################## > # ># Begin Access control >############################################################################## > # > >############################################################################## > # ># ALLOW: Who we are willing to feed ># Allow Entries ># allow <feedset> <hostname pattern> >############################################################################## > # ># ># Giving permission for a Data Sink to perform a request to your LDM ># ># allow <feedset> <hostname pattern> ># ># Giving permission to your own machine and Unidata ># ># Under no circumstances comment out the next allow entry to localhost ># The LDM will NOT start if the lines are commented out. ># >allow ANY > ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$ > )) >allow ANY > ^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.rap\.ucar\.edu\.?$)) >############################################################################## > # > ># send anything to your own machine (and other RAP LDM servers) > >allow ANY chisel >allow ANY level >allow ANY iita >allow ANY ofour >allow ANY front >allow ANY turb > >allow ANY moonbow.rap.ucar.edu >allow ANY serac.rap.ucar.edu >allow ANY saint.rap.ucar.edu >allow ANY ^((localhost|loopback)|(127\.0\.0\.1\.?$)|(^[a-z].*\.rap\.ucar > \.edu\.?$)) > ># ATEC machines (128.117.200.84 atec-ingest) ># ATEC machines (128.117.200.11 mm5-dev) ># ATEC machines (128.117.200.101 uae-c1) >allow ANY uae-c1.rap.ucar.edu >allow ANY atec-ingest >allow ANY mm5-dev ># ># Ceiling and Visibilities Project: >allow ANY ceil >allow ANY vis >allow ANY shaggy >allow ANY scrappy >allow ANY scooter >allow ANY speeder >allow ANY wan >allow ANY scooby > ># AWC machines >allow ANY adds.aviationweather.gov >allow ANY amelia.kc.noaa.gov >allow ANY orville.kc.noaa.gov >allow ANY wilbur.kc.noaa.gov >allow ANY 204.227.127.34 >allow ANY 204.227.127.38 >allow ANY 204.227.127.39 >allow ANY 204.227.127.40 >allow EXP adds.awc-kc.noaa.gov >allow EXP amelia.kc.noaa.gov >allow EXP orville.kc.noaa.gov >allow EXP wilbur.kc.noaa.gov ># ># UCAR LDM servers >allow ANY thelma.ucar.edu >allow ANY ldm.comet.ucar.edu >allow ANY gizmo.comet.ucar.edu >allow ANY zero.unidata.ucar.edu >allow ANY box.mmm.ucar.edu >allow ANY tea.mmm.ucar.edu >allow ANY 128.117.88.210 ># ># FSL LDM servers >allow ANY eldm.fsl.noaa.gov >allow ANY 137.75.1.51 ># WSI LDM servers >allow ANY wsi-204-115.wsi.com > ># Other outside Users >allow ANY-WSI atm.geo.nsf.gov >allow EXP sirocco.srcc.lsu.edu >allow EXP rainbow.eas.purdue.edu >allow EXP sunset.meteor.wisc.edu >allow EXP unidata.ssec.wisc.edu >allow EXP grayskies.atmos.washington.edu >allow EXP drier.atmos.washington.edu >allow EXP dry.atmos.washington.edu >allow EXP redwood.atmos.albany.edu >allow WMO wnnh001.wind.ttu.edu ># >############################################################################## > # ># Accept Entries >############################################################################## > # ># ACCEPT: Who can feed us, currently this action is ONLY needed for WSI data ># ># accept <feedset> <pattern> <hostname pattern> ># ># accept anything from yourself ># >#accept ANY ".*" ^((localhost|loopback)|(127\.0\.0\.1\.?$)) ># ># accept from your upstream site ># ># WSI is using ldm4 protocol so the accept is still required >#accept WSI ># .* ># ^[a-z].*\.uni\.wsicorp\.com$ ># >############################################################################## > # ># End >############################################################################## > # >From address@hidden Tue Jul 15 14:05:10 2003 Tom, I want to thank you so very much for cleaning up my ldmd.conf file, which I adapted many years ago and has some pretty old stuff in it. - like my old garage at home. ;) I also appreciate your suggestion in combining some request lines. I have noticed that there are a lot of rpc.ldmd lines when I do "ldmadmin ps". I just assumed that it was normal. Silly me! Thank you also for suggesting to request the CONDUIT stream from the University of Wisconsin/AOS machine. I will send Pete an email asking him to add an allow for chisel. I guess I didn't understand fully Jeff's message when he mentioned something about these special data feeds. Many thanks again. Celia