This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
>From: Robert Mullenax <address@hidden> >Organization: NMSU/NSBF >Keywords: 200303090454.h294sGB2028582 IDD latency Robert, >Is anyone else experiencing very high latencies with the DDPLUS|IDS feed? I checked the latencies on several machines that are reporting real time statistics, and don't see any systematic latency problems. Who is your upstream IDD feed site currently? It may be that your data originates from a machine that is not reporting real time statistics, so we can't tell if it is having problems. >I have tried all my different sources and the latency with the HDS (not >unusual) and IDS|DDPLUS (ver unusual) is 57-62 minutes. It doesn't seem >to be a bandwidth issue locally and the NNEXRAD feed has almost 0 latency. You can check on the latencies at sites reporting real time statistics at the following: Sites running LDM-5: http://www.unidata.ucar.edu/staff/chiz/latency/stats Sites running LDM-6: http://www.unidata.ucar.edu/staff/chiz/rtstats In either case, I recommend clicking on 'Statistics by Host' and then selecting an IDD host by name. After doing that, you will be presented with a list of feeds the site is getting and different views of data reception: latency, log(latency), volume, # products, and topology. Clicking on the latency link for any feed will result in a time series plot of the latency in seconds for that feed for that host. For sites running LDM-6.x, the differential latency from that host to its upstream feeder can be displayed by clicking on 'topology' and then selecting the upstream hostname link _if it is displayed as a link_. You can only calculate differential latencies if a site's upstream host is also reporting real time statistics. FYI, LDM-6 is in its final stages of test. We expect to announce its general availability real soon now (documentation is being worked on. If you would like to get a jump on LDM-6, and if you don't mind having to load new test versions as they become available, then I recommend that you load our latest release candidate, LDM-6.0.2: I. Download the LDM-6 release candidate and unpack it <login to your IDD machine as 'ldm'> cd ~ldm ftp ftp.unidata.ucar.edu <user> anonymous <pass> your_full_email_address cd pub/ldm/test binary get ldm-6.0.2.tar.Z quit zcat ldm-6.0.2.tar.Z | tar xvf - II. Define build the LDM-6 release candidate <make sure that the Unix environment variable LDMHOME is correctly defined in your .cshrc file; 'source' the file> cd ldm-6.0.2/src ./configure <- LDM-6.x is only available as source make <- if there were no configure errors make install <- if there were no errors in the make step sudo make install_setuids <- must be done as 'root'; do what is necessary cd ../bin III. Modify the LDM-6 startup script, ldmadmin, to match your current setup <compare entries in ldmadmin with those in the ~ldm/bin/ldmadmin you are currently using. Things to look out for are: 1) if a 'uname -n' does not return a fully qualified hostname for your machine, you will have to set the machine name in the hostname section of ldmadmin: change: chop($hostname = `uname -n`); # $hostname = "your.hostname.here"; to: # chop($hostname = `uname -n`); $hostname = "psnldm.nsbf.nasa.gov"; 2) modify the LDM product queue size, $pq_size, to match your existing queue 3) modify the number of LDM log files, $numlogs, to match the number of logs you want to keep on disk at any time IV. Add reporting of real time statistics in your ldmd.conf file add: exec "rtstats -h rtstats.unidata.ucar.edu" in the "exec" section of ~ldm/logs/ldmd.conf V. Examine your ldmd.conf file request lines and adjust them where necessary 1) look for "pathological" regular expressions in the requests. For instance the regular expression in this request line: request NNEXRAD ".*(FTG)" upstream_feed_host is pathological. The leading '.*' is not needed, and causes a LOT of extra work for certain system's (e.g. Sun Solaris) regular expression packages. An equivalent, and non-pathological regular expression is: request NNEXRAD "(FTG)" upstream_feed_host The general guideline we can offer is to make your regular expression as simple as possible. Leading and trailing '.*' fields sorrounding "stuff" that will match is not needed and undesirable. NOTE: the following is _NOT_ a pathological regular expression: request NNEXRAD ".*" upstream.feed.host Regular expression packages pretty much all know that a ".*" pattern means that no matching has to be done. 2) LDM-6 does not accumulate request lines with the same upstream host into a single request like LDM-5 did. For instance, the following two ldmd.conf lines are used differently by the different versions of the LDM: request NNEXRAD ".*" upstream.feed.host request HDS ".*" upstream.feed.host LDM-5 accumulates thes into a single rpc.ldmd to get the data: request NNEXRAD|HDS ".*" upstrea.feed.host LDM-6 uses two separate rpc.ldmds to get the data. The change was made to ease the process of splitting feed requests. If too much data is put on a single feed request, then produts may have increased latencies since they are waiting for transmit until others in the same request have been processed. If the upstream feed host is electronically "close" to your machine, then this issue practically disappears. If, on the other hand, the upstream feed host is electronically distant from the local host, then it has a big impact. The process of organizing feed requests into the fewest number (to keep the number of processes down on both the local host AND on the upstream feed site) while not putting too much on a single request is an iterative one. The guidance I can offer is best illustrated by one example: LDM-5 feed request: request UNIDATA ".*" upstream.feed.host request FLS2 ".*" upstream.feed.host LDM-6 feed requests equivalent but much better for keeping latencies down: request DDPLUS|IDS|FSL2 ".*" upstream.feed.host request HRS ".*" upstream.feed.host request UNIWISC ".*" upstream.feed.host I am _not_ suggesting that LDM-6 is less efficient than LDM-5 when moving data; it is _much_ more efficient. I _am_ suggesting that more attention needs to be paid to request ldmd.conf request lines so that overall latencies can be kept to a minimum. With proper configuration of ldmd.conf request entries and reasonable network connectivity, there is no reason why any continental US site can not have near-zero latencies for _any_ feed. This, by the way, includes the CONDUIT and CRAFT feeds. CONDUIT, for comparison, contains about 75% of the total volume of IDD feeds. We are able to relay all of this data to sites with suitable bandwidth with little to no latencies (I'm talking about one to a few seconds!). VI. Switch to using the new LDM cd ~ldm ldmadmin stop rm runtime ln -s ldm-6.0.2 runtime <make sure that all rpc.ldmd processes have exited> rehash ldmadmin start Note that LDM-6's ldmadmin will not return you to the Unix command prompt until all rpc.ldmd processes have exited. This is nice so we can stop admonishing users to "wait for all rpc.ldmd processes to exit"! After you LDM-6 release candidate is up and running, visit the real time statistics page(s) to see how you are doing. This will take a few minutes to show up, but not many. Please let me know if/when you are going to upgrade to LDM-6 Tom >From address@hidden Wed Mar 12 12:41:25 2003 Tom, At this time with the ULDB thing still going on I don't want to change the LDM. After it is finished I do want to get LDM 6 installed as soon as I can. I don't really understand how we end up with the latencies we do sometimes. I was using ldm.meteo.psu.edu and sunset.aos.wic.edu..they had no issues. Anyway at least as of last night it was toleable again. Thanks, Robert