This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hi Elliott, re: > The hostname in the registry is "irads-ingest0.net.ou.edu". OK, that is what we thought. re: > In the registry, we have 'time-offset' lowered to 600 seconds. Even with that > low of a > period to backfill, it never catches up. In fact, it gets further behind. OK. re: > Yesterday evening, I tested idd.meteo.psu.edu. The same issue occurred. We > were able to > get a bit better performance by splitting the request into two, but it still > doesn't > perform well. Can you let us know when you split the feed REQUEST into two? I'm trying to understand the latency trace that we see in: http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD3+irads-ingest0.net.ou.edu This time series shows that the NEXRAD3 latency grew to over 3600 seconds sometime between 0 and 1 UTC yesterday, November 5. It dropped to near zero sometime around 2 UTC today, stayed low up until around 14 UTC today and then started getting worse. For some reason, there doesn't seem to be any latency information for the period of approx. 15 to 19 UTC today, and then latency values returned, but they are bouncing between high and low values. This last behavior makes me think that your LDM configuration file has the same feed REQUEST to both PSU and to somewhere else, and that somewhere else has content that is totally different than what is available from PSU. What I am looking at is the light pink (or some such color) near zero latency line that can be seen in the period from 15 to 19 UTC. Can you send us your LDM configuration file so we can take a look? If you'd rather not send us the entire file (as an attachment), please send the output of: grep -i ^request ~ldm/etc/ldmd.conf A potential problem I am trying to figure out is one caused by replicated feed REQUESTs to different upstreams that have different contents for the datastream(s) being REQUESTed. When a situation like this is present, the LDM will preferentially get products from one server over the other, and that will, in turn, mean that one would not get the products desired from one of the upstreams. The suspicious latencies are the ones from kfs-mini-01.mesonet. re: > While we've been focused on one host, irads-ingest0, at 156.110.246.56, we > are seeing the > same issue from echo-ingestA.services.ou.edu (129.15.2.32). It has been > tested against > idd.unidata.ucar.edu and idd.aos.wisc.edu and behaves the same as > irads-ingest0. The latency plot for NEXRAD3 on echo-ingesta.services.ou.edu is showing the exact same kind of thing as irads-ingest0, and the suspicious latency in the plot is also the one from the connection to kfs-mini-01.mesonet. Can you send us the LDM configuration file from echo-ingesta? re: > To add > to the issue, both hosts have started having issues receiving the NL2 TDS > feed as well. > > http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD3+echo-ingesta.services.ou.edu > http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD2+echo-ingesta.services.ou.edu The latency plot for NEXRAD2 for echo-ingesta is even stranger since some of the sites are coming in with very low latencies while a slug of others are showing very high latencies. re: > The two hosts are in separate networks on separate campuses, managed by > different parts of > the IT department. With that, we have put in a ticket with our ISP, OneNet. > They have > requested traceroutes from our upstream sources to our hosts, if possible, to > trace the > data path. Another clue: two machines in the NWC (ldmingest01.nwc.ou.edu and ldmingest02.nwc.ou.edu) are also showing higher than previous latencies for CONDUIT starting sometime on Sunday morning, November 4: http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+ldmingest01.nwc.ou.edu Is the NWC in the same building as you? Do you know who to contact as the LDM administrator of the NWC machines? We really want to find the contact since we are seeing LDM connections from their machines breaking and reestablishing connections on more than one backend real-server machines of our idd.unidata.ucar.edu cluster. re: > Would it be possible to get traceroutes from a host behind > idd.unidata.ucar.edu to > 129.15.2.32 and 156.110.246.56? Here are traceroutes from the real server backend machine that have connections from each of these IPs: From uni19.unidata.ucar.edu which is servicing a split NEXRAD3 and an NGRID feed to echo-ingesta.services.ou.edu/129.15.2.32: traceroute to 129.15.2.32 (129.15.2.32), 30 hops max, 60 byte packets 1 flr-n140.unidata.ucar.edu (128.117.140.251) 0.724 ms 0.715 ms 0.854 ms 2 ml2core-fl2core.unet.ucar.edu (128.117.243.194) 0.958 ms 1.449 ms 1.442 ms 3 corel3-ml2core-i2.unet.ucar.edu (128.117.243.141) 1.696 ms 1.692 ms 1.680 ms 4 v3454.rtr-chic.frgp.net (192.43.217.222) 23.159 ms 23.166 ms 23.160 ms 5 et-2-1-0.4079.rtsw.chic.net.internet2.edu (162.252.70.116) 23.742 ms 23.911 ms 23.729 ms 6 ae-3.4079.rtsw.kans.net.internet2.edu (162.252.70.141) 34.575 ms 34.530 ms 34.511 ms 7 et-7-0-0.4079.rtsw.tuls.net.internet2.edu (162.252.70.35) 38.497 ms 38.478 ms 38.876 ms 8 198.71.46.45 (198.71.46.45) 38.520 ms 38.519 ms 38.505 ms 9 164.58.244.44 (164.58.244.44) 38.564 ms 38.827 ms 38.793 ms 10 164.58.244.15 (164.58.244.15) 40.757 ms 40.768 ms 40.755 ms 11 164.58.245.55 (164.58.245.55) 40.836 ms 164.58.245.53 (164.58.245.53) 40.924 ms 40.707 ms 12 164.58.245.58 (164.58.245.58) 40.909 ms 40.907 ms 164.58.245.56 (164.58.245.56) 40.851 ms 13 164.58.244.33 (164.58.244.33) 41.414 ms 41.347 ms 41.386 ms 14 164.58.10.98 (164.58.10.98) 41.467 ms 41.463 ms 41.512 ms 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * From uni19.unidata.ucar.edu which to delta-ingest0.irads.ou.edu/156.110.246.56: traceroute to 156.110.246.56 (156.110.246.56), 30 hops max, 60 byte packets 1 flr-n140.unidata.ucar.edu (128.117.140.251) 0.743 ms 1.081 ms 1.155 ms 2 ml1core-flacore.unet.ucar.edu (128.117.243.78) 1.249 ms fl2core-flacore.unet.ucar.edu (128.117.243.106) 0.839 ms ml1core-flacore.unet.ucar.edu (128.117.243.78) 1.230 ms 3 ml2core-ml1core.unet.ucar.edu (128.117.243.99) 1.094 ms ml2core-fl2core.unet.ucar.edu (128.117.243.194) 1.374 ms ml2core-ml1core.unet.ucar.edu (128.117.243.99) 1.353 ms 4 corel3-ml2core-i2.unet.ucar.edu (128.117.243.141) 1.504 ms 1.499 ms 1.487 ms 5 v3454.rtr-chic.frgp.net (192.43.217.222) 23.160 ms 23.134 ms 23.136 ms 6 et-2-1-0.4079.rtsw.chic.net.internet2.edu (162.252.70.116) 23.471 ms 23.499 ms 23.592 ms 7 ae-3.4079.rtsw.kans.net.internet2.edu (162.252.70.141) 34.663 ms 34.482 ms 34.481 ms 8 et-7-0-0.4079.rtsw.tuls.net.internet2.edu (162.252.70.35) 38.407 ms 38.386 ms 38.270 ms 9 198.71.46.45 (198.71.46.45) 38.521 ms 38.490 ms 38.478 ms 10 164.58.244.44 (164.58.244.44) 38.839 ms 164.58.244.46 (164.58.244.46) 38.513 ms 164.58.244.44 (164.58.244.44) 38.588 ms 11 164.58.244.239 (164.58.244.239) 38.676 ms 38.644 ms 38.643 ms 12 164.58.16.26 (164.58.16.26) 38.481 ms 38.444 ms 38.428 ms 13 156.110.254.97 (156.110.254.97) 41.410 ms 41.195 ms 41.521 ms 14 156.110.254.62 (156.110.254.62) 41.003 ms 40.885 ms 156.110.254.50 (156.110.254.50) 45.609 ms 15 * * * 16 * * * 17 * * * 18 * * * 19 * * * 20 * * * 21 * * * 22 * * * 23 * * * 24 * * * 25 * * * 26 * * * 27 * * * 28 * * * 29 * * * 30 * * * NOTE: - While on uni19.unidata.ucar.edu, I saw that it was feeding lion.caps.ou.edu some CONDUIT data, and all of UNIWISC, NOTHER and NGRID. - the latency plots for NGRID and NOTHER for lion look much the same as for your machine and for the NWC machines: NOTHER http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NOTHER+lion.caps.ou.edu NGRID http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NGRID+lion.caps.ou.edu - also, it is a bit strange that there are two different names associated with the 156.110.246.56 IP address: ~: nslookup 156.110.246.56 Server: 208.67.222.222 Address: 208.67.222.222#53 Non-authoritative answer: 56.246.110.156.in-addr.arpa name = delta-ingest0.irads.ou.edu. 56.246.110.156.in-addr.arpa name = delta-ingest0.ou.edu. This is a bit strange, but it should have no bearing on the latency issues being investigated. - lastly, I don't see any connections from delta-ingest0.irads.ou.edu/156.110.246.56 on any of the real server backend machines that comprise the idd.unidata.ucar.edu cluster. Cheers, Tom -- **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: ZAJ-969398 Department: Support IDD Priority: Normal Status: Closed =================== NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.