This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hi Carissa, This is a quick note about two different topics related to the latencies that Unidata top level relays have experienced/see for the IDD CONDUIT data feed: - first, it would be most useful (and interesting!) to know what, if anything, was done at NCEP to drastically reduce the observed latencies in the CONDUIT feed FYI: - 20-way feed REQUEST splits by Penn State and Unidata did result in reduced latencies, but the 20-way split at Unidata did not return the latencies to earlier levels - I contacted Pete Pokrandt (UW/AOS) to see if he had/would change his 10-way REQUEST split for CONDUIT Before Pete could start changing his existing 10-way split to a 20-way split, the CONDUIT latencies plummeted and have remained as low as they have historically been during good periods. We would _love_ to know what was done to effect the drop in latencies at NCEP (or at some point in the network managed by NOAA). If nothing was done (really nothing that is), it would also be good to know as this might indicate that there was a problem in Internet2. While it is _extremely_ rare for there to be problems in Internet2, we and UW/SSEC did experience a situation where the Internet2 gateway connection to AWS East was significantly under performing, and the slowness affected our ability to uplink NEXRAD Level 2 data to the AWS S3 bucket that we have been populating for over 3 years as part of the NOAA Big Data project. - secondly, CONDUIT latencies have always exhibited a variation from near zero to 30 seconds These latencies are caused by the process(es) that are inserting products into the LDM queue on the CONDUIT origination machine NOT sending a CONT (continue) signal to the LDM to inform it that new products are available in the LDM queue. The default behavior of an LDM is to process (relay to downstreams, etc.) all products it finds in its queue; sleep for 30 seconds; and then wake up and check to see if there are any products to process. The result of this default behavior is latencies that range from zero/near-zero to 30 seconds. The step that needs to be done to eliminate this "artificial" 0-30 second latency is for the process that is inserting the product(s) into the LDM queue to send a CONT signal to the negative of the process ID of the lead LDM process. In practice, this looks like: /bin/kill -s CONT -`cat ~ldm/ldmd.pid` NB: - the system version of 'kill' must be used; 'kill' provided by, for instance, BASH does not work correctly - the file ~ldm/ldmd.pid contains the process ID of the lead LDM process for a running LDM - the CONT signal must be passed to the negative of the process ID for the lead LDm process So, my question to you is how we can go about enhancing the product insertion processes in use in NCEP to include sending of the CONT signal to the LDM so that it is knows each time a new product is inserted into its queue? Cheers, Tom -- **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu ****************************************************************************