This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hi, re: > I'm a part of the Weather team at the Johnson Space Center in > Houston. Our weather system uses multiple LDMs to obtain data > from external sites and redundancy. We have an issue with LDM > redundancy and duplicated files. Please see the below diagram > for our current LDM layout and the data flow (from left to right).\ > The question is as stated in the diagram below: To answer the question posed in the diagram you sent, an LDM will reject duplicate products if the duplicates are received while the original one(s) is(are) still in the LDM queue. NB: the LDM rejects duplicate products by comparing the MD5 signatures that are associated with each product. If the MD5 signatures differs at all, the LDM will see them as being the same. Given how the LDM reject duplicates, the question is how the MD5 signatures are being calculated for products originating from LDM1 and LDM2? The default way that an MD5 signature is calculated for a product (and that occurs when the product is first inserted into an LDM queue) is to use all of the bytes in the product in the calculation. If one is using the LDM utility 'pqinsert' to insert the products into the originating LDM queue, the calculation of the MD5 signature can either be set as the default or only using the characters in the product's Product ID. Since your LDMA and LDMB machines are seeing the "same" products as being different, then it is most likely that they are either a) not identical or b) have their MD5 signatures calculated differently. re: > Please let me know what you think. This file duplication issue is > a major issue for our weather data processing so we appreciate you inputs. Again, the question (posed above) is how the MD5 signatures are being calculated when the products are being inserted into the first LDM queue? The second question is how long after is a duplicate product received by either/both LDMA or LDMB? Associated with that question is what is the residency time of products in LDMA's and LDMB's queues? If you turned on LDM metrics gathering (via a cron entry), you will have a record of the age of the oldest product in the local LDM queue as a function of time. The entries most sites setup for doing this monitoring looks like: # # LDM-6.8.1+ metrics gathering # * * * * * bin/ldmadmin addmetrics > /dev/null 0 0 1 * * bin/ldmadmin newmetrics > /dev/null The last question for this email is have you setup metrics gathering on either/both LDMA or/and LDMB? re: > Thank you in advance! No worries. Cheers, Tom -- **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: BMK-187963 Department: Support LDM Priority: Normal Status: Closed =================== NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.