This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hi, re: > Fist of all I would like to thank you for your help in > trying to solve my problem. No worries. > I made the changes as suggested, spliting the request as > by your e-mail. The results was good if you look at the > statistics (laency bellow 500). This was our objective. > However there is now a new > problem because no file are reveived with the size it > should have ( comparing with the ncep's computer data file > size). The size is almost ok since as an exemple, instead > of the order of 27048228 bytes I am receiving around > 21873520 --> all files with about the same file size are > being received (but with no correct file size). Please remember that each product in the CONDUIT datastream is a single GRIB/GRIB2 message, and none of these messages will be larger than a few hundred thousand kilobytes. The size you note above represents the sum of all GRIB/GRIB2 messages produced for a model run. Also, comparing the total size of products received with the size published by NCEP can differ since the LDM injection process will eliminate duplicate products. We regularly see one or more duplications of a GRIB/GRIB2 message in the model output file. The LDM's duplicate product detection and rejection mechanism will send the first instance of the product, but not the duplicates. In order to convince yourself that you are receiving all of the data that is being sent in the IDD feed, you should compare what you received with the list of files that were sent. The list of files sent is contained in a catalog file that is added at the end of the stream of GRIB/GRIB2 messages. You can learn more about the catalog file at: http://www.unidata.ucar.edu/data/conduit/ldm_idd/index.html > Just to remember you, if I make one request instead of > several requests I was receiving almost 80 % of data with > correct size; but with high latency. I would suspect that if you really are not seeing all the data, your pqact.conf action(s) are not writing all messages in the way you want. If they are not, then the size of your output file will differ from what the model creates. Also, if the model output file contains many duplicate fields, the volume of the data you receive will be less than what was originally available ** but you will have received everything so this should not be a problem **. > Today I tested: seting computer clock to UTC but the > statistics shows negative values. So that I returned back > to local time ( UTC + 1 hour ). I would run your computer on your local time. The conversion to UTC will be automatic if your machine's clock is correct. > Also, I started with -m option to see if makes any > difference. I don't think that you are not receiving products that are available. However, to make sure, please send us the following: - the size of your LDM queue - your ~ldm/etc/ldmd.conf file - your ~ldm/etc/pqact.conf file > Is there any other test I could make for try ? The statistics indicate that you should be receiving all of the data. Since you are now getting the data faster than you were previously, it is possible that your LDM queue size is not large enough to hold the data long enough to process it before it gets overwritten by new data being received. Letting us know the size of your LDM queue will go a long way towards identifying the possible problem you are seeing. My instinct tells me that the most likely things are: - your LDM queue is too small to allow processing of the products before getting overwritten by new ones received - the removal of duplicate products is making it look you are receiving less data than you should Cheers, Tom **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: TOL-308327 Department: Support IDD Priority: Emergency Status: Closed