This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
-------- Original Message -------- Subject: Re: 20010215: pqing stopped inserting when downstream requestedfeed Date: Sun, 18 Feb 2001 19:54:32 +0000 From: Jessica Thomale <address@hidden> To: Steve Chiswell <address@hidden> Steve, Thank you for pointing me to the LDM trouble shooting web page. Increasing to queue size to 1GB has solved our crashing problem. We appreciate your responsiveness, Jessica -- Jessica M. Thomale Oklahoma Climatological Survey E-mail: address@hidden Mail: 100 E. Boyd, Suite 1210 Norman, OK 73019-1012 Phone: (405) 325-7809 Fax: (405) 325-2550 On Fri, 16 Feb 2001, Russ Rew wrote: > Anne, > > I just talked to Chiz and also talked to Dale Morris (Univ. of > Oklahoma) at the CRAFT workshop about the pqing problem Jessica > Thomale had reported. Chiz had noticed that there were only 127 > products in a 260 Mbyte queue: > > <131>Feb 15 00:10:33 pqing[38196]: pq_del_oldest: conflict on 65407472 > <131>Feb 15 00:10:33 pqing[38196]: pq_insert: Resource temporarily > unavailable > <133>Feb 15 00:10:33 pqing[38196]: Exiting > <133>Feb 15 00:10:33 pqing[38196]: Queue usage (bytes):300003328 > <133>Feb 15 00:10:33 pqing[38196]: (nregions): 14577 > <133>Feb 15 00:10:33 pqing[38196]: Duplicates rejected: 0 > <133>Feb 15 00:10:33 pqing[38196]: WMO Messages seen: 127 > <133>Feb 15 00:10:33 pqing[38196]: SOH/ETX missing : 0 > <133>Feb 15 00:10:33 pqing[38196]: parity/chksum err: 0 > <133>Feb 15 00:10:33 pqing[38196]: WMO format errors: 2 > <133>Feb 15 00:10:33 pqing[38196]: FILE Bytes read: 260290889 > > so the products must be larger than what we see on WMO feeds. > > Dale Morris said they were actually using pqing for ingesting the > imagery data on their Unisys NOAAPORT machine. The GOES visible > products are more than 25 Mbytes each, so that's probably one source > of the problem. We don't use pqing for ingesting huge products like > this. It takes about 1 1/2 minutes just to compute the MD5 signature > of one of these, which is so long that pqing will drop other products > while it's computing the signature. Plus if these are in a 300 Mbyte > queue, finding room for a new image (10% of the queue space) will > require deleting so many products it will delay things still more. > > This latter problem can be helped by using a much larger queue, but > pqing still won't be ideal for huge products such as this imagery. > For our non-SSEC NOAAPORT ingester, Chiz wrote a program that reads > 5000 bytes of the image at a time and updates the MD5 signature > calculation incrementally, so there are no big pauses. This code > isn't useful to OU because it uses a proprietary library to read bytes > from the board, but Mike is checking into getting code from the > manufacturer that would let us open the board and read from it as if > it were a file. > > In the mean time, I think all we can recommend is using a much larger > queue and using pqing only for the products on the NWSTG channel of > NOAAPORT. > > --Russ >