[Fwd: Re: 20010215: pqing stopped inserting when downstream requestedfeed]

-------- Original Message --------
Subject: Re: 20010215: pqing stopped inserting when downstream
requestedfeed
Date: Sun, 18 Feb 2001 19:54:32 +0000
From: Jessica Thomale <address@hidden>
To: Steve Chiswell <address@hidden>

Steve,

Thank you for pointing me to the LDM trouble shooting web page. 
Increasing
to queue size to 1GB has solved our crashing problem.

We appreciate your responsiveness,

Jessica

-- 
Jessica M. Thomale
Oklahoma Climatological Survey
E-mail: address@hidden
Mail: 100 E. Boyd, Suite 1210 Norman, OK 73019-1012
Phone: (405) 325-7809
Fax: (405) 325-2550





On Fri, 16 Feb 2001, Russ Rew wrote:

> Anne,
> 
> I just talked to Chiz and also talked to Dale Morris (Univ. of
> Oklahoma) at the CRAFT workshop about the pqing problem Jessica
> Thomale had reported.  Chiz had noticed that there were only 127
> products in a 260 Mbyte queue:
> 
>  <131>Feb 15 00:10:33 pqing[38196]: pq_del_oldest: conflict on 65407472
>  <131>Feb 15 00:10:33 pqing[38196]: pq_insert: Resource temporarily
>  unavailable
>  <133>Feb 15 00:10:33 pqing[38196]: Exiting
>  <133>Feb 15 00:10:33 pqing[38196]:   Queue usage (bytes):300003328
>  <133>Feb 15 00:10:33 pqing[38196]:            (nregions):   14577
>  <133>Feb 15 00:10:33 pqing[38196]:   Duplicates rejected:       0
>  <133>Feb 15 00:10:33 pqing[38196]:   WMO Messages seen:       127
>  <133>Feb 15 00:10:33 pqing[38196]:   SOH/ETX missing  :         0
>  <133>Feb 15 00:10:33 pqing[38196]:   parity/chksum err:         0
>  <133>Feb 15 00:10:33 pqing[38196]:   WMO format errors:         2
>  <133>Feb 15 00:10:33 pqing[38196]:   FILE Bytes read:    260290889
> 
> so the products must be larger than what we see on WMO feeds.  
> 
> Dale Morris said they were actually using pqing for ingesting the
> imagery data on their Unisys NOAAPORT machine.  The GOES visible
> products are more than 25 Mbytes each, so that's probably one source
> of the problem.  We don't use pqing for ingesting huge products like
> this.  It takes about 1 1/2 minutes just to compute the MD5 signature
> of one of these, which is so long that pqing will drop other products
> while it's computing the signature.  Plus if these are in a 300 Mbyte
> queue, finding room for a new image (10% of the queue space) will
> require deleting so many products it will delay things still more.
> 
> This latter problem can be helped by using a much larger queue, but
> pqing still won't be ideal for huge products such as this imagery.
> For our non-SSEC NOAAPORT ingester, Chiz wrote a program that reads
> 5000 bytes of the image at a time and updates the MD5 signature
> calculation incrementally, so there are no big pauses.  This code
> isn't useful to OU because it uses a proprietary library to read bytes
> from the board, but Mike is checking into getting code from the
> manufacturer that would let us open the board and read from it as if
> it were a file.
> 
> In the mean time, I think all we can recommend is using a much larger
> queue and using pqing only for the products on the NWSTG channel of
> NOAAPORT.
> 
> --Russ
>