[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: losing products

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: Re: losing products
  • Date: Tue, 06 Aug 2002 15:48:17 -0600
Robert,

Sorry to have taken so long to answer this ...

> I am still trying to figure out why we are missing some grib messages.
> We save all the thin AVN files into a raw directory and then run a
> check to see if they are all there before we do some processing.  We
> consistently are missing 10-20 messages.

We've discovered an interaction between the SDI ingest system and our
pqing code run at the ingest sites that results in some dropped
products.  For example, in the last hour, 3 GRIB products were dropped
because of this combination of bugs.  We're close to having a
workaround and a more robust ingest solution that won't have this
problem, but for now, the only "solution" is to get a redundant feed
from the wunderground.com ingester (you'd have to ask Jeff Masters for
exactly which host), where they're running an older version of the SDI
code that doesn't exhibit this particular problem.  That's what we're
doing in our top-level ring, so motherlode gets a few of its products
from wunderground and the rest from the other two top-level ingest
sites.

> I have my queue set to 975MB (full NOAAPORT SDI feed including
> NEXRAD).  I noticed I just got a nfree of 0 while running pqmon.  Is
> that a sign of a problem?
> 
> Aug 02 14:02:24 pqmon: Starting Up (877)
> Aug 02 14:02:24 pqmon: nprods nfree  nempty      nbytes  maxprods  maxfree
> minempty    maxext  age
> Aug 02 14:02:24 pqmon: 165379     1   72657   975003208    179024        8
> 59012       440 16397
> Aug 02 14:02:39 pqmon: 165442     0   72595   975003648    179024        8
> 59012         0 16395

No, nfree just goes to zero occasionally when the last free region in
the product queue has just been allocated and no more free regions are
available.  When the next product comes in, it will request a free
region and the queue code will delete the oldest products in the queue
until it can find one large enough, which will replenish the free
region list and cause nfree to increase to a larger value again.

--Russ