Re: 20040909: Possible pqact issue in LDM?

Steven,

>Date: Fri, 10 Sep 2004 13:53:20 -0500
>From: "Steven Danz" <address@hidden>
>Organization: Aviation Weather Center
>To: Steve Emmerson <address@hidden>
>Subject: Re: 20040909: Possible pqact issue in LDM?
>Keywords: 200409091803.i89I3pnJ023109

The above message contained the following:

> >Is there anything relevant in the LDM logfile from the pqact(1) process?
> 
> Same problem, different angle.  Trying between meetings to delve into the
> workings of pqact and how products are tagged and how pqact knows
> where it left off.  Could there be some sort of race condition between
> an inserter and its use of set_timestamp() and how/when it actually
> performs the pq_insert() with pqact and when it updates
> its cursor into the queue?

pqact(1)'s "cursor" is a timestamp from the system clock together with
a data-product byte-offset.  When a ata-product is inserted into the
product-queue, a reference to it is added to a sorted map whose key
comprises the data-products's insertion-time and byte-offset.  The
insertion-time is the major component of the key and the byte-offset is
the minor component.  When pqact(1) looks for the "next" data-product,
it uses the pq(3) library to get the data-product whose key is the least
greater than the "cursor".

A data-product could be missed if the system clock were set backwards.
How is the system clock kept accurate on your system?

> I'm looking at the code for the utility that reads our NOAAPort stream
> and inserts the products into the queue, and I'm wondering if its
> doing 'bad things' in how it works with the queue that may be causing
> this problem.  I'm wondering if there is too much work being done
> between the set_timestamp() and the actual pq_insert() in such a way
> that the product is inserted and pqact views it as 'old' and doesn't
> processes it.

That shouldn't be a problem.  The insertion-time is not settable by
clients of the pq(3) module and that module ensures that readers (such
as pqact(1)) are locked-out while a data-product is being inserted.

> Again, not real sure yet how it all plays together so I could
> very well be looking in the wrong area.  Thanks!

Regards,
Steve Emmerson