[LDM #EZK-400759]: Problem inserting int ldm prod queue - reg.

Sathya,

> I am Sathya, working for NWS in Silver Spring,MD. We are in the
> process of understanding and using LDM functionality for newer version
> of AWIPS.

Interesting! Are you with NCEP working on the National Centers Perspective for 
AWIPS-II?

> As the products are received from the SBN, we wanted to
> insert it into the LDM product queue. For now, we do NOT use
> dvb_multicast and readnoaaport processes. Instead we use our ingest
> processes.
> 
> We have developed a code (insert_prod_2_ldm) that is called from
> ldmd.conf during LDM startup. We have 3 instances of the same process
> (insert_prod_2_ldm) invoked from the  config file with different
> arguments. But all 3 instances correspond to the same channel (NWSTG).
> 
> First instance will process all text products; Second will handle all
> radar products; Third instance will handle all grib products.
> 
> Our code gets the data from our dvb ingest software, perform some
> operations on the product like uncompress/decrypt and writes to the
> pipe from where insert_prod_2_ldm process reads it. After computing
> md5 signature, the product is inserted into the ldm product queue.
> Insertion is done using the pq_insert API.
> 
> When I ran pqcat -s, the output is displayed as
> 
> ERROR: pqcat queueSanityCheck: Product count doesn't match
> products tallied: 3098  Value in queue: 9238
> Number of products: 3098

If products are being put into the product-queue, then the number of products 
that "pqcat -s" finds in the queue will be greater, in general, than the number 
of products that the queue contains because pqcat(1) doesn't lock the entire 
queue while it's tallying the products.

> We write to a log file after pq_insert returns successful. The number
> of products from the log file matches the "value in queue". In this
> case it is 9238. But the actual products in the queue is less than the
> number we tried to insert.

In order to determine if the difference between the number of products you 
inserted and the number of products in the queue is meaningful, would you 
please send me the output of the commands "ldmadmin config" and "pqmon".

> Also pq_insert returned success (0).

If pq_insert(3) returned 0, then the product was put into the queue.

> When
> we ran pqcat -p<prodid>, the number of matching products was 0. What
> is causing the products not get inserted into the queue? Do you think
> running multiple processes to insert into the queue is the cause for
> the issue. Based on the man pages, we assume access to ldm queue is
> synchronized with write lock. Not sure what is causing the products to
> be missed from insertion.

How certain are you that the products were inserted into the queue with 
product-identifiers that match the regular-expression you used? Do you have any 
logging turned-on that prints the metadata of the products that were inserted? 
If the only way products get into the queue is via your "insert_prod_2_ldm" 
processes, then just use the command "pqcat -vl- >/dev/null" to see what 
products are in the queue.

> Your timely help will be much appreciated.
> 
> Thanks,
> Sathya.
> 301.741.3489

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: EZK-400759
Department: Support LDM
Priority: Normal
Status: Closed