[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #PAK-752919]: product duplication

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [LDM #PAK-752919]: product duplication
  • Date: Wed, 21 Jun 2006 11:29:40 -0600
HI Karen,

> We have been using ldm6.0.14 to move some data (algorithm products)
> between machines for quite a while, but recently started using it for a
> "new" set of products and we seem to be having some problems.
> 
> Only a small subset of the data is getting in appropriately, and we are
> getting messages from pqinsert about "Product already in queue" even
> though the name of the products are unique.
> 
> An example is:
> Jun 21 11:06:02 nmqxrt-12 pqinsert: Product already in queue:
> 8a6ab6685c1f80411e1de1a9232ac46c     9009 20060621160602.253     DDS
> 000  /home/qpesums/data/realtime/vpr/KUDX/vpr/VPR_INST.20060621.1605.dat
> Jun 21 11:06:02 nmqxrt-12 pqinsert:     9009 20060621160602.259     DDS
> 000  /home/qpesums/data/realtime/vpr/KUDX/vpr/VPR_AVG.20060621.1605.dat
> Jun 21 11:06:02 nmqxrt-12 pqinsert: Product already in queue:
> 8a6ab6685c1f80411e1de1a9232ac46c     9009 20060621160602.445     DDS
> 000  /home/qpesums/data/realtime/vpr/KVNX/vpr/VPR_INST.20060621.1605.dat
> Jun 21 11:06:02 nmqxrt-12 pqinsert: Product already in queue:
> 8a6ab6685c1f80411e1de1a9232ac46c     9009 20060621160602.451     DDS
> 000  /home/qpesums/data/realtime/vpr/KVNX/vpr/VPR_AVG.20060621.1605.dat
> Jun 21 11:06:02 nmqxrt-12 pqinsert: Product already in queue:
> 8a6ab6685c1f80411e1de1a9232ac46c     9009 20060621160602.624     DDS
> 000  /home/qpesums/data/realtime/vpr/KVTX/vpr/VPR_INST.20060621.1605.dat
> Jun 21 11:06:02 nmqxrt-12 pqinsert: Product already in queue:
> 8a6ab6685c1f80411e1de1a9232ac46c     9009 20060621160602.630     DDS
> 000  /home/qpesums/data/realtime/vpr/KVTX/vpr/VPR_AVG.20060621.1605.dat
> 
> As you can see we are giving the full directory path for pqinsert, as
> some of the file names are common, but paths are not.  Yet each of the
> products is being given the same product ID according to pqinsert.
> Could you tell me how it determines whether a product is unique?  Is it
> possible our string is too long -- and the first parts are identical?
> Or is it that each of these is getting inserted every 5 minutes with
> only the very end of the filename (the .hhmm.data part being modified?
> And yet we are still getting *some* of them into the queue....

I think that the rejected data-products are identical.  pqinsert(1) computes
the data-product "signature", which is an MD5 checksum, of the data portion
of the data-product.  It doesn't use the product-identifier in computing the
signature.  If the data in the data-products that you're inserting is the same,
then the resulting signatures will be the same and the data-product will not
be allowed in the product-queue.

If you really want the same data to be assigned to multiple data-products, 
then you'll have to vary them somehow.  One way would be to insert a counter
somewhere in the data-product.  Another way would be to add a timestamp
to the data-product.  Anything that will make the data portion of the data-
product unique will work.

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: PAK-752919
Department: Support LDM
Priority: Normal
Status: Closed