[IDD #QBL-783661]: split LDM REQUEST for CONDUIT feed

Hi Rick,

re:
> Over the weekend I ran ldm for just conduit on my machine that was idle 
> except for ldm.
> For the half degree global gfs grids I there were at least 100000 grids for a 
> 12z model
> run. Our noaaport machines are getting less than 50k.

CONDUIT does not originate from the NOAAPort SBN, so the number of model
output products seen in NOAAPort derived feeds and CONDUIT will have no
relation to each other.

re:
> Do you have anyway of checking the
> amount of grids you are receiving from conduit?

We do not routinely count the number of grids received in the CONDUIT feed,
so we do not have any ready numbers to give you.

Some of the most useful products included in CONDUIT are manifests of the set
of files that were inserted into the original LDM queue where the products
then get distributed.  The entries in the manifest/status files show exactly
which products were inserted into the LDM queue successfully AND those that were
not inserted into the LDM queue.  One reason that products might not be inserted
into the LDM queue is when there is already a product that has the same
MD5 signature in the queue when the insertion is attempted.  This situation
is not often seen now, but it was routine several years ago since the
timestep file output by the model was prone to have the same field several
times.

Here is an LDM pattern-action file action that can be used to FILE
the status messages in the CONDUIT feed:

CONDUIT ^.status\.(data.*) [0-9][0-9][0-9][0-9][0-9][0-9]
        FILE    data/pub/native/conduit/status/\1

(you will likely need to adjust the pathname of the output file)

To see if you are getting all of the files that have been put
into the feed, you would:

- save the status files
- setup a notifyme invocation to log the receipt of each product
- potentially setup a notifyme to log the availability of each
  product the upstream LDM has in its queue
- compare what you receive with what was made available

re:
> Our machines running the newer version XCD are not currently ingesting 
> conduit grids.

OK.

re:
> I did do some verifications to make sure there were no duplicate grids 
> received.

Duplicate grids would only be received if the local LDM queue is small
enough that du0licates are not automatically rejected, AND the latency
of the redundant grids was large enough to exceed the residency time
in the local LDM queue.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: QBL-783661
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.