[SCOOP #JBC-457912]: Thought on notification of reception of data

Hi Gerry,

re:
> We've been playing, in the SCOOP project, with notification of
> completion of an LDM transfer.  We still have some folks who are
> cowboying their LDM configs a bit... point-to-point rather than a
> hierarchical structure, lots of duplication, etc., because they're
> afraid data aren't being delivered and there's no clean way to check on
> delivery.

What we do in CONDUIT is create a catalog of all products that injected
into the originating LDM's queue and send that as an additional product
_after_ all of the real products.  If you recall, our CONDUIT feed
is created by 'carving' up model output files into individual GRIB
(GRIB2) messages and injecting each one individually.  The process that
is doing the injection (meaning the insertion into the originating
queue) records each product that it puts into the queue in what eventually
becomes a manifest for products that were supposed to be sent.  This
approach has worked very well in CONDUIT, so I strongly recommend that
you consider adopting it for SCOOP.

Another thing done in CONDUIT is the inclusion of a monotonically
increasing counter in the metadata of each product.  This allows
one to see if there are any missing products in the series of products
being sent.

> At this point, the main reason we (TAMU) don't get something from SCOOP
> would be due to not having a pqact entry pattern to work from.  With the
> plethora of patterns we're saving, we di discreet matching to each and
> then file it, update our local inventory database, and notify the SCOOP
> catalog of its availability.  Still, this doesn't seem to be enough.

So, you don't have a generic pattern that matches everything that can be
used to verify the receipt of the product independently of processing
(e.g., FILE) it?

> We've been thinking of using SOAP services, or something else, to notify
> folks, provide an RSS feed, update a webpage, or maybe take out a banner
> add on Google when things are in.

If the data delivery is serialized, then the last product in the set
could contain a known pattern in the name that is recognized by
a specific pqact.conf entry.  This is what is being done in the NEXRAD2
datastream.

> What I just got to thinking about, reviewing yet another gripe that
> something wasn't available, was to use 'notifyme' and a java application
> (or perl script) to parse the stream, look at the file names coming in,
> and fire the SOAP or other update.

Since the delivery of products is guaranteed in the LDM under the constraint
that the connection stays up and the downstream LDM queue is sufficient
to hold the products it is receiving, I see no real advantage in adding
a different service to  notify the user of the completion of the transfer
of data.

>   Are we reinventing the wheel?

In a way I do think you are reinventing the wheel.  It may be a good time
to step back and consider how data is currently flowing and decide on
a course of remedial action that:

- insures that the end user has an easy way to know that s/he has
  received all of the products that s/he was supposed to receive

- the same data products are not being sent redundantly through the
  system.  This not only wastes bandwidth (not a problem at TAMU,
  but potentially a problem at one or more other sites), but can cause
  products to be received more than once and potentially cause products
  that have been received to be scoured out of a receiver's queue before
  they have been processed.  The latter situtation is encountered when
  a receiver's LDM queue is too small, or they are receiving too much
  data to process, or their processing setup is not efficient enough to
  keep up with the stream of data coming in

> Worrying over nothing?

I would not say that you are worrying over nothing since it is very important
that the receiving sites have confidence that they are, in fact, receiving
the full set of products that they are supposed to be receiving.

> Do you have any thoughts on this?

Please read through the above and let me know if you would like to
talk further on any particular issue and/or open new avenues of
discussion.

> Thanks, Gerry
> --
> Gerry Creager -- address@hidden
> Texas Mesonet -- AATLT, Texas A&M University
> Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.862.3983
> Office: 1700 Research Parkway Ste 160, TAMU, College Station, TX 77843
> 
> 

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: JBC-457912
Department: Support IDD SCOOP
Priority: Normal
Status: Closed