Re: 20020618: LDM - feed type problems

Unidata Support wrote:
> 
> ------- Forwarded Message
> 
> >To: address@hidden
> >From: Harry Edmon <address@hidden>
> >Subject: LDM - feed type problems
> >Organization: UCAR/Unidata
> >Keywords: 200206181536.g5IFaj614056
> 
> Today's NCAR outage pointed out to me another deficiency in the current 
> system.
> Let's use my topology:
> 
> Feeding sunny: WMO  - thelma, MCIDAS - unidata.ssec.wisc.edu
> 
> Feedtype to air from sunny - UNIDATA
> 
> Now, thelma had been unavailable for 30 minutes, but the MCIDAS feed continued
> to work.  I stopped and restarted sunny with WMO from LSU.  When air requested
> data again from sunny after the restart - the request time was the last 
> UNIDATA
> product received, which was the last MCIDAS product.  Since the MCIDAS feed 
> had
> not been disrupted, this was a very recent time.  Thus, air would not have
> received the missing 30 minutes of WMO data!  I fixed this by changing air to
> seperately request MCIDAS and WMO.
> 
> This points out a serious architectural problem with the LDM - one I think we
> have discussed before.  The times that really should be used in the requests 
> to
> upstream hosts are the times the product was inserted into the upstream host's
> queue, not the time the product was inserted into the LDM system by the
> originating host.
> 
> --
> Dr. Harry Edmon                 E-MAIL: address@hidden
> 206-543-0547                            address@hidden
> Dept of Atmospheric Sciences    FAX:    206-543-0308
> University of Washington, Box 351640, Seattle, WA 98195-1640
> 
> ------- End of Forwarded Message

Hi Harry,

I agree that the LDM has a serious architectural problem.  It orders
products in the local queue by local ingest time, not IDD ingest time. 
(It does, however, use IDD ingest timestamp of the product in forming
the request.)  We discussed changing it to order products by IDD ingest
time, but found that there were scenarios where that did not solve the
problem of skipping products that should be relayed.  

Are you suggesting ordering products by time of ingest at the upstream
hosts or ordering products by local ingest time but stamping them with
the time of ingest at the upstream host?  But, I'm not convinced either
approach fixes the problem.  In your scenario, all air knows is the
timestamp of its most recent product from sunny, the MCIDAS product. 
Whether that time stamp is IDD ingest time or sunny's ingest time (which
would be even later than IDD ingest time), that time is still probably
later than some products from thelma.  How could this help air in making
a request?  I'm probably missing something.

Perhaps a workable solution would be to request starting from the
*oldest* product from a particular host, rather than the newest.  This
would cause duplicates to be relayed, but seems like it would eliminate
the problem of skipping products that should be relayed.  It would be
better than using the -o option, which determines a time value relative
to "now", rather than relative to anything in the queue.  I'm coming to
think that given our current architecture the best we can do is either
miss products or relay duplicates.

Another solution is to have a separate connection for each feed type, as
you did, although this doesn't scale well and may use connections poorly
in case of a sporadic feed type.

(INN, the news server, keeps a queue of products to be relayed to each
"peer".  This is a more complex solution, but also seems more correct.)

Btw, you probably noticed that I've been sidetracked since working on
using pqutil to log products for the purpose of tallying product loss. 
But, we feel it's important and I hope to get back to it soon.

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************