Re: 20000913: Multiple feed types, dupilicate data, and downstreamclients

On Wed, 13 Sep 2000, Steven Danz wrote:

> Thanks for the info.  I sorta expected the answer, but I wanted to
> be sure I wasn't missing something.
> 
> Now for the other part of the problem.... The WMO and FSL5
> feedtypes overlap, and the non-overlapping part of the FSL5 feed can't
> be sent downstream for various reasons.  So, the downstream client isn't
> allowed FSL5, but they want the WMO part (since any 'early' FSL5
> product will cause the WMO product to be thrown out as a dup).  It
> sounds like I'm stuck either taking out the FSL5 feed from the intermediate
> server, or somehow allowing FSL5 to be shipped downstream since
> ldmd.conf doesn't have the ability to restrict products being sent
> downstream... True?

Steve,

If you have control of the intermediate server, it could only request the
subset of the FSL5 products ( WMO type products ) from the FSL5 server.
Therefore the downstream site could receive WMO|FSL5 from the
intermediate server and only get the subset of products equal to the WMO.
If you have other machines that need the overlap products they could
obtain them directly from the FSL5 server. That's a solution, don't know
if it will work for you.

Robb... 




> 
> Thanks again!
> 
> Steven
> 
> Robb Kambic wrote:
> 
> > On Wed, 13 Sep 2000, Unidata Support wrote:
> >
> > >
> > > ------- Forwarded Message
> > >
> > > >To: LDM Support at Unidata <address@hidden>
> > > >From: Steven Danz <address@hidden>
> > > >Subject: Multiple feed types, dupilicate data, and downstream clients
> > > >Organization: Aviation Weather Center
> > > >Keywords: 200009122100.e8CL0db03231
> > >
> > > Hello
> > >
> > > I have a situation where I have two different LDM feeds with
> > > similar data sets, call them WMO and FSL5 for the feed types
> > > for the sake of argument.  They both carry NWSTG data, just
> > > non-identical sets of it.  So, the FSL5 data is used as a 'backup'
> > > for the case if/when the data line feeding the WMO server is out.
> > >
> > Steven,
> >
> > The LDM does product duplication on the product regardless of what
> > feedtype the product arrives under.
> >
> > > If I have a client that requests data from both servers, then the rpc.ldmd
> > > does dup elimination whenever data from one is exactly the same as the
> > > other.  Good.  So, for local processing my pqact.conf has to have both
> > > feed types listed on each action or else I may not process the data thru
> > > the decoders. Fine, I've come to grips with that ;-).
> >
> > That's true, so the pqact entries are similar to this:
> >
> > WMO|FSL5        ^S(A....|P....|XUS91) .... ([0-3][0-9])([0-2][0-9])
> >        STDIOFILE       data/surface/sao/(\2:yyyy)(\2:mm)\2\3_sao.wmo
> >
> >
> > >
> > > Now, what about downstream clients to my 'intermediate' server?  Will
> > > they need to request both WMO and FSL5 from my intermediate server
> > > to get all the data either feed type?
> >
> > Yep, the LDM does not change the products in any manner, including
> > changing feedtypes
> >
> > It feels like they will since the intermediate
> > > rpc.ldmd tossed the dup the clients downstream from it will not see the 
> > > data...
> > > Of course, rpc.ldmd could say 'hey, this FSL5 data is the same as the
> > > WMO, don't reinsert it in the queue since it is a dup, but take the queued
> > > data for WMO and call it FSL5 and send it to the downstream clients'.
> > >
> > > Is there a 'right'/better way to do this?  Should both data sources be 
> > > setup
> > > as the same feedtype?
> >
> > Yes, that's how the top source sites are configured for the IDD.  The top
> > sites receive the data from a satellite ingestor and also from one other
> > source site using the same feedtype. That way the downstream node can
> > request only one feedtype even though the products are coming from the
> > different sources.  The LDM stats files show the product source, it's
> > field 8.  In the example below, 634 HDS products came from UCAR satellite
> > noaaport.unidata.ucar.edu and 276 HDS products came from
> > sunshine.ssec.wisc.edu for hour 2000091319.
> >
> > ldm-5.1.2 motherlode.ucar.edu 2000091319 HDS 634 1951390 +
> > 20000913193456 noaaport.unidata.ucar.edu 0.04 1@3234
> >
> > ldm-5.1.2 motherlode.ucar.edu 2000091319 HDS 276 842606 +
> > 20000913193358 sunshine.ssec.wisc.edu 0.68 3@3051
> >
> > I think this would be a much simpler way of operating because one can
> > still see the status of the source sites.
> >
> > The ldmprods man page has the description of all the fields.
> >
> > Robb...
> >
> >  Confused minds want to know...
> > >
> > > Thanks
> > >
> > > Steven Danz
> > >
> > > +------------+              +---------------------+             
> > > +-------------------+
> > > | WMO Server |------------->| Intermediate Server |------------>| 
> > > Downstream Client |
> > > +------------+              +---------------------+             
> > > +-------------------+
> > >            +----------------------+    ^
> > >            | "Backup" FSL5 Server |----+
> > >            +----------------------+
> > >
> > >
> > > ------- End of Forwarded Message
> > >
> >
> > ===============================================================================
> > Robb Kambic                                Unidata Program Center
> > Software Engineer III                      Univ. Corp for Atmospheric 
> > Research
> > address@hidden                   WWW: http://www.unidata.ucar.edu/
> > ===============================================================================
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================