Re: 20000913: Multiple feed types, dupilicate data, and downstream clients

On Wed, 13 Sep 2000, Unidata Support wrote:

> 
> ------- Forwarded Message
> 
> >To: LDM Support at Unidata <address@hidden>
> >From: Steven Danz <address@hidden>
> >Subject: Multiple feed types, dupilicate data, and downstream clients
> >Organization: Aviation Weather Center
> >Keywords: 200009122100.e8CL0db03231
> 
> Hello
> 
> I have a situation where I have two different LDM feeds with
> similar data sets, call them WMO and FSL5 for the feed types
> for the sake of argument.  They both carry NWSTG data, just
> non-identical sets of it.  So, the FSL5 data is used as a 'backup'
> for the case if/when the data line feeding the WMO server is out.
> 
Steven,

The LDM does product duplication on the product regardless of what
feedtype the product arrives under.  

> If I have a client that requests data from both servers, then the rpc.ldmd
> does dup elimination whenever data from one is exactly the same as the
> other.  Good.  So, for local processing my pqact.conf has to have both
> feed types listed on each action or else I may not process the data thru
> the decoders. Fine, I've come to grips with that ;-).


That's true, so the pqact entries are similar to this:

WMO|FSL5        ^S(A....|P....|XUS91) .... ([0-3][0-9])([0-2][0-9])
       STDIOFILE       data/surface/sao/(\2:yyyy)(\2:mm)\2\3_sao.wmo
 

> 
> Now, what about downstream clients to my 'intermediate' server?  Will
> they need to request both WMO and FSL5 from my intermediate server
> to get all the data either feed type?  

Yep, the LDM does not change the products in any manner, including
changing feedtypes

It feels like they will since the intermediate
> rpc.ldmd tossed the dup the clients downstream from it will not see the 
> data...
> Of course, rpc.ldmd could say 'hey, this FSL5 data is the same as the
> WMO, don't reinsert it in the queue since it is a dup, but take the queued
> data for WMO and call it FSL5 and send it to the downstream clients'.
> 
> Is there a 'right'/better way to do this?  Should both data sources be setup
> as the same feedtype? 

Yes, that's how the top source sites are configured for the IDD.  The top
sites receive the data from a satellite ingestor and also from one other
source site using the same feedtype. That way the downstream node can
request only one feedtype even though the products are coming from the
different sources.  The LDM stats files show the product source, it's
field 8.  In the example below, 634 HDS products came from UCAR satellite
noaaport.unidata.ucar.edu and 276 HDS products came from
sunshine.ssec.wisc.edu for hour 2000091319.


ldm-5.1.2 motherlode.ucar.edu 2000091319 HDS 634 1951390 +
20000913193456 noaaport.unidata.ucar.edu 0.04 1@3234

ldm-5.1.2 motherlode.ucar.edu 2000091319 HDS 276 842606 +
20000913193358 sunshine.ssec.wisc.edu 0.68 3@3051


I think this would be a much simpler way of operating because one can
still see the status of the source sites.

The ldmprods man page has the description of all the fields.


Robb...

 Confused minds want to know...
> 
> Thanks
> 
> Steven Danz
> 
> +------------+              +---------------------+             
> +-------------------+
> | WMO Server |------------->| Intermediate Server |------------>| Downstream 
> Client |
> +------------+              +---------------------+             
> +-------------------+
>            +----------------------+    ^
>            | "Backup" FSL5 Server |----+
>            +----------------------+
> 
> 
> ------- End of Forwarded Message
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================