Re: NLDM LDM future... 32 types, server side restriction.

Hello Peter,

Sounds like you've been doing some research regarding our relay
efforts.  While I can't promise to solve all your problems :) (wish I
could solve my own!), I'll try to enlighten you further regarding both
the LDM and NLDM.

Peter Silva wrote:
> 
> Hello Ms. Wilson,
> 
> I work for the Canadian weather service and we have been asked to
> provide access to some commercial clients using the LDM protocol.  To
> that end, Daniel has, over the the last six months, looked at LDM rather
> thoroughly, and encountered some issues with routing and data types.
> There is no discussion about future plans for the package anywhere on
> the web site or in the mailing lists.   On the other hand, I just
> noticed your intriguing paper about using NNTP as a transport for LDM
> data, so I though I might poke your brains.
> 
> There are two capabilities that were identified as key limitations in
> the current LDM, for our purposes:
> 
>     -- number of data types limited to 32.
>     -- inability to limit access to products on the server side.
> 
> The first limitation in the number of data types is troublesome because
> Canadian RADAR data is not the same as it´s American counterparts, and
> there will be other things, such as lightning or other types that we may
> want to add in the future.   All 32 types in the current design are
> taken, and there is no indication that it will change in the future.
> 

Regarding the LDM, I presume that you know about selecting from within a
feed type (what you're calling a 'data type') using regular
expressions.  These can be used in ldmd.conf (on the client end) to
specify more precisely what products are wanted, and in pqact.conf to
specify more precisely what actions to perform on which products.  Thus,
if all radar data is categorized within the same feed type, you can
select radars from, say, a particular station by using a regular
expression to match the stations portion of the product ID.  This is one
way to deal with the limited number of feed types, but I don't think it
solves your problem.

Also, I assume you know that if you're not using a feed type for
relaying data for the data types for which we've intended it, then you
can reuse it for some other purpose.  

We recognize that the limited number of feed type is a problem.  There
has been some talk of modifying the LDM to accomdate more feed types. 
It would still use a bitwise representation, but, unlike now, different
bit combinations would represent different feed types.  This way we
would get approximately 2^^32 feedtypes.  If we stay with LDM, this
modification would probably be critical.

However, we are indeed considering using NDLM for data relay, and one
reason for this is because of the virtually unlimited number of
hierarchically structured newsgroups.  Thus, I'm relaying radar data and
I've created over 100 newsgroups for that purpose - one for each
station.  Also, NNTP supports cross-posting.  So, in the future I will
create additional newsgroups based on product type (such as N0R, N1R, 
etc.).  Then, when a N0R product from station KABC arrives I will post
it to both the KABC group and the N0R group.  This will allow users to
select products easily by either station ID or product type.  In
general, I believe that cross posting will be useful to give users
different views of the data, although I have not yet tested this.

> The second limitation means that we cannot use the same server to feed
> multiple commercial clients who are paying for access to different data
> sets, since we cannot allow one client to access data without permitting
> all of them the same access.
> 

I'm not completely following this.  A server can specify which feed
types to feed to a client.  However, a server can't specify products
within a feed type.  (Specification within a feed type can be defined on
the client end.  This decision was made some time ago with the thought
that the network would consist entirely of "trusted" sites.)  I assume
that your problem is that there are not enough feed types to categorize
the products finely enough so that you can specify what products to send
strictly by feed type.  At least one other site dealing with propriety
data has mentioned this as a problem.  We have no plans to modify this
in the LDM.  


> The conclusion for the moment is that we have to run a separate instance
> of LDM on a separate server for every client, and feed each machine only
> the data that client has contracted for.   Each client will also be able
> to choose which "fake" data type we will feed them various types of data
> with, since there are no official type id's available.    The need to
> have a server per client is pains us from the aesthetic point of view,
> and will be costly to support.  We would much prefer to serve multiple
> client with a single machine, as we do now, with an in-house package
> built over FTP.
> 

I can see how this would an unsatisfying solution.  The problem is that
you're trying to use the LDM in a way in which it wasn't intended. 
Perhaps a more general design would have better years ago, but, here we
are...  (Although the LDM has worked very well for the purpose for which
it was intended.)


> So the questions are as follows:
>     -- Is NLDM being seriously considered for the next version of LDM.
>        Is their a time table for release ?
>     -- Will NLDM address the issues above, so that we can
>        serve commercial clients with a single LDM instance ?
> 

Yes, NLDM is being seriously considered.  I am building a prototype
network now for testing purposes.  Our plan is to make a decision as to
which route to pursue sometime this spring.  Turns out that NNTP and INN
(the open source news server on which NLDM is based) offer many features
that would be useful for us that LDM doesn't have.  One big downside
would be making the change within our community from the LDM protocol to
NNTP.  Also, INN is a much more complex package than LDM and thus harder
to configure.  But, we'll be considering all aspects of each approach
when the time comes.

Would NLDM solve your problem of wanting to relay different sets of data
to different clients?  I suspect it would because you can specify
newsgroups much more finely.  Also, with INN , you can specify
newsgroups negatively.  Thus your server could have a subscription list
for a client that looks something like this:

        unidata.binaries.radar.level3.*,!unidata.binaries.radar.level3.KABC

which would cause the server to relay all level 3 radar products to the
client except those from KABC.

At this point NLDM is simply INN with a statistics package built on
top.  (Although I did have to develop an encoding that is acceptable to
NNTP.)  That is, I have used NLDM "out of the box".  At the moment I
know of two minor modifications that I would like to make to INN to
better suit our purposes.  But, the point of this is to say that you
could try relaying with INN right now.  Just get the INN distribution
from ISC, encode your data, and you're ready to go.  Although INN is
complex to configure, I would be happy to share what I've learned with
you.   Also, you're welcome to try our encoding algorithm.  


> I hope you can solve all our problems :-)
> 

... or, perhaps complicate the picture even further...


> Thanks for your time.
> 

I hope this helps.   Please let me know if I can be of further
assistance.  I'm excited about this project and am happy to share what
I've learned.

Sincerely,

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                  P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************