This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
Hello Peter, Sounds like you've been doing some research regarding our relay efforts. While I can't promise to solve all your problems :) (wish I could solve my own!), I'll try to enlighten you further regarding both the LDM and NLDM. Peter Silva wrote: > > Hello Ms. Wilson, > > I work for the Canadian weather service and we have been asked to > provide access to some commercial clients using the LDM protocol. To > that end, Daniel has, over the the last six months, looked at LDM rather > thoroughly, and encountered some issues with routing and data types. > There is no discussion about future plans for the package anywhere on > the web site or in the mailing lists. On the other hand, I just > noticed your intriguing paper about using NNTP as a transport for LDM > data, so I though I might poke your brains. > > There are two capabilities that were identified as key limitations in > the current LDM, for our purposes: > > -- number of data types limited to 32. > -- inability to limit access to products on the server side. > > The first limitation in the number of data types is troublesome because > Canadian RADAR data is not the same as it´s American counterparts, and > there will be other things, such as lightning or other types that we may > want to add in the future. All 32 types in the current design are > taken, and there is no indication that it will change in the future. > Regarding the LDM, I presume that you know about selecting from within a feed type (what you're calling a 'data type') using regular expressions. These can be used in ldmd.conf (on the client end) to specify more precisely what products are wanted, and in pqact.conf to specify more precisely what actions to perform on which products. Thus, if all radar data is categorized within the same feed type, you can select radars from, say, a particular station by using a regular expression to match the stations portion of the product ID. This is one way to deal with the limited number of feed types, but I don't think it solves your problem. Also, I assume you know that if you're not using a feed type for relaying data for the data types for which we've intended it, then you can reuse it for some other purpose. We recognize that the limited number of feed type is a problem. There has been some talk of modifying the LDM to accomdate more feed types. It would still use a bitwise representation, but, unlike now, different bit combinations would represent different feed types. This way we would get approximately 2^^32 feedtypes. If we stay with LDM, this modification would probably be critical. However, we are indeed considering using NDLM for data relay, and one reason for this is because of the virtually unlimited number of hierarchically structured newsgroups. Thus, I'm relaying radar data and I've created over 100 newsgroups for that purpose - one for each station. Also, NNTP supports cross-posting. So, in the future I will create additional newsgroups based on product type (such as N0R, N1R, etc.). Then, when a N0R product from station KABC arrives I will post it to both the KABC group and the N0R group. This will allow users to select products easily by either station ID or product type. In general, I believe that cross posting will be useful to give users different views of the data, although I have not yet tested this. > The second limitation means that we cannot use the same server to feed > multiple commercial clients who are paying for access to different data > sets, since we cannot allow one client to access data without permitting > all of them the same access. > I'm not completely following this. A server can specify which feed types to feed to a client. However, a server can't specify products within a feed type. (Specification within a feed type can be defined on the client end. This decision was made some time ago with the thought that the network would consist entirely of "trusted" sites.) I assume that your problem is that there are not enough feed types to categorize the products finely enough so that you can specify what products to send strictly by feed type. At least one other site dealing with propriety data has mentioned this as a problem. We have no plans to modify this in the LDM. > The conclusion for the moment is that we have to run a separate instance > of LDM on a separate server for every client, and feed each machine only > the data that client has contracted for. Each client will also be able > to choose which "fake" data type we will feed them various types of data > with, since there are no official type id's available. The need to > have a server per client is pains us from the aesthetic point of view, > and will be costly to support. We would much prefer to serve multiple > client with a single machine, as we do now, with an in-house package > built over FTP. > I can see how this would an unsatisfying solution. The problem is that you're trying to use the LDM in a way in which it wasn't intended. Perhaps a more general design would have better years ago, but, here we are... (Although the LDM has worked very well for the purpose for which it was intended.) > So the questions are as follows: > -- Is NLDM being seriously considered for the next version of LDM. > Is their a time table for release ? > -- Will NLDM address the issues above, so that we can > serve commercial clients with a single LDM instance ? > Yes, NLDM is being seriously considered. I am building a prototype network now for testing purposes. Our plan is to make a decision as to which route to pursue sometime this spring. Turns out that NNTP and INN (the open source news server on which NLDM is based) offer many features that would be useful for us that LDM doesn't have. One big downside would be making the change within our community from the LDM protocol to NNTP. Also, INN is a much more complex package than LDM and thus harder to configure. But, we'll be considering all aspects of each approach when the time comes. Would NLDM solve your problem of wanting to relay different sets of data to different clients? I suspect it would because you can specify newsgroups much more finely. Also, with INN , you can specify newsgroups negatively. Thus your server could have a subscription list for a client that looks something like this: unidata.binaries.radar.level3.*,!unidata.binaries.radar.level3.KABC which would cause the server to relay all level 3 radar products to the client except those from KABC. At this point NLDM is simply INN with a statistics package built on top. (Although I did have to develop an encoding that is acceptable to NNTP.) That is, I have used NLDM "out of the box". At the moment I know of two minor modifications that I would like to make to INN to better suit our purposes. But, the point of this is to say that you could try relaying with INN right now. Just get the INN distribution from ISC, encode your data, and you're ready to go. Although INN is complex to configure, I would be happy to share what I've learned with you. Also, you're welcome to try our encoding algorithm. > I hope you can solve all our problems :-) > ... or, perhaps complicate the picture even further... > Thanks for your time. > I hope this helps. Please let me know if I can be of further assistance. I'm excited about this project and am happy to share what I've learned. Sincerely, Anne -- *************************************************** Anne Wilson UCAR Unidata Program address@hidden P.O. Box 3000 Boulder, CO 80307 ---------------------------------------------------- Unidata WWW server http://www.unidata.ucar.edu/ ****************************************************