[IDD #ELV-696863]: Access to your databases

Hi Margaret,

re:
> We have a research group in the Geological Sciences Department here at
> Brown that would like to download some of your data.

Presumably you are referring to the group that Baird Langenbrunner is
associated with.

re:
> Our department is behind a firewall which does not allow incoming
> traffic on port 338 unless the inside user initiates the request.

We informed Baird that he should request that the port 388 be opened
in the Brown firewall mainly because giving us access to his machine
would allow us to monitor his data reception.  The reality is that the
LDM transfers are initiated from the user's side, so opening port
388 for incoming traffic is not absolutely required.  Again, we would
prefer that it was open ** at least to machines in the unidata.ucar.edu
domain ** so that we can help troubleshoot data ingest problems.

re:
> I have the impression that your data automatically downloads to the
> user when new data is collected.

The design of our Internet Data Distribution system (IDD) is for products
that are inserted into an originating LDM's queue be sent to all of
the downstreams that have requested the data as soon as possible (within
a fraction of a second).  The downstream machine is the entity that registers
the request(s) for which data is desired, so it initiates the flow through
transfer process.  The connection is turned around after the initial
request is made, so the downstream waits for data to be sent and then
can kick off whatever processing it wants to do when the product(s)
are received.

re:
> Isn't there a possibility that the user instead of using the automatic
> download can use rsync to connect to your computers and download the
> data in which he is interested?

No, this is not possible for several reasons:

- we don't allow rsync types of connections

- the data that Baird has been asking about (NLDN lightning) is proprietary
  to Vaisala corporation.  The arrangement between Vaisala and SUNY
  Albany is that the Department of Atmospheric Science at SUNY Albany
  is allowed to send the NLDN lightning data to U.S. university participants
  in the IDD in a point-to-point LDM transfer.  No other transfer mechanism
  is allowed/provided for.

- the LDM/IDD is a well known facility for moving data in realtime:

  - it is used by the majority of U.S. universities with atmospheric science
    programs to receive realtime weather data

  - it is used by a variety of U.S. government agencies for collection and
    distribution of realtime weather-related data (like full volume scan
    radar data)

  - it is used around the world by educational, governmental and commercial
    entities for the collection and distribution of weather-related ata
    in realtime

  - port 388 use has been registered with IANA for years.  Virtually all
    Linux distributions contain the LDM in their list of well known
    services (/etc/services)

  - the LDM has been the number one advanced application user of Internet2
    bandwidth for a number of years.  Please see Table 7 in the Internet2
    network statistics site for additional information: 
    http://netflow.i2.edu/weekly/<yyyymmdd>

So, the bottom line is that the only way for Baird's group to get the
data they are interested in is via point-to-point LDM transfer from
SUNY Albany.

Please let us know if you have additional questions.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ELV-696863
Department: Support IDD
Priority: Normal
Status: Closed