[CONDUIT #RAI-834865]: Very high latency...help?

Hola Luis,

re:
> We're running the LDM on one machine to get CONDUIT data.
> From ldmd.conf,
> 
> REQUEST CONDUIT "(prod/gfs.*pgrb2.*[05]$)" idd.cise-nsf.gov     PRIMARY
> REQUEST CONDUIT "(prod/gfs.*pgrb2.*[16]$)" idd.cise-nsf.gov     PRIMARY
> REQUEST CONDUIT "(prod/gfs.*pgrb2.*[27]$)" idd.cise-nsf.gov     PRIMARY
> REQUEST CONDUIT "(prod/gfs.*pgrb2.*[38]$)" idd.cise-nsf.gov     PRIMARY
> REQUEST CONDUIT "(prod/gfs.*pgrb2.*[49]$)" idd.cise-nsf.gov     PRIMARY
> #
> REQUEST CONDUIT "(prod/gfs.*pgrb2.*[05]$)" idd.unidata.ucar.edu PRIMARY
> REQUEST CONDUIT "(prod/gfs.*pgrb2.*[16]$)" idd.unidata.ucar.edu PRIMARY
> REQUEST CONDUIT "(prod/gfs.*pgrb2.*[27]$)" idd.unidata.ucar.edu PRIMARY
> REQUEST CONDUIT "(prod/gfs.*pgrb2.*[38]$)" idd.unidata.ucar.edu PRIMARY
> REQUEST CONDUIT "(prod/gfs.*pgrb2.*[49]$)" idd.unidata.ucar.edu PRIMARY

These request lines look correct.

> The problem is that, consulting
> http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+atm77.fis.ua.pt,
> the latency is very high.

I agree, the latencies are very high.  The latency for the much lower
volume data feed IDS|DDPLUS is also very high.  This tells me that the
likely cause of the high latencies is the bandwidth into your site, the
Universidade de Aveiro.

> Can you please give me some hints on how to "speed-up" data reception?

If limited bandwidth is really your problem, then the only help would
be to reduce your request for data.

Questions:

- are you using the data from the HRS feed?  It looks like you are
  requesting all of it.

- are you using the satellite imagery from the UNIWISC feed?  It
  looks like you are requesting all of it as well.

The volume plots for the UNIWISC and HRS feeds for your machine,
atm77.fis.ua.pt, indicate that you are requesting all of the data in
those feeds. If your latency problem really is caused by limited
bandwidth, it would be a _very_ good idea to remove requests for
any/all data that you absolutely do not need.  I think that the
HRS and UNIWISC feeds are the first ones to look at to conserve
bandwidth:

UNIWISC - satellite image sectors from GOES-East and GOES-West
HRS     - lower resolution model data from NOAAPort.  Most of
          the model fields in the HRS feed are U.S.-centric

Comment:

- The latencies for feeds to atm77 used to be low.  The fact that they
  are now high suggests that something has been changed to limit
  bandwidth to your site.  Since the latencies for all feeds you are
  receiving are consistently bad -- they approach 3000 seconds when
  the CONDUIT data is being received -- I would assert that the
  problem is not at either of the two upstream feed sites (idd.cise-nsf.gov
  and idd.unidata.ucar.edu) or in the trans-Atlantic link from I2 to
  GEANT2.  If I am correct in this assertion, the bottleneck would
  be close to your machine... something like a departmental or university
  packet shaper that is acting on all of your feeds.

Suggestion:

- Contact your departmental and university network groups to see if
  there has been some bandwidth limiting placed on the feeds to atm77.
  If there has, you will have the job of convincing the group responsible
  that your use of bandwidth is justified.  After all, you are transferring
  data not music or movies.

> Many Thanks in Advance,

No worries.  Please let us know if we can help further!

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: RAI-834865
Department: Support CONDUIT
Priority: Normal
Status: Closed