[Support #FAC-765998]: [conduit] Slow connection/big lag from conduit.ncep.noaa.gov

Hi Pete,

re:
> The past few days my connection for conduit data feeding from
> conduit.ncep.noaa.gov (to idd.aos.wisc.edu) has become slower, resulting
> in large lag times and lost data.

We have experienced increased latencies in the CONDUIT feeds from both
NCEP top level sites as well.  Our latencies are not nearly as bad as
yours, however.

re:
> The attached gif shows my lag over the past few days. There are no time
> stamps on the image, unfortunately, but the large peaks are from each 6
> hrly GFS/ensemble suite. The right side of the graph is now (14:30 UTC
> Tuesday, March 31) and the left side is ~00 UTC March 29.

Yup.  It is interesting to compare your latencies to those being
experienced by us and Penn State:

daffy.unidata.ucar.edu (the CONDUIT accumulator here in the UPC):
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+daffy.unidata.ucar.edu

idd.meteo.psu.edu
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.meteo.psu.edu

idd.aos.wisc.edu
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu

As you can see, your latencies are much worse than either PSU or us.  I
would interpret this to mean that some part of the problem is closer to
you than it is to NCEP (but I could be wrong).

What is _really_ interesting (and goes against the conjecture in the previous
sentence) is the latencies for the test CONDUIT feed that contains all of
the standard CONDUIT feed plus the 0.25 degree GFS data:

lead.unidata.ucar.edu
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+lead.unidata.ucar.edu

Even though the data volumes in the test CONDUIT feed are up to twice what
they are in the operational feed, the latencies are _much_ lower.  I believe
that the NCEP machine that is feeding lead.unidata.ucar.edu is located in
the same datacenter (and perhaps even on the same physical machine since it
looks like a VM) as the cluster that is relaying the operational feed.  Why
latencies from this single machine would be lower is a mystery especially
since we are only using a single REQUEST for the higher volume, test CONDUIT
feed!

re:
> During the open peaks on the left side of the graph, I was feeding from
> conduit.ncep.noaa.gov and ncepldm4.woc.noaa.gov.

Yes, we are too.

re:
> About half-way across the graph, where the peaks begin to be filled in
> and the green line starts, I added idd.unidata.ucar.edu to the mix. Lags
> were still large.

Given that our latencies are lower, this result suggests that there
is something going on closer to you.

re:
> Prior to the 00 UTC run last night (the first of the two lower blue
> peaks on the right hand side) I removed conduit.ncep.noaa.gov form my
> requests, so I am only requesting from idd.unidata.ucar.edu and
> ncepldm4.woc.noaa.gov.

This was not a good move mainly since we were told by Becky Cosgrove
at the User Committee meeting last week that the CONDUIT relay presence
in Boulder, ncepldm4.woc.noaa.gov, is not getting all of the data that
should be available on the top level back east, conduit.ncep.noaa.gov.

re:
> My perception from this sequence of events is that my connectivity to
> conduit.ncep.noaa.gov has degraded. This seems to have begun within the
> last week or so, I didn't have the large lags before.

I agree, but your adding a REQUEST to idd.unidata.ucar.edu suggests that
the problem is limited to connections to conduit.ncep.noaa.gov.

re:
> Any ideas?

Sending a note out to the address@hidden mail list was a
good first step as it alerted folks in the NCEP Data Flow team that
there is a problem.  I will be opening an inquiry to those same folks
mainly so that they will be made aware of how much better the test
CONDUIT feed to lead.unidata.ucar.edu is.

re:
> What are the root conduit servers that I should be feeding from? I
> currently have
> 
> conduit.ncep.noaa.gov
> and
> ncepldm4.woc.noaa.gov

These are correct.  Our CONDUIT accumulator, daffy.unidata.ucar.edu, is
REQUESTing from the same machines.

Question:

- are your CONDUIT feed REQUESTs split, or are you using a single
  REQUEST line to each upstream?

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: FAC-765998
Department: Support CONDUIT
Priority: Normal
Status: Open