[Support #FAC-765998]: [conduit] Slow connection/big lag from conduit.ncep.noaa.gov

Hi Pete,

re:
> After the CONDUIT outage this morning and NCEP resolving that problem, I
> figured I'd try again feeding CONDUIT data from conduit.ncep.noaa.gov.
> Data was coming in on only one of the 5 split feeds, and was incredibly
> lagged. I have since switched back to feeding CONDUIT data exclusively
> from idd.unidata.ucar.edu.

We would you rather redundantly feed from both top level NCEP sites
(conduit.ncep.noaa.gov and ncepldm4.woc.noaa.gov) and add an additional
redundant feed if/when needed.

re:
> I would really like to get this figured out, since we have historically
> been a top level CONDUIT relay, and should be feeding from the source.

Yes, absolutely.

re:
> I'm happy to contact our NOC but I don't really know what to tell them.
> I doubt they know what the ldm/idd is, or what I mean when I tell them I
> am losing data because of a large lag. Ideas?

I don't think that describing the LDM to your network folks should be
needed for the same reason you noted, they likely know nothing about
the LDM.  You could, however, describe how data requests have been
split into mutually exclusive subsets in order to minimize latency,
but that you are experiencing high latencies on connections to
particular upstream feeders (but not others?).

re:
> Here are the relevant CONDUIT messages from my ldmd.log file when I
> attempted to feed from conduit.ncep.noaa.gov a few minutes ago:
> 
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28040] NOTE: Starting Up(6.12.6): 
> conduit.ncep.noaa.gov:388 20150414151243.722 TS_ENDT {{CONDUIT,  "[09]$"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28040] NOTE: Previous 
> product-information file ".b31dc8a364fc8b9081b9a46ce7060d8c.info" doesn't 
> exist
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28042] NOTE: Starting Up(6.12.6): 
> conduit.ncep.noaa.gov:388 20150414151243.722 TS_ENDT {{CONDUIT,  "[27]$"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28041] NOTE: Starting Up(6.12.6): 
> conduit.ncep.noaa.gov:388 20150414151243.722 TS_ENDT {{CONDUIT,  "[18]$"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28042] NOTE: Previous 
> product-information file ".1c3ed1861ae23a46b586e6aa36ef0f72.info" doesn't 
> exist
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28041] NOTE: Previous 
> product-information file ".98e61731d1cdf5f9c85a2fccba71279e.info" doesn't 
> exist
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28043] NOTE: Starting Up(6.12.6): 
> conduit.ncep.noaa.gov:388 20150414151243.722 TS_ENDT {{CONDUIT,  "[36]$"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28043] NOTE: Previous 
> product-information file ".a2cc2a78b26e4be6935df5ea685e28bb.info" doesn't 
> exist
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28044] NOTE: Starting Up(6.12.6): 
> conduit.ncep.noaa.gov:388 20150414151243.723 TS_ENDT {{CONDUIT,  "[45]$"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28044] NOTE: Previous 
> product-information file ".39e8c258d23ee9fdbba1273f17ca1ffb.info" doesn't 
> exist
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28043] NOTE: LDM-6 desired 
> product-class: 20150414151243.783 TS_ENDT {{CONDUIT, "[36]$"},{NONE, 
> "SIG=fb1008d746949f95a20dc4fc99d728bc"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28040] NOTE: LDM-6 desired 
> product-class: 20150414151243.870 TS_ENDT {{CONDUIT, "[09]$"},{NONE, 
> "SIG=0decb8a9a968940cd8646cb1ebdadc28"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28041] NOTE: LDM-6 desired 
> product-class: 20150414151243.879 TS_ENDT {{CONDUIT, "[18]$"},{NONE, 
> "SIG=34fecfd9f22ff6867ac9af438e520a14"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28042] NOTE: LDM-6 desired 
> product-class: 20150414151243.909 TS_ENDT {{CONDUIT, "[27]$"},{NONE, 
> "SIG=f90c8859cfae8a213a82e9bf0b07735e"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28044] NOTE: LDM-6 desired 
> product-class: 20150414151243.924 TS_ENDT {{CONDUIT, "[45]$"},{NONE, 
> "SIG=d2655891d671ee5d35c3bba51dd6934f"}}
> Apr 14 11:12:44 idd conduit.ncep.noaa.gov[28041] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:12:44 idd conduit.ncep.noaa.gov[28040] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:12:45 idd conduit.ncep.noaa.gov[28042] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:13:14 idd conduit.ncep.noaa.gov[28043] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:13:23 idd conduit.ncep.noaa.gov[28042] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:13:23 idd conduit.ncep.noaa.gov[28042] NOTE: one_svc_run(): RPC 
> layer closed connection
> Apr 14 11:13:23 idd conduit.ncep.noaa.gov[28042] NOTE: Connection to upstream 
> LDM closed: pid=1585
> Apr 14 11:13:28 idd conduit.ncep.noaa.gov[28041] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:13:28 idd conduit.ncep.noaa.gov[28041] NOTE: one_svc_run(): RPC 
> layer closed connection
> Apr 14 11:13:28 idd conduit.ncep.noaa.gov[28041] NOTE: Connection to upstream 
> LDM closed: pid=1584
> Apr 14 11:13:58 idd conduit.ncep.noaa.gov[28043] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:13:58 idd conduit.ncep.noaa.gov[28043] NOTE: one_svc_run() RPC 
> layer closed connection
> Apr 14 11:13:58 idd conduit.ncep.noaa.gov[28043] NOTE: Connection to upstream 
> LDM closed: pid=1709
> Apr 14 11:14:09 idd conduit.ncep.noaa.gov[28040] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:14:09 idd conduit.ncep.noaa.gov[28040] NOTE: one_svc_run(): RPC 
> layer closed connection
> Apr 14 11:14:09 idd conduit.ncep.noaa.gov[28040] NOTE: Connection to upstream 
> LDM closed: pid=1583
> Apr 14 11:14:23 idd conduit.ncep.noaa.gov[28042] NOTE: LDM-6 desired 
> product-class: 20150414151423.465 TS_ENDT {{CONDUIT, "[27]$"},{NONE, 
> "SIG=00afb67028a0bf027b2c29a917aaa683"}}
> Apr 14 11:14:23 idd conduit.ncep.noaa.gov[28042] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:14:28 idd conduit.ncep.noaa.gov[28041] NOTE: LDM-6 desired 
> product-class: 20150414151428.623 TS_ENDT {{CONDUIT, "[18]$"},{NONE, 
> "SIG=34fecfd9f22ff6867ac9af438e520a14"}}
> Apr 14 11:14:37 idd conduit.ncep.noaa.gov[28041] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:14:49 idd conduit.ncep.noaa.gov[28044] WARN: Couldn't connect to 
> LDM on conduit.ncep.noaa.gov using either port 388 or portmapper; : RPC: 
> Remote system error - Connection timed out
> Apr 14 11:14:58 idd conduit.ncep.noaa.gov[28043] NOTE: LDM-6 desired 
> product-class: 20150414151458.687 TS_ENDT {{CONDUIT, "[36]$"},{NONE, 
> "SIG=fb1008d746949f95a20dc4fc99d728bc"}}
> Apr 14 11:15:00 idd conduit.ncep.noaa.gov[28043] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:15:09 idd conduit.ncep.noaa.gov[28040] NOTE: LDM-6 desired 
> product-class: 20150414151509.836 TS_ENDT {{CONDUIT, "[09]$"},{NONE, 
> "SIG=08b460987109548b192f28a501953bd0"}}
> Apr 14 11:15:11 idd conduit.ncep.noaa.gov[28040] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:15:36 idd conduit.ncep.noaa.gov[28042] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:15:36 idd conduit.ncep.noaa.gov[28042] NOTE: one_svc_run(): RPC 
> layer closed connection
> Apr 14 11:15:36 idd conduit.ncep.noaa.gov[28042] NOTE: Connection to upstream 
> LDM closed: pid=2197
> Apr 14 11:15:49 idd conduit.ncep.noaa.gov[28044] NOTE: LDM-6 desired 
> product-class: 20150414151549.925 TS_ENDT {{CONDUIT, "[45]$"},{NONE, 
> "SIG=d2655891d671ee5d35c3bba51dd6934f"}}
> Apr 14 11:15:51 idd conduit.ncep.noaa.gov[28043] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:15:51 idd conduit.ncep.noaa.gov[28043] NOTE: one_svc_run(): RPC 
> layer closed connection
> Apr 14 11:15:51 idd conduit.ncep.noaa.gov[28043] NOTE: Connection to upstream 
> LDM closed: pid=2215
> Apr 14 11:16:05 idd conduit.ncep.noaa.gov[28044] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:16:21 idd conduit.ncep.noaa.gov[28040] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:16:21 idd conduit.ncep.noaa.gov[28040] NOTE: one_svc_run(): RPC 
> layer closed connection
> Apr 14 11:16:21 idd conduit.ncep.noaa.gov[28040] NOTE: Connection to upstream 
> LDM closed: pid=2348
> Apr 14 11:16:23 idd conduit.ncep.noaa.gov[28040] NOTE: Exiting
> Apr 14 11:16:23 idd conduit.ncep.noaa.gov[28041] NOTE: Exiting
> Apr 14 11:16:23 idd conduit.ncep.noaa.gov[28043] NOTE: Exiting
> Apr 14 11:16:23 idd conduit.ncep.noaa.gov[28042] NOTE: Exiting
> Apr 14 11:16:23 idd conduit.ncep.noaa.gov[28044] NOTE: Exiting

The 'select() timeout on socket 3' messages are important since they show that 
your machine was not able
to contact the upstream machine (Steve E. can provide a more concise wording).

Question:

- is the above listing reasonably current WRT NCEP fixing their problem, or is
  it possible that they were still having problems when you grabbed this 
snippit?

  I ask because it is possible that NCEP fixed one problem and introduced 
another
  (e.g., their firewall no longer is allowing your feed REQUESTs to pass to the
  LDM cluster in question).  We have been through this kind of situation with 
NCEP
  before, so it is a possibility now.

re:
> Thanks for any help or ideas you can provide.

It may be useful to send your concerns to NCEP by directing an email to Carissa
Klemmer and the netflow data team:

Carissa Klemmer - NOAA Federal <address@hidden>
Dataflow Team <address@hidden>

If you decide to send a note to the above, please CC us and Becky Cosgrove:

Rebecca Cosgrove <address@hidden>

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: FAC-765998
Department: Support CONDUIT
Priority: Normal
Status: Closed