Re: Fwd: Re: [conduit] Slow connection/big lag from conduit.ncep.noaa.gov

Hi Pete,

We will be performing some maintenance on the conduit servers next week which I 
believe will help increase performance. I'll continue to investigate however. 
Thanks for the update. 

---
Kyle Nevins
NCWCP_HELPDESK

On Tue, 14 Apr 2015 12:26:02 -0400, Pete Pokrandt <address@hidden> wrote:
> All,
> 
> After the CONDUIT outage this morning and NCEP resolving that problem, I 
> figured I'd try again feeding CONDUIT data from conduit.ncep.noaa.gov. 
> Data was coming in on only one of the 5 split feeds, and was incredibly 
> lagged. I have since switched back to feeding CONDUIT data exclusively 
> from idd.unidata.ucar.edu.
> 
> I would really like to get this figured out, since we have historically 
> been a top level CONDUIT relay, and should be feeding from the source. 
> I'm happy to contact our NOC but I don't really know what to tell them. 
> I doubt they know what the ldm/idd is, or what I mean when I tell them I 
> am losing data because of a large lag. Ideas?
> 
> Here are the relevant CONDUIT messages from my ldmd.log file when I 
> attempted to feed from conduit.ncep.noaa.gov a few minutes ago:
> 
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28040] NOTE: Starting 
> Up(6.12.6): conduit.ncep.noaa.gov:388 20150414151243.722 TS_ENDT 
> {{CONDUIT,  "[09]$"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28040] NOTE: Previous 
> product-information file ".b31dc8a364fc8b9081b9a46ce7060d8c.info" 
> doesn't exist
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28042] NOTE: Starting 
> Up(6.12.6): conduit.ncep.noaa.gov:388 20150414151243.722 TS_ENDT 
> {{CONDUIT,  "[27]$"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28041] NOTE: Starting 
> Up(6.12.6): conduit.ncep.noaa.gov:388 20150414151243.722 TS_ENDT 
> {{CONDUIT,  "[18]$"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28042] NOTE: Previous 
> product-information file ".1c3ed1861ae23a46b586e6aa36ef0f72.info" 
> doesn't exist
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28041] NOTE: Previous 
> product-information file ".98e61731d1cdf5f9c85a2fccba71279e.info" 
> doesn't exist
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28043] NOTE: Starting 
> Up(6.12.6): conduit.ncep.noaa.gov:388 20150414151243.722 TS_ENDT 
> {{CONDUIT,  "[36]$"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28043] NOTE: Previous 
> product-information file ".a2cc2a78b26e4be6935df5ea685e28bb.info" 
> doesn't exist
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28044] NOTE: Starting 
> Up(6.12.6): conduit.ncep.noaa.gov:388 20150414151243.723 TS_ENDT 
> {{CONDUIT,  "[45]$"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28044] NOTE: Previous 
> product-information file ".39e8c258d23ee9fdbba1273f17ca1ffb.info" 
> doesn't exist
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28043] NOTE: LDM-6 desired 
> product-class: 20150414151243.783 TS_ENDT {{CONDUIT, "[36]$"},{NONE,  
> "SIG=fb1008d746949f95a20dc4fc99d728bc"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28040] NOTE: LDM-6 desired 
> product-class: 20150414151243.870 TS_ENDT {{CONDUIT, "[09]$"},{NONE,  
> "SIG=0decb8a9a968940cd8646cb1ebdadc28"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28041] NOTE: LDM-6 desired 
> product-class: 20150414151243.879 TS_ENDT {{CONDUIT, "[18]$"},{NONE,  
> "SIG=34fecfd9f22ff6867ac9af438e520a14"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28042] NOTE: LDM-6 desired 
> product-class: 20150414151243.909 TS_ENDT {{CONDUIT, "[27]$"},{NONE,  
> "SIG=f90c8859cfae8a213a82e9bf0b07735e"}}
> Apr 14 11:12:43 idd conduit.ncep.noaa.gov[28044] NOTE: LDM-6 desired 
> product-class: 20150414151243.924 TS_ENDT {{CONDUIT, "[45]$"},{NONE,  
> "SIG=d2655891d671ee5d35c3bba51dd6934f"}}
> Apr 14 11:12:44 idd conduit.ncep.noaa.gov[28041] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:12:44 idd conduit.ncep.noaa.gov[28040] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:12:45 idd conduit.ncep.noaa.gov[28042] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:13:14 idd conduit.ncep.noaa.gov[28043] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:13:23 idd conduit.ncep.noaa.gov[28042] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:13:23 idd conduit.ncep.noaa.gov[28042] NOTE: one_svc_run(): 
> RPC layer closed connection
> Apr 14 11:13:23 idd conduit.ncep.noaa.gov[28042] NOTE: Connection to 
> upstream LDM closed: pid=1585
> Apr 14 11:13:28 idd conduit.ncep.noaa.gov[28041] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:13:28 idd conduit.ncep.noaa.gov[28041] NOTE: one_svc_run(): 
> RPC layer closed connection
> Apr 14 11:13:28 idd conduit.ncep.noaa.gov[28041] NOTE: Connection to 
> upstream LDM closed: pid=1584
> Apr 14 11:13:58 idd conduit.ncep.noaa.gov[28043] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:13:58 idd conduit.ncep.noaa.gov[28043] NOTE: one_svc_run(): 
> RPC layer closed connection
> Apr 14 11:13:58 idd conduit.ncep.noaa.gov[28043] NOTE: Connection to 
> upstream LDM closed: pid=1709
> Apr 14 11:14:09 idd conduit.ncep.noaa.gov[28040] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:14:09 idd conduit.ncep.noaa.gov[28040] NOTE: one_svc_run(): 
> RPC layer closed connection
> Apr 14 11:14:09 idd conduit.ncep.noaa.gov[28040] NOTE: Connection to 
> upstream LDM closed: pid=1583
> Apr 14 11:14:23 idd conduit.ncep.noaa.gov[28042] NOTE: LDM-6 desired 
> product-class: 20150414151423.465 TS_ENDT {{CONDUIT, "[27]$"},{NONE,  
> "SIG=00afb67028a0bf027b2c29a917aaa683"}}
> Apr 14 11:14:23 idd conduit.ncep.noaa.gov[28042] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:14:28 idd conduit.ncep.noaa.gov[28041] NOTE: LDM-6 desired 
> product-class: 20150414151428.623 TS_ENDT {{CONDUIT, "[18]$"},{NONE,  
> "SIG=34fecfd9f22ff6867ac9af438e520a14"}}
> Apr 14 11:14:37 idd conduit.ncep.noaa.gov[28041] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:14:49 idd conduit.ncep.noaa.gov[28044] WARN: Couldn't connect 
> to LDM on conduit.ncep.noaa.gov using either port 388 or portmapper; : 
> RPC: Remote system error - Connection timed out
> Apr 14 11:14:58 idd conduit.ncep.noaa.gov[28043] NOTE: LDM-6 desired 
> product-class: 20150414151458.687 TS_ENDT {{CONDUIT, "[36]$"},{NONE,  
> "SIG=fb1008d746949f95a20dc4fc99d728bc"}}
> Apr 14 11:15:00 idd conduit.ncep.noaa.gov[28043] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:15:09 idd conduit.ncep.noaa.gov[28040] NOTE: LDM-6 desired 
> product-class: 20150414151509.836 TS_ENDT {{CONDUIT, "[09]$"},{NONE,  
> "SIG=08b460987109548b192f28a501953bd0"}}
> Apr 14 11:15:11 idd conduit.ncep.noaa.gov[28040] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:15:36 idd conduit.ncep.noaa.gov[28042] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:15:36 idd conduit.ncep.noaa.gov[28042] NOTE: one_svc_run(): 
> RPC layer closed connection
> Apr 14 11:15:36 idd conduit.ncep.noaa.gov[28042] NOTE: Connection to 
> upstream LDM closed: pid=2197
> Apr 14 11:15:49 idd conduit.ncep.noaa.gov[28044] NOTE: LDM-6 desired 
> product-class: 20150414151549.925 TS_ENDT {{CONDUIT, "[45]$"},{NONE,  
> "SIG=d2655891d671ee5d35c3bba51dd6934f"}}
> Apr 14 11:15:51 idd conduit.ncep.noaa.gov[28043] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:15:51 idd conduit.ncep.noaa.gov[28043] NOTE: one_svc_run(): 
> RPC layer closed connection
> Apr 14 11:15:51 idd conduit.ncep.noaa.gov[28043] NOTE: Connection to 
> upstream LDM closed: pid=2215
> Apr 14 11:16:05 idd conduit.ncep.noaa.gov[28044] NOTE: Upstream LDM-6 on 
> conduit.ncep.noaa.gov is willing to be a primary feeder
> Apr 14 11:16:21 idd conduit.ncep.noaa.gov[28040] NOTE: [svc_tcp.c:333] 
> select() timeout on socket 3
> Apr 14 11:16:21 idd conduit.ncep.noaa.gov[28040] NOTE: one_svc_run(): 
> RPC layer closed connection
> Apr 14 11:16:21 idd conduit.ncep.noaa.gov[28040] NOTE: Connection to 
> upstream LDM closed: pid=2348
> Apr 14 11:16:23 idd conduit.ncep.noaa.gov[28040] NOTE: Exiting
> Apr 14 11:16:23 idd conduit.ncep.noaa.gov[28041] NOTE: Exiting
> Apr 14 11:16:23 idd conduit.ncep.noaa.gov[28043] NOTE: Exiting
> Apr 14 11:16:23 idd conduit.ncep.noaa.gov[28042] NOTE: Exiting
> Apr 14 11:16:23 idd conduit.ncep.noaa.gov[28044] NOTE: Exiting
> 
> Thanks for any help or ideas you can provide.
> 
> Pete
> 
> 
> 
> 
> 
> On 04/07/2015 11:14 PM, Pete Pokrandt wrote:
> > The slow connection between conduit.ncep.noaa.gov and idd.aos.wisc.edu 
> > is happening again, and I am getting short files, especially when the 
> > GFS and GFS ensemble products are coming through.
> >
> > http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+idd.aos.wisc.edu
> >  
> >
> >
> > For now, I have added idd.unidata.ucar.edu to my request lines so that 
> > I don't lose more data. Let me know what I can do to help diagnose the 
> > issue.
> >
> > Thanks,
> > Pete
> >
> >
> > On 4/3/2015 8:40 AM, address@hidden wrote:
> >> Sounds good. I'm glad its working again. Please let me know if it 
> >> happens again and I will review in real time.
> >>
> >> ---
> >> Kyle Nevins
> >> NCWCP_HELPDESK
> >>
> >> On Thu, 02 Apr 2015 16:50:12 -0400, Pete Pokrandt 
> >> <address@hidden> wrote:
> >>> I was alerted to it on Sunday evening (March 29) and apparently it had
> >>> been happening for a few days already.
> >>>
> >>> Oddly enough, I have switched back over to feeding from
> >>> conduit.ncep.noaa.gov and my latencies are back down under 60 
> >>> seconds or
> >>> so. I thought you did something, restarted the ldm on
> >>> conduit.ncep.noaa.gov or something, but it appears whatever was 
> >>> going on
> >>> has been resolved.
> >>>
> >>> I'll let you know if it happens again.
> >>>
> >>> Pete
> >>>
> >>>
> >>> On 04/02/2015 03:43 PM, address@hidden wrote:
> >>>> Pete,
> >>>>
> >>>> Thanks for the information, I am still looking into this, have not 
> >>>> yet found the root cause of the latency increase. You said it 
> >>>> started on Monday or before that?
> >>>>
> >>>> ---
> >>>> Kyle Nevins
> >>>> NCWCP_HELPDESK
> >>>>
> >>>> On Wed, 01 Apr 2015 16:28:22 -0400, Pete Pokrandt 
> >>>> <address@hidden> wrote:
> >>>>> Kyle,
> >>>>>
> >>>>> I had changed to feed conduit from idd.unidata.ucar.edu for a 
> >>>>> while, as
> >>>>> a result of the slow connection to conduit.ncep.noaa.gov. Just 
> >>>>> earlier
> >>>>> this afternoon I changed back. I currently have this in my 
> >>>>> ldmd.conf on
> >>>>> idd.aos.wisc.edu:
> >>>>>
> >>>>> REQUEST CONDUIT "[05]$" ncepldm4.woc.noaa.gov
> >>>>> REQUEST CONDUIT "[16]$" ncepldm4.woc.noaa.gov
> >>>>> REQUEST CONDUIT "[27]$" ncepldm4.woc.noaa.gov
> >>>>> REQUEST CONDUIT "[38]$" ncepldm4.woc.noaa.gov
> >>>>> REQUEST CONDUIT "[49]$" ncepldm4.woc.noaa.gov
> >>>>> REQUEST CONDUIT "[05]$" conduit.ncep.noaa.gov
> >>>>> REQUEST CONDUIT "[16]$" conduit.ncep.noaa.gov
> >>>>> REQUEST CONDUIT "[27]$" conduit.ncep.noaa.gov
> >>>>> REQUEST CONDUIT "[38]$" conduit.ncep.noaa.gov
> >>>>> REQUEST CONDUIT "[49]$" conduit.ncep.noaa.gov
> >>>>>
> >>>>> Pete
> >>>>>
> >>>>>
> >>>>>
> >>>>> On 04/01/2015 03:24 PM, address@hidden wrote:
> >>>>>> Hi Pete,
> >>>>>>
> >>>>>> So it looks like you are only going to one of our conduit servers 
> >>>>>> and I'm trying to determine if that is our load balancer or not. 
> >>>>>> Can you confirm that your LDM feed goes to conduit.ncep.noaa.gov 
> >>>>>> (140.90.101.42)? Thanks.
> >>>>>>
> >>>>>> ---
> >>>>>> Kyle Nevins
> >>>>>> NCWCP_HELPDESK
> >>>>>>
> >>>>>> On Tue, 31 Mar 2015 12:05:18 -0400, Pete Pokrandt 
> >>>>>> <address@hidden> wrote:
> >>>>>>> Per the below response, I am submitting a ticket to the NCWCP Help
> >>>>>>> desk.  See my description of the slow connection between
> >>>>>>> conduit.ncep.noaa.gov and idd.aos.wisc.edu below.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>> Pete
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> -------- Forwarded Message --------
> >>>>>>> Subject:     Re: [conduit] Slow connection/big lag from 
> >>>>>>> conduit.ncep.noaa.gov
> >>>>>>> Date:     Tue, 31 Mar 2015 10:56:11 -0400
> >>>>>>> From:     Justin Cooke - NOAA Federal <address@hidden>
> >>>>>>> To:     Pete Pokrandt <address@hidden>, NCWCP helpdesk - 
> >>>>>>> NOAA Service
> >>>>>>> Account <address@hidden>
> >>>>>>> CC:     address@hidden <address@hidden>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> Pete,
> >>>>>>>
> >>>>>>> Please submit a ticket to the NCWCP Helpdesk 
> >>>>>>> (address@hidden
> >>>>>>> <mailto:address@hidden>), someone on our system 
> >>>>>>> administration
> >>>>>>> team will look into why you're seeing degraded performance from
> >>>>>>> conduit.ncep.noaa.gov <http://conduit.ncep.noaa.gov>
> >>>>>>>
> >>>>>>> Justin Cooke
> >>>>>>> NCEP Central Operations
> >>>>>>>
> >>>>>>> On Tue, Mar 31, 2015 at 10:45 AM, Pete Pokrandt <address@hidden
> >>>>>>> <mailto:address@hidden>> wrote:
> >>>>>>>
> >>>>>>>        The past few days my connection for conduit data feeding 
> >>>>>>> from
> >>>>>>>        conduit.ncep.noaa.gov <http://conduit.ncep.noaa.gov> (to
> >>>>>>>        idd.aos.wisc.edu <http://idd.aos.wisc.edu>) has become 
> >>>>>>> slower,
> >>>>>>>        resulting in large lag times and lost data.
> >>>>>>>
> >>>>>>>        The attached gif shows my lag over the past few days. 
> >>>>>>> There are no
> >>>>>>>        time stamps on the image, unfortunately, but the large 
> >>>>>>> peaks are
> >>>>>>>        from each 6 hrly GFS/ensemble suite. The right side of 
> >>>>>>> the graph is
> >>>>>>>        now (14:30 UTC Tuesday, March 31) and the left side is 
> >>>>>>> ~00 UTC March 29.
> >>>>>>>
> >>>>>>>        During the open peaks on the left side of the graph, I 
> >>>>>>> was feeding
> >>>>>>>        from conduit.ncep.noaa.gov <http://conduit.ncep.noaa.gov> 
> >>>>>>> and
> >>>>>>>        ncepldm4.woc.noaa.gov <http://ncepldm4.woc.noaa.gov>.
> >>>>>>>
> >>>>>>>        About half-way across the graph, where the peaks begin to 
> >>>>>>> be filled
> >>>>>>>        in and the green line starts, I added idd.unidata.ucar.edu
> >>>>>>>        <http://idd.unidata.ucar.edu> to the mix. Lags were still 
> >>>>>>> large.
> >>>>>>>
> >>>>>>>        Prior to the 00 UTC run last night (the first of the two 
> >>>>>>> lower blue
> >>>>>>>        peaks on the right hand side) I removed 
> >>>>>>> conduit.ncep.noaa.gov
> >>>>>>>        <http://conduit.ncep.noaa.gov> form my requests, so I am 
> >>>>>>> only
> >>>>>>>        requesting from idd.unidata.ucar.edu 
> >>>>>>> <http://idd.unidata.ucar.edu>
> >>>>>>>        and ncepldm4.woc.noaa.gov <http://ncepldm4.woc.noaa.gov>.
> >>>>>>>
> >>>>>>>        My perception from this sequence of events is that my 
> >>>>>>> connectivity
> >>>>>>>        to conduit.ncep.noaa.gov <http://conduit.ncep.noaa.gov> has
> >>>>>>>        degraded. This seems to have begun within the last week 
> >>>>>>> or so, I
> >>>>>>>        didn't have the large lags before.
> >>>>>>>
> >>>>>>>        Any ideas?
> >>>>>>>
> >>>>>>>        What are the root conduit servers that I should be 
> >>>>>>> feeding from? I
> >>>>>>>        currently have
> >>>>>>>
> >>>>>>>        conduit.ncep.noaa.gov <http://conduit.ncep.noaa.gov>
> >>>>>>>        and
> >>>>>>>        ncepldm4.woc.noaa.gov <http://ncepldm4.woc.noaa.gov>
> >>>>>>>
> >>>>>>>        Thanks,
> >>>>>>>        Pete
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>        --
> >>>>>>>        Pete Pokrandt - Systems Programmer
> >>>>>>>        UW-Madison Dept of Atmospheric and Oceanic Sciences
> >>>>>>>        608-262-3086 <tel:608-262-3086> - address@hidden
> >>>>>>>        <mailto:address@hidden>
> >>>>>>>
> >>>>>>>
> >>>>>>>        _______________________________________________
> >>>>>>>        conduit mailing list
> >>>>>>>        address@hidden <mailto:address@hidden>
> >>>>>>>        For list information or to unsubscribe, visit:
> >>>>>>>        http://www.unidata.ucar.edu/mailing_lists/
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> -- 
> >>>>>>> Pete Pokrandt - Systems Programmer
> >>>>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
> >>>>>>> 608-262-3086  - address@hidden
> >>>>> -- 
> >>>>> Pete Pokrandt - Systems Programmer
> >>>>> UW-Madison Dept of Atmospheric and Oceanic Sciences
> >>>>> 608-262-3086  - address@hidden
> >>>
> >>> -- 
> >>> Pete Pokrandt - Systems Programmer
> >>> UW-Madison Dept of Atmospheric and Oceanic Sciences
> >>> 608-262-3086  - address@hidden
> >
> 
> 
> -- 
> Pete Pokrandt - Systems Programmer
> UW-Madison Dept of Atmospheric and Oceanic Sciences
> 608-262-3086  - address@hidden