20030707: LDM RPC timeouts while still getting data

>From: "Alan Hall" <address@hidden>
>Organization: NOAA/NCDC
>Keywords: 200307071833.h67IXQLd027774

Alan,

>See sample ldmd.log below.  The remote site is getting data, but what could
>be causing the Port mapper failures? 

The situtation presented is _very_ strange.  It takes a very unusual
set of circumstances to continue to get data but not be able to connect
up.  Here is one scenario that would make this happen:

- tradewinds connects to your LDM server and starts getting data
- extra security (e.g., firewall) is put in place on your side so
  that now the downstream LDM would not be able to make a new connection.
  This happens while the downstream LDM request rpc.ldmd stays running
  it doesn't disconnect
- no data is received by the downstream LDM for 30 seconds forcing it
  to ask the upstream if it is still alive.  Since this asking is through
  a new connection, it fails because of the newly added security.  This
  would have to happen _without_ the original connection dying/exiting/
  being killed

The liklihood of the above is small, but possible.  Our question is
if tradewinds is getting data from your LDM server, or if it is
getting data from a different server and someone is mistaking in
thinking that the data is coming from your LDM?

Tom

>Alan. 
>
>-------- Original Message --------
>Subject: RE: [Fwd: Help-Close:ahall:Route to ERC]
>Date: Mon, 7 Jul 2003 13:09:29 -0500
>From: "Chad Hill" <address@hidden>
>To: "'Alan Hall'" <address@hidden>
>
>Hi Alan,
>
>I *AM* getting data.. but I am still seeing this in my logs...
>
>Jul 07 18:02:46 tradewinds 205.167.25.163[15159]: Couldn't connect to
>upstream L
>DM-6 using portmapper: RPC: Port mapper failure - RPC: Timed out
>Jul 07 18:02:46 tradewinds 205.167.25.163[15159]: Couldn't connect to
>upstream L
>DM-5 using port 388: RPC: Remote system error - Connection refused
>Jul 07 18:03:46 tradewinds 205.167.25.163[15159]: Couldn't connect to
>upstream L
>DM-5 using portmapper: RPC: Port mapper failure - RPC: Timed out
>Jul 07 18:04:16 tradewinds 205.167.25.163[15159]: Desired product class:
>2003063
>0202644.536 TS_ENDT {{CRAFT,  ".*"}}
>Jul 07 18:04:16 tradewinds 205.167.25.163[15159]: Couldn't connect to
>upstream L
>DM-6 using port 388: RPC: Remote system error - Connection refused
>Jul 07 18:04:22 tradewinds 192.67.134.137[15160]: Couldn't connect to
>upstream L
>DM-5 using port 388: RPC: Remote system error - Connection timed out
>Jul 07 18:05:16 tradewinds 205.167.25.163[15159]: Couldn't connect to
>upstream L
>DM-6 using portmapper: RPC: Port mapper failure - RPC: Timed out
>Jul 07 18:05:16 tradewinds 205.167.25.163[15159]: Couldn't connect to
>upstream L
>DM-5 using port 388: RPC: Remote system error - Connection refused
>Jul 07 18:05:22 tradewinds 192.67.134.137[15160]: Couldn't connect to
>upstream L
>DM-5 using portmapper: RPC: Port mapper failure - RPC: Timed out
>
>Is this because I can't see you? When I do an ldmping for you, it
>doesn't work.... I get this:
>Jul 07 18:08:04      State    Elapsed Port   Remote_Host
>rpc_stat
>Jul 07 18:08:16      NAMED  12.051692    0   205.167.25.163  can't
>contact portmapper: RPC: Timed out
>
>Do I need to add your LDM5 & 6 to my hosts files perhaps?
>
>-Chad
>
>-----Original Message-----
>From: Alan Hall [mailto:address@hidden] 
>Sent: Monday, July 07, 2003 11:37 AM
>To: Chadwick Hill
>Subject: [Fwd: Help-Close:ahall:Route to ERC]
>
>Ok, it looks like we have the route setup.  Let's start kind of simple
>with the
>Level II data.  I'll open up one of our servers and see if I can ping
>and
>ldmping and then see if the two connect up.  
>
>Alan. 
>
>-------- Original Message --------
>Subject: Help-Close:ahall:Route to ERC
>Date: Mon, 7 Jul 2003 11:43:59 -0400
>From: IT Branch <address@hidden>
>Reply-To: address@hidden,
>address@hidden,address@hidden
>To: address@hidden, address@hidden
>
>User: ahall
>Date submitted: Mon Jul  7 11:18:06 2003
>Priority: ASAP
>Number: 20.31079
>Section: Network Communication (20)
>When did it start: 20030707
>Computer sent from: ahall.ncdc.noaa.gov
>
>Description:
>Please set up a route such that 205.167.25.163 can communicate with
>65.165.189.35
>via the fibre connection with the ERC.  Currently, the routes take the
>long way
>around with 22 hops out thru the Internet and back in to the ERC.  Mark
>Smith
>has approved this.  
>Last Updated: Mon Jul  7 11:43:59 2003
>Solution:
>In SSR16:
>ip add route 65.165.189.35 host gateway 205.167.25.230
>
>Done.
>
>[root@ldm1 root]# traceroute 65.165.189.35
>traceroute to 65.165.189.35 (65.165.189.35), 30 hops max, 38 byte
>packets
> 1  * * *
> 2   (205.167.25.65)  1.128 ms  1.062 ms  0.973 ms
> 3  205.167.25.230 (205.167.25.230)  1.436 ms  1.066 ms  1.620 ms
> 4  63.172.117.161 (63.172.117.161)  28.689 ms  28.334 ms  27.988 ms
> 5  65.165.189.35 (65.165.189.35)  27.919 ms  27.632 ms  27.688 ms
>
>John F.
>Computer sent from: jrf.ncdc.noaa.gov
>
>Estimated hours spent: .5
>Mon Jul  7 11:43:59 2003