[LDM #JCB-153899]: idd.cise-nsf.gov to *.fis.ua.pt failure

Hi Yoshihiro,

I apologize for not being able to get back to you sooner on your problem
(we are holding training workshops at the moment).

re: 
> Yes, the ldm is running on the atm77.fis.ua.pt but I am
> not receiving any data at all.

I just checked and see that atm77.fis.ua.pt is contacting idd.cise-nsf.gov
to request CONDUIT data:

Oct 23 18:34:08 atm.cise-nsf.gov atm77.fis.ua.pt(feed)[10827] NOTE: Starting 
Up(6.4.5/6): 20061023173404.789 TS_ENDT {{CONDUIT,  "prod/gfs.*pgrb2.*[16]$"}}, 
Primary
Oct 23 18:34:08 atm.cise-nsf.gov atm77.fis.ua.pt(feed)[10827] NOTE: topo:  
atm77.fis.ua.pt {{CONDUIT, (.*)}}
Oct 23 18:34:12 atm.cise-nsf.gov atm77.fis.ua.pt(feed)[10861] NOTE: Starting 
Up(6.4.5/6): 20061023173411.629 TS_ENDT {{CONDUIT,  "prod/gfs.*pgrb2.*[27]$"}}, 
Primary
Oct 23 18:34:12 atm.cise-nsf.gov atm77.fis.ua.pt(feed)[10861] NOTE: topo:  
atm77.fis.ua.pt {{CONDUIT, (.*)}}
Oct 23 18:34:13 atm.cise-nsf.gov atm77.fis.ua.pt(feed)[10862] NOTE: Starting 
Up(6.4.5/6): 20061023173411.678 TS_ENDT {{CONDUIT,  "prod/gfs.*pgrb2.*[38]$"}}, 
Primary
Oct 23 18:34:13 atm.cise-nsf.gov atm77.fis.ua.pt(feed)[10862] NOTE: topo:  
atm77.fis.ua.pt {{CONDUIT, (.*)}}
Oct 23 18:34:13 atm.cise-nsf.gov atm77.fis.ua.pt(feed)[10863] NOTE: Starting 
Up(6.4.5/6): 20061023173411.720 TS_ENDT {{CONDUIT,  "prod/gfs.*pgrb2.*[49]$"}}, 
Primary
Oct 23 18:34:13 atm.cise-nsf.gov atm77.fis.ua.pt(feed)[10863] NOTE: topo:  
atm77.fis.ua.pt {{CONDUIT, (.*)}}

Even though it looks like the data should flow to atm77, it is not.

Interesting observations:

1) An 'ldmping' to from idd.cise-nsf.gov to atm77 does not work:

ldmping atm77.fis.ua.pt
Oct 23 18:53:56 INFO:      State    Elapsed Port   Remote_Host           
rpc_stat
Oct 23 18:53:56 INFO: Resolving atm77.fis.ua.pt to 193.137.81.77 took 0.04078 
seconds
Oct 23 18:54:06 ERROR: SVC_UNAVAIL  10.039241    0   atm77.fis.ua.pt  
h_clnt_create(atm77.fis.ua.pt): Timed out while creating connection

2) a 'traceroute' from idd.cise-nsf.gov to atm77 seems to hang in geant2.net:

traceroute to atm77.fis.ua.pt (193.137.81.77), 30 hops max, 40 byte packets
 1  cise-7204-1 (192.12.209.1)  1 ms  0 ms  0 ms
 2  arlg-nsf.maxgigapop.net (206.196.177.137)  3 ms  5 ms  2 ms
 3  abilene-rtr.maxgigapop.net (206.196.177.2)  11 ms  17 ms  6 ms
 4  abilene-wash.rt1.fra.de.geant2.net (62.40.125.17)  141 ms  110 ms  108 ms
 5  so-6-2-0.rt1.gen.ch.geant2.net (62.40.112.21)  118 ms  116 ms  116 ms
 6  so-7-0-0.rt1.mad.es.geant2.net (62.40.112.26)  145 ms  140 ms  140 ms
 7  so-6-0-0.rt1.lis.pt.geant2.net (62.40.112.98)  180 ms  159 ms  189 ms

3) a 'traceroute' from idd.unidata.ucar.edu to atm77 works:

traceroute atm77.fis.ua.pt
traceroute to atm77.fis.ua.pt (193.137.81.77), 30 hops max, 40 byte packets
 1  flra-n140.unidata.ucar.edu (128.117.140.252)  1.173 ms   0.233 ms   0.264 ms
 2  gin-n243-80.ucar.edu (128.117.243.81)  0.472 ms   0.409 ms   0.415 ms
 3  frgp-gw-1.ucar.edu (128.116.254.250)  1.396 ms   1.427 ms   1.444 ms
 4  192.43.217.166 (192.43.217.166)  1.524 ms   1.566 ms   1.616 ms
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  198.32.11.51 (198.32.11.51)  45.516 ms   45.398 ms   45.282 ms
10  so-7-0-0.rt1.ams.nl.geant2.net (62.40.112.133)  145.608 ms   145.675 ms   
145.668 ms
11  so-1-0-0.rt1.lon.uk.geant2.net (62.40.112.138)  153.794 ms   153.787 ms   
153.720 ms
12  so-5-0-0.rt1.lis.pt.geant2.net (62.40.112.145)  180.718 ms   180.762 ms   
180.772 ms
13  * * *
14  ROUTER3.GE.Lisboa.fccn.pt (193.137.0.27)  180.719 ms   180.707 ms   180.714 
ms
15  Router2.10GE.Porto.fccn.pt (193.136.1.222)  184.505 ms   184.481 ms   
184.253 ms
16  UA.Aveiro.fccn.pt (193.136.1.194)  185.663 ms   185.642 ms   185.512 ms
17  fw1-ext.core.ua.pt (193.137.173.253)  185.416 ms   185.548 ms   185.369 ms
18  gt-cicua.core.ua.pt (193.136.86.193)  185.914 ms   186.067 ms   188.092 ms
19  * * *
20  atm77.fis.ua.pt (193.137.81.77)  185.620 ms   185.794 ms   185.762 ms

4) an 'ldmping' from idd.unidata.ucar.edu to atm77 does not work:

 ldmping atm77.fis.ua.pt
Oct 23 19:06:00 INFO:      State    Elapsed Port   Remote_Host           
rpc_stat
Oct 23 19:06:00 INFO: Resolving atm77.fis.ua.pt to 193.137.81.77 took 0.005775 
seconds
Oct 23 19:06:02 ERROR:  ADDRESSED   1.463389    0   atm77.fis.ua.pt  RPC: 
Unable to receive; errno = Connection reset by peer

5) a 'notifyme' from idd.unidata.ucar.edu to atm77 returns with an Access 
denied:

notifyme -vxl- -f ANY -o 10000 -h atm77.fis.ua.pt
Oct 23 19:07:19 notifyme[10336] NOTE: Starting Up: atm77.fis.ua.pt: 
20061023162039.995 TS_ENDT {{ANY,  ".*"}}
Oct 23 19:07:19 notifyme[10336] NOTE: LDM-5 desired product-class: 
20061023162039.995 TS_ENDT {{ANY,  ".*"}}
Oct 23 19:07:19 notifyme[10336] INFO: Resolving atm77.fis.ua.pt to 
193.137.81.77 took 0.002277 seconds
Oct 23 19:07:20 notifyme[10336] ERROR: NOTIFYME(atm77.fis.ua.pt): 7: Access 
denied by remote server
Oct 23 19:07:45 notifyme[10336] NOTE: LDM-5 desired product-class: 
20061023162039.995 TS_ENDT {{ANY,  ".*"}}
Oct 23 19:07:45 notifyme[10336] INFO: Resolving atm77.fis.ua.pt to 
193.137.81.77 took 0.00205 seconds
Oct 23 19:07:45 notifyme[10336] ERROR: NOTIFYME(atm77.fis.ua.pt): 7: Access 
denied by remote server

6) the ~ldm/logs/ldmd.log messages on idd.unidata.ucar.edu show feed requests 
from atm77:
Oct 23 16:33:44 uni1 atm77.fis.ua.pt(feed)[3769] NOTE: Starting Up(6.4.5/6): 
20061023153343.078 TS_ENDT {{CONDUIT,  "MT.gfs_CY.(00|06|12|18).*[05]$"}}, 
Alternate
Oct 23 16:33:44 uni1 atm77.fis.ua.pt(feed)[3769] NOTE: topo:  atm77.fis.ua.pt 
{{CONDUIT, (.*)}}
Oct 23 16:33:44 uni1 atm77.fis.ua.pt(feed)[3770] NOTE: Starting Up(6.4.5/6): 
20061023153343.080 TS_ENDT {{CONDUIT,  "MT.gfs_CY.(00|06|12|18).*[16]$"}}, 
Alternate
Oct 23 16:33:44 uni1 atm77.fis.ua.pt(feed)[3770] NOTE: topo:  atm77.fis.ua.pt 
{{CONDUIT, (.*)}}
Oct 23 16:33:44 uni1 atm77.fis.ua.pt(feed)[3771] NOTE: Starting Up(6.4.5/6): 
20061023153343.083 TS_ENDT {{CONDUIT,  "MT.gfs_CY.(00|06|12|18).*[49]$"}}, 
Alternate
Oct 23 16:33:44 uni1 atm77.fis.ua.pt(feed)[3771] NOTE: topo:  atm77.fis.ua.pt 
{{CONDUIT, (.*)}}
Oct 23 16:33:44 uni1 atm77.fis.ua.pt(feed)[3772] NOTE: Starting Up(6.4.5/6): 
20061023153343.085 TS_ENDT {{CONDUIT,  "MT.gfs_CY.(00|06|12|18).*[27]$"}}, 
Alternate
Oct 23 16:33:44 uni1 atm77.fis.ua.pt(feed)[3772] NOTE: topo:  atm77.fis.ua.pt 
{{CONDUIT, (.*)}}
Oct 23 16:33:44 uni1 atm77.fis.ua.pt(feed)[3773] NOTE: Starting Up(6.4.5/6): 
20061023153343.087 TS_ENDT {{CONDUIT,  "MT.gfs_CY.(00|06|12|18).*[38]$"}}, 
Alternate
Oct 23 16:33:44 uni1 atm77.fis.ua.pt(feed)[3773] NOTE: topo:  atm77.fis.ua.pt 
{{CONDUIT, (.*)}}
Oct 23 16:34:42 uni1 atm77.fis.ua.pt(feed)[21534] ERROR: Couldn't flush 
connection; nullproc_6() failure to atm77.fis.ua.pt: RPC: Timed out
Oct 23 16:34:42 uni1 atm77.fis.ua.pt(feed)[21533] ERROR: Couldn't flush 
connection; nullproc_6() failure to atm77.fis.ua.pt: RPC: Timed out
Oct 23 16:34:42 uni1 atm77.fis.ua.pt(feed)[21532] ERROR: Couldn't flush 
connection; nullproc_6() failure to atm77.fis.ua.pt: RPC: Timed out


> That�s what I am not
> undestanding because according with computer�s department
> they did no change at all.

It appears that something changed either in a firewall somewhere or some 
network routing
configuration.

> Is there any way to check all my conection with ldm ??
> best regards,

In addition to the checks above, I tried using 'rpcinfo' to interrogate your
machine:

/usr/sbin/rpcinfo -p atm77.fis.ua.pt

This got no response.

Questions:

- what is the operating systme on atm77 (please return results of 'uname -a')
- was the operating system on atm77 upgraded recently
- do you have an 'allow' line in your ~ldm/etc/ldmd.conf file for all machines
  from the unidata.ucar.edu domain?  It should look like:

# ALLOW anything to your own machine and all Unidata machines
allow   ANY
        
^((localhost|loopback)|(127\.0\.0\.1\.?$)|([a-z].*\.unidata\.ucar\.edu\.?$)

- when you did your LDM installation, did you remember to change the mode/owner 
of
  ~ldm/bin/rpc.ldmd and ~ldm/bin/hupsyslog using the command:

  <as 'ldm'>
  cd ~ldm
  cd ldm-6.4.6.4/src
  make distclean
  ./configure
  make
  make install
  sudo make install_setuids         <- the line I am referring to
  cd ~
  rm runtime
  ln -s ldm-6.4.6.4 runtime

I am working with our system administrator to try and figure out why data 
products
are not flowing to you even though both idd.unidata.ucar.edu and 
idd.cise-nsf.gov
are receiving AND honoring feed requests.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: JCB-153899
Department: Support LDM
Priority: Normal
Status: Open