[IDD #AQC-617871]: Local Data Manager Question: UCSD aeolus not pulling data

Hi Mary,

re:
> I hope this finds you well and enjoying summer!

Aside from the fire that was only 3 miles from my house and caused the
evacuation of LOTS of folks in the area, the summer has been very nice,
thanks.

re:
> I made the decision this spring to retire from UCSD - my last official day
> in the office was June 29th.

Wow, I didn't know this (or, at least, I forgot).  Congratuations!

re:
> I agreed to help with a few transitional details - ldm on aeolus being one
> of them.
> 
> This morning (Saturday, July 30th) I was checking aeolus and noticed the
> data files were not updating (last data came in on July 25th). Usually I
> have a few users who alert me when there is a problem but surprisingly I
> have not heard from them.

I just checked, and aeolus is not reporting real time LDM statistics either:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex

Just so you know, it used to report statistics back to us, so this is
a new situation.

re:
> ldm was running so I went through the usual routine (stop, clean, delete
> queue, make queue, start). The entire process went smoothly:
> 
> [ldm@aeolus ~]$ ldmadmin clean
> The LDM system is running!  Stop it first.
> [ldm@aeolus ~]$ ldmadmin stop
> Stopping the LDM server...
> Waiting for the LDM server to terminate...
> Waiting for the LDM server to terminate...
> Waiting for the LDM server to terminate...
> Waiting for the LDM server to terminate...
> Waiting for the LDM server to terminate...
> Waiting for the LDM server to terminate...
> Waiting for the LDM server to terminate...
> Waiting for the LDM server to terminate...
> Waiting for the LDM server to terminate...
> Waiting for the LDM server to terminate...
> [ldm@aeolus ~]$ ldmadmin clean
> [ldm@aeolus ~]$ ldmadmin delqueue
> [ldm@aeolus ~]$ ldmadmin mkqueue -f
> [ldm@aeolus ~]$ ldmadmin start
> The product-queue is OK.
> Checking pqact(1) configuration-file(s)...
> /home/ldm/etc/pqact.conf: syntactically correct
> Checking LDM configuration-file (/home/ldm/etc/ldmd.conf)...
> Starting the LDM server...
> 
> Some files are coming in - however some other files (metar files) are not.
> I'm looking at ldmd.log and there are errors which may be why I'm not
> seeing the data. Here is the tail of that file:
> 
> [ldm@aeolus ~/logs]$ pwd
> /home/ldm/logs
> [ldm@aeolus ~/logs]$ pwd
> /home/ldm/logs
> [ldm@aeolus ~/logs]$ tail ldmd.log
> Jul 31 12:34:42 aeolus idd.unidata.ucar.edu[4888] ERROR: Disconnecting due to 
> LDM failure; Couldn't connect to LDM on idd.unidata.ucar.edu using either 
> port 388 or portmapper; : RPC: Remote system error - Connection timed out
> Jul 31 12:34:44 aeolus striker.atmos.albany.edu[4881] ERROR: Disconnecting 
> due to LDM failure; Couldn't connect to LDM on striker.atmos.albany.edu using 
> either port 388 or portmapper; : RPC: Remote system error - Connection timed 
> out
> Jul 31 12:35:12 aeolus idd.unidata.ucar.edu[4886] NOTE: LDM-6 desired 
> product-class: 20160731183512.567 TS_ENDT {{NEXRAD3,  ".*"},{NONE, 
> "SIG=1f570c0e7121afa28e1144dd906a7c9c"}}
> Jul 31 12:35:12 aeolus idd.unidata.ucar.edu[4887] NOTE: LDM-6 desired 
> product-class: 20160731183512.596 TS_ENDT {{DIFAX|FNEXRAD|FSL2, ".*"},{NONE,  
> "SIG=3f703908ec96c4ea82bf6bcfccc62ab6"}}
> Jul 31 12:35:12 aeolus idd.unidata.ucar.edu[4882] NOTE: LDM-6 desired 
> product-class: 20160731183512.596 TS_ENDT {{UNIWISC|IDS|DDPLUS, ".*"},{NONE,  
> "SIG=6e12deb14dd4b2f9c16aab009f323fd6"}}
> Jul 31 12:35:12 aeolus idd.unidata.ucar.edu[4884] NOTE: LDM-6 desired 
> product-class: 20160731183512.621 TS_ENDT {{NGRID,  ".*"},{NONE, 
> "SIG=4cf26d9b88177de07b26e69a7ace7edb"}}
> Jul 31 12:35:12 aeolus idd.unidata.ucar.edu[4883] NOTE: LDM-6 desired 
> product-class: 20160731183512.628 TS_ENDT {{HDS,  ".*"},{NONE, 
> "SIG=ad9d82f0b476685b4d77b8cf227e6315"}}
> Jul 31 12:35:12 aeolus idd.unidata.ucar.edu[4885] NOTE: LDM-6 desired 
> product-class: 20160731183512.630 TS_ENDT {{NIMAGE,  ".*"},{NONE, 
> "SIG=fe5a6b12d0c872b34e78b83439c9f98e"}}
> Jul 31 12:35:12 aeolus idd.unidata.ucar.edu[4888] NOTE: LDM-6 desired 
> product-class: 20160731183512.631 TS_ENDT {{CONDUIT, 
> "(MT.gfs|MT.nam|prod/gfs)"},{NONE,  "SIG=9d596df024c4aa745eec3e6ea5e8738c"}}
> Jul 31 12:35:14 aeolus striker.atmos.albany.edu[4881] NOTE: LDM-6 desired 
> product-class: 20160731183514.119 TS_ENDT {{LIGHTNING,  ".*"},{NONE, 
> "SIG=054a6eb17ed4d65183083149dbca5723"}}
> [ldm@aeolus ~/logs]$
> 
> What I'm wondering about is the RPC error (first line above).

This is indicating that the LDM on aeolus is not able to contact the
LDMs on its upstream feed hosts.  This is typically the result of 
something wrong/different with networking like a new firewall block
for outbound traffic to port 388 on the upstream feed hosts.

re:
> When you have time would you please provide some help on this?

The fact that 'rtstats' is no longer reporting real time stats back to
us reinforces the notion that something has changed in the network setup
either on aeolus, or in the domain where it lives.

I just tried to SSH to aeolus, and I get no response.  If nothing has
changed in the domain in which it lives, perhaps it simply needs its
network services restarted:

<as 'root'>
service network restart

If the machine is up and accessible from other UCSD subnets, it would
suggest that something has changed in a firewall somewhere, and that
is preventing aeolus from communicating with the outside world.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: AQC-617871
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.