[LDM #CTE-620214]: Satellite images slow or halt each day

> ldm:~/var/logs$ fgrep '[6954]' `ls -rt $HOME/var/logs/ldmd.log*` | tail -100
> 
> /home/ldm/var/logs/ldmd.log.3:20190813T205733.658289Z 
> idd.unidata.ucar.edu[6954] NOTE ldm_config_file.c:733:requester_exec() 
> Starting Up(6.13.6): idd.unidata.ucar.edu:388 20190813195733.658252 TS_ENDT 
> {{NEXRAD2, ".*"}}
> /home/ldm/var/logs/ldmd.log.3:20190813T205733.658338Z 
> idd.unidata.ucar.edu[6954] NOTE ldm_config_file.c:552:getPreviousProdInfo() 
> Previous product-information file ".5700c941fc2b72cc871186229e170a1d.info" 
> doesn't exist
> /home/ldm/var/logs/ldmd.log.3:20190813T205733.658469Z 
> idd.unidata.ucar.edu[6954] NOTE requester6.c:588:req6_new() LDM-6 desired 
> product-class: 20190813195733.658452 TS_ENDT {{NEXRAD2, ".*"}}
> /home/ldm/var/logs/ldmd.log.3:20190813T205733.843580Z 
> idd.unidata.ucar.edu[6954] NOTE requester6.c:311:make_request() Upstream 
> LDM-6 on idd.unidata.ucar.edu is willing to be a primary feeder

The above shows a successful connection and subscription for the NEXRAD level 
II products.

> /home/ldm/var/logs/ldmd.log.3:20190813T211802.850167Z 
> idd.unidata.ucar.edu[6954] NOTE svc_tcp.c:338:readtcp() select() timeout on 
> socket 6
> /home/ldm/var/logs/ldmd.log.3:20190813T211802.850197Z 
> idd.unidata.ucar.edu[6954] NOTE one_svc_run.c:95:one_svc_run() one_svc_run(): 
> RPC layer closed connection
> /home/ldm/var/logs/ldmd.log.3:20190813T211802.850202Z 
> idd.unidata.ucar.edu[6954] NOTE ldm_config_file.c:841:requester_exec() Log 
> messages flushed
> /home/ldm/var/logs/ldmd.log.3:20190813T211802.850213Z 
> idd.unidata.ucar.edu[6954] NOTE error.c:236:err_log() Connection to upstream 
> LDM closed: pid=27194

The above shows that the downstream LDM didn't receive anything (including 
heartbeat packets) from the upstream LDM for so long that it declared the 
connection dead.

> /home/ldm/var/logs/ldmd.log.3:20190813T211902.851241Z 
> idd.unidata.ucar.edu[6954] NOTE requester6.c:588:req6_new() LDM-6 desired 
> product-class: 20190813201902.851196 TS_ENDT {{NEXRAD2, ".*"},{NONE, 
> "SIG=f4e4c8cdc49c5e33beedca82831eb63e"}}
> /home/ldm/var/logs/ldmd.log.3:20190813T211902.975253Z 
> idd.unidata.ucar.edu[6954] NOTE requester6.c:311:make_request() Upstream 
> LDM-6 on idd.unidata.ucar.edu is willing to be a primary feeder

The above shows a successful re-connection and re-subscription one minute later 
for the NEXRAD Level II products.

> /home/ldm/var/logs/ldmd.log.3:20190813T222814.807184Z 
> idd.unidata.ucar.edu[6954] NOTE ldmd.c:306:signal_handler() SIGTERM received
> /home/ldm/var/logs/ldmd.log.3:20190813T222814.807261Z 
> idd.unidata.ucar.edu[6954] ERROR svc_tcp.c:361:readtcp() Bad file descriptor
> /home/ldm/var/logs/ldmd.log.3:20190813T222814.807268Z 
> idd.unidata.ucar.edu[6954] ERROR svc_tcp.c:361:readtcp() read() error on 
> socket 6
> /home/ldm/var/logs/ldmd.log.3:20190813T222814.807282Z 
> idd.unidata.ucar.edu[6954] ERROR svc_tcp.c:383:writetcp() Bad file descriptor
> /home/ldm/var/logs/ldmd.log.3:20190813T222814.807285Z 
> idd.unidata.ucar.edu[6954] ERROR svc_tcp.c:383:writetcp() writetcp(): write() 
> error on socket 6
> /home/ldm/var/logs/ldmd.log.3:20190813T222814.807301Z 
> idd.unidata.ucar.edu[6954] NOTE ldmd.c:187:cleanup() Exiting

The above shows that the downstream LDM was externally terminated a little more 
than an hour later.

I don't see anything untoward.

I'm afraid I had you search for the wrong PID based on a previous email. I do 
apologize.

What feed are you having problems with (i.e., what LDM feedtype corresponds to 
your satellite imagery)?

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: CTE-620214
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.