[LDM #UAK-912261]: data flow problems

Karen,

> Also I left the notifyme using pluto's IP address (172.16.20.32) running
> while I went to lunch.  It seems to get a lot of disconnects.
> 
> The log shows:
> 
> May 29 16:38:31 pluto pluto.protect.nssl(noti)[10755] NOTE: Starting
> Up(6.6.5/5): 20080529163831.634 TS_ENDT {{ANY,  ".*"}}
> May 29 16:38:31 pluto pluto.protect.nssl(noti)[10755] NOTE: topo:
> pluto.protect.nssl ANY
> May 29 16:43:39 pluto pluto.protect.nssl(noti)[10755] ERROR:
> /data/2008/05/29/KFDR_RVP.20080529.164211.vcp21.1.lipc.gz: RPC: Unable
> to receive
> May 29 16:43:39 pluto pluto.protect.nssl(noti)[10755] ERROR: pq_sequence
> failed: Input/output error (errno = 5)
> May 29 16:43:57 pluto pluto.protect.nssl(noti)[11095] NOTE: Starting
> Up(6.6.5/5): 20080529163831.634 TS_ENDT {{ANY,  ".*"}}
> May 29 16:43:57 pluto pluto.protect.nssl(noti)[11095] NOTE: topo:
> pluto.protect.nssl ANY
> May 29 16:52:22 pluto pluto.protect.nssl(noti)[11095] ERROR:
> nullproc5(pluto.protect.nssl): RPC: Unable to receive
> May 29 16:52:22 pluto pluto.protect.nssl(noti)[11095] NOTE: Exiting
> May 29 16:52:41 pluto pluto.protect.nssl(noti)[11787] NOTE: Starting
> Up(6.6.5/5): 20080529164455.430 TS_ENDT {{ANY,  ".*"}}
> May 29 16:52:41 pluto pluto.protect.nssl(noti)[11787] NOTE: topo:
> pluto.protect.nssl ANY
> May 29 17:01:24 pluto pluto.protect.nssl(noti)[11787] ERROR:
> nullproc5(pluto.protect.nssl): RPC: Unable to receive
> May 29 17:01:24 pluto pluto.protect.nssl(noti)[11787] NOTE: Exiting
> May 29 17:01:49 pluto pluto.protect.nssl(noti)[12417] NOTE: Starting
> Up(6.6.5/5): 20080529165616.303 TS_ENDT {{ANY,  ".*"}}
> May 29 17:01:49 pluto pluto.protect.nssl(noti)[12417] NOTE: topo:
> pluto.protect.nssl ANY
> May 29 17:11:26 pluto pluto.protect.nssl(noti)[12417] ERROR:
> nullproc5(pluto.protect.nssl): RPC: Unable to receive
> May 29 17:11:26 pluto pluto.protect.nssl(noti)[12417] NOTE: Exiting
> May 29 17:11:26 pluto pluto.protect.nssl(noti)[13042] NOTE: Starting
> Up(6.6.5/5): 20080529170555.621 TS_ENDT {{ANY,  ".*"}}
> May 29 17:11:26 pluto pluto.protect.nssl(noti)[13042] NOTE: topo:
> pluto.protect.nssl ANY
> May 29 17:20:59 pluto pluto.protect.nssl(noti)[13042] ERROR:
> nullproc5(pluto.protect.nssl): RPC: Unable to receive
> May 29 17:20:59 pluto pluto.protect.nssl(noti)[13042] NOTE: Exiting
> May 29 17:21:02 pluto pluto.protect.nssl(noti)[14205] NOTE: Starting
> Up(6.6.5/5): 20080529171526.962 TS_ENDT {{ANY,  ".*"}}
> May 29 17:21:02 pluto pluto.protect.nssl(noti)[14205] NOTE: topo:
> pluto.protect.nssl ANY
> May 29 17:30:06 pluto pluto.protect.nssl(noti)[14205] ERROR:
> nullproc5(pluto.protect.nssl): RPC: Unable to receive
> May 29 17:30:06 pluto pluto.protect.nssl(noti)[14205] NOTE: Exiting
> May 29 17:30:27 pluto pluto.protect.nssl(noti)[14825] NOTE: Starting
> Up(6.6.5/5): 20080529172456.096 TS_ENDT {{ANY,  ".*"}}
> May 29 17:30:27 pluto pluto.protect.nssl(noti)[14825] NOTE: topo:
> pluto.protect.nssl ANY
> May 29 17:40:03 pluto pluto.protect.nssl(noti)[14825] ERROR:
> nullproc5(pluto.protect.nssl): RPC: Unable to receive
> May 29 17:40:03 pluto pluto.protect.nssl(noti)[14825] NOTE: Exiting
> May 29 17:40:27 pluto pluto.protect.nssl(noti)[15483] NOTE: Starting
> Up(6.6.5/5): 20080529173455.890 TS_ENDT {{ANY,  ".*"}}
> May 29 17:40:27 pluto pluto.protect.nssl(noti)[15483] NOTE: topo:
> pluto.protect.nssl ANY
> May 29 17:50:05 pluto pluto.protect.nssl(noti)[15483] ERROR:
> nullproc5(pluto.protect.nssl): RPC: Unable to receive
> May 29 17:50:05 pluto pluto.protect.nssl(noti)[15483] NOTE: Exiting
> May 29 17:50:11 pluto pluto.protect.nssl(noti)[16177] NOTE: Starting
> Up(6.6.5/5): 20080529174435.519 TS_ENDT {{ANY,  ".*"}}
> May 29 17:50:11 pluto pluto.protect.nssl(noti)[16177] NOTE: topo:
> pluto.protect.nssl ANY
> May 29 17:59:13 pluto pluto.protect.nssl(noti)[16177] ERROR:
> nullproc5(pluto.protect.nssl): RPC: Unable to receive
> May 29 17:59:13 pluto pluto.protect.nssl(noti)[16177] NOTE: Exiting
> May 29 17:59:36 pluto pluto.protect.nssl(noti)[16822] NOTE: Starting
> Up(6.6.5/5): 20080529175405.564 TS_ENDT {{ANY,  ".*"}}
> May 29 17:59:36 pluto pluto.protect.nssl(noti)[16822] NOTE: topo:
> pluto.protect.nssl ANY
> May 29 18:08:45 pluto pluto.protect.nssl(noti)[16822] ERROR:
> nullproc5(pluto.protect.nssl): RPC: Unable to receive
> May 29 18:08:45 pluto pluto.protect.nssl(noti)[16822] NOTE: Exiting
> May 29 18:09:09 pluto pluto.protect.nssl(noti)[17371] NOTE: Starting
> Up(6.6.5/5): 20080529180337.695 TS_ENDT {{ANY,  ".*"}}
> May 29 18:09:09 pluto pluto.protect.nssl(noti)[17371] NOTE: topo:
> pluto.protect.nssl ANY

The log messages show the notifyme(1) process connecting,
requesting data, and then the upstream LDM failing to receive
a response from the notifyme(1) process after sending it
the metadata of a data-product.  The log messages indicate
that the RPC call from the upstream LDM to the notifyme(1)
process did not return (i.e., that the upstream LDM wasn't
able to obtain an acknowledgment of receipt from the
notifyme(1) process.

I've never seen this before and am at a loss to explain it.

Mike?

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: UAK-912261
Department: Support LDM
Priority: Normal
Status: Open