Re: data stops ingesting from SDI (fwd)

David,

Were you helped on the following?

Regards,
Steve Emmerson

> Date: 13 Oct 2003 12:15:19 -0500
> From: David Larson <address@hidden>
> To: address@hidden
> Subject: data stops ingesting from SDI
>
> Robb,
> 
> Sorry to email you directly like this, but I thought I'd run something
> by you that seems kinda low-level.  But before I get to that ...
> 
> Every day or so, data stops arriving to our LDM from SSEC/SDI.
> 
> Post mortem analysis goes something like this:
> 
> Hmmm, the data in our system is old!  Lets check out LDM:
> 
>       [ldm@decoder3 ~]$ ldmadmin watch
>       (Type ^D or ^C when finished)
>       ^C
>       [ldm@decoder3 ~]$
> 
> After a minute or so (plenty of time), no data arrived so I killed it.
> Well, what is the pqing doing?
> 
>       [ldm@decoder3 ~/etc]$ strace -p 14206
>       write(0, "\0", 1 <unfinished ...>
>       [ldm@decoder3 ~/etc]$
> 
> Hmmmm, that's odd, it is stuck on a write to fd#0.  What file is it
> trying to write a null byte to?  Let's see what the file descriptor
> points to:
> 
> [ldm@decoder3 ~/etc]$ /usr/sbin/lsof | grep pqing | grep 0u
> pqing     14206     ldm    0u  IPv4    2087720                 TCP
> decoder3.digitalcyclone.com:39519->noaaport.digitalcyclone.com:1501
> (ESTABLISHED)
> 
> Yikes!  That is the TCP connection to our SDI box!  And the other end of
> that connection is just a "cat" sending data from a FIFO to the TCP
> connection ...
> 
> Why is that happening?  I looked at pqing.c and saw some VOODOO that
> seemed pretty reasonable, and also gave me a clue as to why this is
> somewhat unpredictable (it only does the write when the select
> times-out).
> 
> It makes perfect sense to try to write to the socket and induce a
> connection reset by peer, but unfortunately, I believe that if the other
> end of the connection doesn't eat these up, the buffers on both sides of
> the TCP connection will eventually fill up and cause the pqing to block.
> 
> Actually, I kinda like the VOODOO and I'd really like to enhance the SDI
> side so that it discards anything that arrives (it only needs to do a
> non-blocking read every now and then to empty the buffer).  Otherwise, I
> might settle for TCP KEEPALIVE (not as good).
> 
> Thoughts?
> 
> Thanks,
> Dave