[Support #HDQ-517625]: Assistance requested for "Gap in packet sequence" log entries from noaaportIngester

Hi Gregg,

Sorry for the silence.  When I read your latest email below, I saw
that you were quoting an email that you apparently had sent to us
on June 19.  We never saw this email arrive in our inquiry tracking
system, so I figured that you were busy doing other things.  Your
reference to that email, however, prompted me to look at the email
that we received, and I quickly figured out why it did not make it
into our inquiry tracking system - the overall message size exceeded
the 2MB limit that our inquiry tracking system is setup to use.
Since we file all emails that we receive outside of our inquiry
tracking system, I am happy to say that we do have your previous
email, and I just extracted the attachments that you included in
it.  I am now taking a look...

Comment:

- after I sent my last email (in which I pointed out that the set
  of PIDs being processed on your Novra S300N did not match the
  set that we are configured for), Steve and I came to the conclusion
  that the channel that was causing essentially all of the Gap
  messages on your system was on port 1208, not 1205

  We figured this out by reviewing the startup messages in one of
  the log files you sent, and noting that the process ID for the
  routine that was logging all of the Gap messages was the third
  instance of 'noaaportIngester' that was sending log output to
  log facility '7'.  Comparing that with the listing of the EXEC
  lines from your LDM configuration file, ~ldm/etc/ldmd.conf, we
  saw that that instance was setup for port 1208, not 1205.

  So, what we need is some output from 'tcpdump' for port 1208.
  We don't need a LOT of 'tcpdump' output, just enough to show the
  packet sizes for when data is being received.  The packet length
  for when data is not being received should be listed as '48';
  the packet lengths for when data is being received should be '4032'.
  If the packet length is different, then it might mean that there
  is other UDP traffic being received on this port, and that traffic
  does not originate from the NOAAPort broadcast.  This is a BIG
  reach, but, quite frankly, your situation is not making much sense
  to us at the moment - i.e., why would all channels except one be
  full of good data from NOAAport when the other has data that does
  not come from NOAAPort.  The only possible explanation would be
  that the Novra S300N is sending bad data on one channel while the
  others are OK, and does not make any sense to me.

re:
> My coworker changed the PID setting, rebooted the NOVRA box, and I
> stopped/started the LDM/noaaportIngester last week.  The "show pid"
> settings at the time included both "150" and "151".  Later in the day is
> whey I cut and past the output, that I reran later, from show pid.  This
> Monday afternoon, in reviewing my email to you, I noticed "151" was missing.

Hmm... that is very strange indeed.  Is it possible that someone else
is messing around with the Novra S300N that is feeding your machine?
I ask this because I have _never_ heard of a Novra S300N changing its
configuration "spontaneously" before.

re:
> So, I just added "151", rebooted the NOVRA, stopped started
> LDM/noaaportIngester and see lots of Gap errors in the 8.exp.log file
> within the first few minutes.

Can I assume that the '8.exp.log' file is the log file for port
1208 and that is the experimental channel?  This would make sense
since those two things fit, but I want to make sure that we are
talking about the same things.

I think that the log file is for port 1208 traffic since that is
what we figured out must be the the one that is generating all
of the Gap messages.

Can you do a 'tcpdump' for port 1208 traffic and send us a snippit
of output that contains both the receipt of data and the presumed
heartbeat?  Here is an example of the kind of output that I am 
referring to:

<as 'root' on one of our NOAAPort ingest machines:

tcpdump -i eth1 -n port 1208

 ...
16:05:51.069077 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 4032
16:05:51.069524 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 4032
16:05:51.069969 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 4032
16:05:51.073029 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 4032
16:05:51.073465 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 1519
16:05:53.274342 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 48
16:05:57.039425 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 4068
16:05:58.041601 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 48
16:05:58.041837 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 4032
16:05:58.047162 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 4032
16:05:58.052490 IP 10.0.9.51.35637 > 224.0.1.8.seagull-ais: UDP, length 4032
 ...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: HDQ-517625
Department: Support NOAAPORT
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.