[Support #HDQ-517625]: Assistance requested for "Gap in packet sequence" log entries from noaaportIngester

Hi Greg,

re:
> Sorry for the delayed response.  I just updated the LDM config and
> restarted LDM and unfortunately see Gap errors.

Rats!

re:
> [ldmcp@sbn1 ~/logs]$ grep Gap *.log
> 
> 8.exp.log:20200624T161033.643115Z noaaportIngester[12944]
> productMaker.c:pmStart:439          WARN  *Gap* in packet sequence: 851048
> to 851758 [skipped 709]
> 
> 8.exp.log:20200624T161033.668640Z noaaportIngester[12944]
> productMaker.c:pmStart:439          WARN  *Gap* in packet sequence: 851758
> to 851781 [skipped 22]
> 
> 8.exp.log:20200624T161041.293238Z noaaportIngester[12944]
> productMaker.c:pmStart:439          WARN  *Gap* in packet sequence: 851781
> to 851834 [skipped 52]
> 
> 8.exp.log:20200624T161045.071009Z noaaportIngester[12944]
> productMaker.c:pmStart:439          WARN  *Gap* in packet sequence: 851834
> to 851836 [skipped 1]
> 
> 8.exp.log:20200624T161047.848035Z noaaportIngester[12944]
> productMaker.c:pmStart:439          WARN  *Gap* in packet sequence: 851836
> to 852438 [skipped 601]
> 
> 8.exp.log:20200624T161053.771598Z noaaportIngester[12944]
> productMaker.c:pmStart:439          WARN  *Gap* in packet sequence: 852438
> to 853528 [skipped 1089]

Given that the port 1208 ingest on sbn2 showed/shows that some products
are being processed, it would be better to do a 'less' of the 8.exp.log
file on sbn1 since it would show if there are any products being processed.
Quite frankly, we did not expect that setting up multicast routing and
switching the 'noaaportIngester' EXEC invocations without the '-I' flag
would fix anything, but it was worth the try.

I sent a note off to Steve this morning asking him to let us know 
immediately if he has any epiphanies on what may be causing the
problem.  I did some more Googling last night, but I didn't find
anything that looked even remotely promising.  Quite frankly, I
am baffled!

re:
> [ldmcp@sbn1 ~]$ ps -ef |grep noaa
> 
> root     12939 12935  5 16:10 ?        00:00:02 *noaa*portIngester -m
> 224.0.1.1 -n -l /home/ldmcp/logs/1.nmc.log
> 
> root     12940 12935  3 16:10 ?        00:00:01 *noaa*portIngester -m
> 224.0.1.2 -n -l /home/ldmcp/logs/2.goes.log
> 
> root     12941 12935 14 16:10 ?        00:00:05 *noaa*portIngester -m
> 224.0.1.3 -n -l /home/ldmcp/logs/3.nmc2.log
> 
> root     12942 12935  4 16:10 ?        00:00:01 *noaa*portIngester -m
> 224.0.1.4 -n -l /home/ldmcp/logs/4.nopt.log
> 
> root     12943 12935  4 16:10 ?        00:00:01 *noaa*portIngester -m
> 224.0.1.5 -n -l /home/ldmcp/logs/5.nmc3.log
> 
> root     12944 12935  3 16:10 ?        00:00:01 *noaa*portIngester -m
> 224.0.1.8 -n -l /home/ldmcp/logs/8.exp.log
> 
> root     12945 12935  5 16:10 ?        00:00:02 *noaa*portIngester -m
> 224.0.1.9 -n -l /home/ldmcp/logs/9.grw.log
> 
> root     12946 12935  4 16:10 ?        00:00:01 *noaa*portIngester -m
> 224.0.1.10 -n -l /home/ldmcp/logs/10.gre.log
> 
> ldmcp    13107 11940  0 16:11 pts/0    00:00:00 grep --color=auto *noaa*

The 'noaaportIngester' invocations running on sb1 now look like ours,
LSU/SRCC's, UW/SSEC's and those being run at NOAA/GSL and by NOAA/GSL
in AWS.

re:
> [ldmcp@sbn1 ~/etc]$ grep ^EXEC ldmd.conf |grep noaa
> 
> EXEC "*noaa*portIngester -m 224.0.1.1  -n -l /home/ldmcp/logs/1.nmc.log"
> EXEC "*noaa*portIngester -m 224.0.1.2  -n -l /home/ldmcp/logs/2.goes.log"
> EXEC "*noaa*portIngester -m 224.0.1.3  -n -l /home/ldmcp/logs/3.nmc2.log"
> EXEC "*noaa*portIngester -m 224.0.1.4  -n -l /home/ldmcp/logs/4.nopt.log"
> EXEC "*noaa*portIngester -m 224.0.1.5  -n -l /home/ldmcp/logs/5.nmc3.log"
> EXEC "*noaa*portIngester -m 224.0.1.8  -n -l /home/ldmcp/logs/8.exp.log"
> EXEC "*noaa*portIngester -m 224.0.1.9  -n -l /home/ldmcp/logs/9.grw.log"
> EXEC "*noaa*portIngester -m 224.0.1.10 -n -l /home/ldmcp/logs/10.gre.log"

Two thumbs up.

It would be instructive (and perhaps useful) if you could setup
monitoring of the number of Gap messages and associated missed frames
on sbn1.  What would be instructive is to be able to compare the
number of Gaps and missed frames for all channels except 1208 with
the number of Gaps being logged for data coming in on port 1208.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: HDQ-517625
Department: Support NOAAPORT
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.