[LDM #MFT-584849]: Errors from noaaportIngester process

Hi Carissa, et. al,

Carissa Klemmer wrote:
> I've got great news. Bhagat made the change to net.ipv4.ipfrag_max_dist=4096
> in Boulder. We went from seeing 4-5 OOPS errors/minute to 6 over the last 
> hour.
> This is now on par with the errors we see in College Park. We will wait a 
> little
> more time and then test the setting to 0 to see if we can eliminate the errors
> completely. But given this wonderful improvement I think it is safe to cancel
> this meeting. I appreciate the support of everyone jumping in to assist, and
> discovering discrepancies on the systems that we will still work towards 
> fixing. 

This is excellent news!

For interest, I counted the number of "OOPS" error messages (ERROR: OOPS) that
we have seen on both of our Linux-based NOAAport ingest machines here in UCAR
for the past week.  Here are the results:

Day       # OOPS error messages
CCYYMMDD  leno      chico
---------+-------------------------
20180212  0         0 (partial day)
20180211  3         1
20180210  0         1
20180209  0         0
20180208  2         1
20180207  126       125
20180206  6         7
20180205  2         1

Comments:

- each of our ingest machines is connected to its own Novra S300N receiver

  We interpret the difference in ingest quality to be entirely a function
  of the Novra S300N receiver it is connected to.  We have been told that
  the Novra S300N receivers are essentially hand built, and that there can
  be quite a bit of difference in their ability to process the NOAAPort
  datastream.

- both Novra S300N receivers get their signal from the same NOAAport dish

  We split the signal from the dish 7 ways, but only have Novra S300Ns
  connected to 4 our the 7 outputs; two of the Novra S300Ns are ours
  and 2 are COMET's.  The other outputs have terminators installed.

- we do not routinely count OOPS messages in the NOAAPort ingest log
  files

  Instead, we farm our log files every day and record measures of ingest
  quality.  Here are the last 7 days for both of our ingest machines
  mentioned above:

  leno:

  leno:: 20180205.204102: nGap:     23 nFrame:         30 nGsec:    10 nGmin:   
 8
  leno:: 20180206.235958: nGap:    277 nFrame:       5027 nGsec:    59 nGmin:   
29
  leno:: 20180207.222601: nGap:  14512 nFrame: 1327620418 nGsec:  1240 nGmin:   
34
  leno:: 20180208.235340: nGap:     14 nFrame:         33 nGsec:     6 nGmin:   
 5
  leno:: 20180209.225601: nGap:     90 nFrame:       1644 nGsec:    20 nGmin:   
16
  leno:: 20180210.234001: nGap:     25 nFrame:         74 nGsec:     9 nGmin:   
 9
  leno:: 20180211.221201: nGap:     53 nFrame:        157 nGsec:    13 nGmin:   
12

  chico:

  chico:: 20180205.135535: nGap:     10 nFrame:         18 nGsec:     6 nGmin:  
  6
  chico:: 20180206.235958: nGap:    273 nFrame:       2379 nGsec:    57 nGmin:  
 28
  chico:: 20180207.221436: nGap:  14515 nFrame: 1327612491 nGsec:  1240 nGmin:  
 30
  chico:: 20180208.235340: nGap:     15 nFrame:         38 nGsec:     8 nGmin:  
  7
  chico:: 20180209.224701: nGap:     84 nFrame:       1657 nGsec:    18 nGmin:  
 13
  chico:: 20180210.234002: nGap:     29 nFrame:         99 nGsec:     5 nGmin:  
  4
  chico:: 20180211.213301: nGap:     52 nFrame:        177 nGsec:    10 nGmin:  
  8


  The numbers being logged are:

  nGap   - number of Gap messages (we focus on Gap instead of OOPS messages)
  nFrame - number of missed frames
  nGsec  - number of seconds in which we saw Gap messages
  nGmin  - number of minutes in which we saw Gap messages

I hope that this info is useful!

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: MFT-584849
Department: Support NOAAPORT
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.