[IDD #CLZ-441499]: Noticing retransmitted? products over IDD, noaaport or IDD issue?

Hi Daryl,

re:
> Over the past day or so, I have noticed a considerable number of 
> "duplicate" products flowing over the IDD, of course after a considerable
> delay as LDM deduplication would squash these otherwise. I am curious
> from your noaaport ingest logs, if you can see any further details on
> this or if there is some issue with the IDD.

It is most likely that the high number of retransmissions of products is
a result of the twice per year solar interference (SI) period that
NOAAPort ingest sites are experiencing.  The number of duplicates
detected and rejected by the LDMs on all of the ingest machines that
feed into the IDD have been much larger over the past three days as
the SI period has progressed, so it would not be surprising that the
numbers that were not detected and rejected would be way up also.
I would guess that the cause of the "late" duplicates is the intentional
retransmission of products in NOAAPort so that WFO sites get the
products that they missed because of their own SI caused data loss.

re:
> Here's an example, that my id3b site shows nicely.
> 
> https://id3b.agron.iastate.edu/
> 
> Put in "/pMWWMHX" for the Product ID.
> 
> It shows this product:
> 
> WHUS72 KMHX 050539
> MWWMHX
> 
> URGENT - MARINE WEATHER MESSAGE
> National Weather Service Newport/Morehead City NC
> 1239 AM EST Fri Mar 5 2021
> 
> Arriving once at: 2021-03-05T05:39:18Z  (good) and then again at
> 2021-03-05T08:06:30Z  (bad).  So that ~3 hour defeats the LDM 1-hour
> queue size for most hosts on the IDD.  The LDM sequence numbers are
> different.
> 
> 319 for the first and 214 for the second.
> 
> My feed topology to idd-dc.agron.iastate.edu is just
> 
> request FSL2|WMO|NGRID|FNEXRAD  ".*"    iddb.unidata.ucar.edu
> 
> So I am curious if you could kindly check your noaaport ingest logs
> to see if it was actually received twice.  

There is no doubt that products were received multiple times, and
I believe that the biannual SI period is the cause.

re:
> Maybe the NWS has some
> bad receive site that is creating lots of retransmission requests?  Shrug.

Exactly!

At the risk of providing too much information, I am including what I call
summary Gap stats for the past 7 days for all of the Unidata NOAAPort
ingest machines in the following:

burns.unidata.ucar.edu MAC=00-06-76-05-02-a9 Qsize=2100M
  burns:: 20210226.213112: nGap:      5 nFrame:       4164 nG1sec:     4 
nG5sec:     5 nG15sec:     5 nG1min:     5 nProds:  6414367 nDups:  13830 
nPinfo:     14 nPtot:  6428197
  burns:: 20210227.040737: nGap:      2 nFrame:          4 nG1sec:     2 
nG5sec:     2 nG15sec:     2 nG1min:     2 nProds:  6482936 nDups:  30206 
nPinfo:      0 nPtot:  6513142
  burns:: 20210228.214732: nGap:    222 nFrame:        969 nG1sec:   219 
nG5sec:   219 nG15sec:   220 nG1min:   220 nProds:  6553229 nDups:  26394 
nPinfo:      0 nPtot:  6579623
  burns:: 20210301.180050: nGap:  15015 nFrame:     189682 nG1sec: 14941 
nG5sec: 14990 nG15sec: 15006 nG1min: 15013 nProds:  6725350 nDups:  83207 
nPinfo:      0 nPtot:  6808557
  burns:: 20210302.225104: nGap:   3600 nFrame:     904040 nG1sec:  3575 
nG5sec:  3588 nG15sec:  3594 nG1min:  3597 nProds:  6556313 nDups: 157711 
nPinfo:      0 nPtot:  6714024
  burns:: 20210303.212917: nGap:   3112 nFrame:    1141919 nG1sec:  3095 
nG5sec:  3105 nG15sec:  3109 nG1min:  3109 nProds:  6283289 nDups: 109530 
nPinfo:      0 nPtot:  6392819
  burns:: 20210304.180325: nGap:   3267 nFrame:    1043845 nG1sec:  3252 
nG5sec:  3261 nG15sec:  3265 nG1min:  3266 nProds:  6559690 nDups: 112331 
nPinfo:      0 nPtot:  6672021
desi.unidata.ucar.edu MAC=00-06-76-05-02-a9 Qsize=2100M
   desi:: 20210226.223842: nGap:      9 nFrame:      19880 nG1sec:     2 
nG5sec:     4 nG15sec:     5 nG1min:     5 nProds:  6412759 nDups:  13831 
nPinfo:     14 nPtot:  6426590
   desi:: 20210227.232320: nGap:     24 nFrame:        186 nG1sec:     2 
nG5sec:     2 nG15sec:     2 nG1min:     2 nProds:  6482915 nDups:  30183 
nPinfo:      0 nPtot:  6513098
   desi:: 20210228.233816: nGap:    227 nFrame:        988 nG1sec:   219 
nG5sec:   219 nG15sec:   220 nG1min:   220 nProds:  6553361 nDups:  26381 
nPinfo:      0 nPtot:  6579742
   desi:: 20210301.235447: nGap:  15032 nFrame:     189754 nG1sec: 14943 
nG5sec: 14992 nG15sec: 15008 nG1min: 15016 nProds:  6725352 nDups:  83180 
nPinfo:      0 nPtot:  6808532
   desi:: 20210302.225104: nGap:   3605 nFrame:     904065 nG1sec:  3575 
nG5sec:  3588 nG15sec:  3594 nG1min:  3597 nProds:  6556313 nDups: 157704 
nPinfo:      0 nPtot:  6714017
   desi:: 20210303.212917: nGap:   3119 nFrame:    1141952 nG1sec:  3095 
nG5sec:  3105 nG15sec:  3109 nG1min:  3109 nProds:  6283291 nDups: 109522 
nPinfo:      0 nPtot:  6392813
   desi:: 20210304.225059: nGap:   3274 nFrame:    1043909 nG1sec:  3251 
nG5sec:  3260 nG15sec:  3264 nG1min:  3265 nProds:  6559591 nDups: 112413 
nPinfo:      0 nPtot:  6672004
jackie.unidata.ucar.edu MAC=00-06-76-05-02-a9 Qsize=2100M
 jackie:: 20210219.215758: nGap:     27 nFrame:         70 nG1sec:    25 
nG5sec:    25 nG15sec:    25 nG1min:    25 nProds:  6238746 nDups:  17671 
nPinfo:      0 nPtot:  6256417
 jackie:: 20210220.202710: nGap:     34 nFrame:        875 nG1sec:    32 
nG5sec:    34 nG15sec:    34 nG1min:    34 nProds:  6107137 nDups:  17222 
nPinfo:      0 nPtot:  6124359
 jackie:: 20210221.000000: nGap:      0 nFrame:          0 nG1sec:     0 
nG5sec:     0 nG15sec:     0 nG1min:     0 nProds:  5973069 nDups:   9884 
nPinfo:      0 nPtot:  5982953
 jackie:: 20210222.152239: nGap:      2 nFrame:          2 nG1sec:     1 
nG5sec:     1 nG15sec:     1 nG1min:     1 nProds:  6285471 nDups:  35673 
nPinfo:      0 nPtot:  6321144
 jackie:: 20210223.182731: nGap:    217 nFrame:       1107 nG1sec:   210 
nG5sec:   213 nG15sec:   215 nG1min:   215 nProds:  6130550 nDups:  12099 
nPinfo:      0 nPtot:  6142649
 jackie:: 20210224.225854: nGap:     16 nFrame:         73 nG1sec:    15 
nG5sec:    15 nG15sec:    15 nG1min:    15 nProds:  6077215 nDups:  11299 
nPinfo:      0 nPtot:  6088514
 jackie:: 20210225.175214: nGap:     76 nFrame:        245 nG1sec:    75 
nG5sec:    75 nG15sec:    75 nG1min:    75 nProds:  6135929 nDups:  17036 
nPinfo:     14 nPtot:  6152965
leno.unidata.ucar.edu MAC=00-06-76-05-02-a9 Qsize=12G
   leno:: 20210226.212606: nGap:      5 nFrame:       5749 nG1sec:     4 
nG5sec:     4 nG15sec:     4 nG1min:     4 nProds:  6370113 nDups:  56925 
nPinfo:     14 nPtot:  6427038
   leno:: 20210227.040737: nGap:      2 nFrame:          4 nG1sec:     2 
nG5sec:     2 nG15sec:     2 nG1min:     2 nProds:  6441722 nDups:  71417 
nPinfo:      0 nPtot:  6513139
   leno:: 20210228.214732: nGap:    222 nFrame:        969 nG1sec:   219 
nG5sec:   219 nG15sec:   220 nG1min:   220 nProds:  6510125 nDups:  69716 
nPinfo:      0 nPtot:  6579841
   leno:: 20210301.180050: nGap:  15012 nFrame:     189633 nG1sec: 14938 
nG5sec: 14987 nG15sec: 15003 nG1min: 15010 nProds:  6661065 nDups: 147500 
nPinfo:      0 nPtot:  6808565
   leno:: 20210302.225104: nGap:   3597 nFrame:     904033 nG1sec:  3572 
nG5sec:  3585 nG15sec:  3591 nG1min:  3594 nProds:  6507362 nDups: 206664 
nPinfo:      0 nPtot:  6714026
   leno:: 20210303.212917: nGap:   3111 nFrame:    1141912 nG1sec:  3094 
nG5sec:  3104 nG15sec:  3108 nG1min:  3108 nProds:  6237688 nDups: 155128 
nPinfo:      0 nPtot:  6392816
   leno:: 20210304.180325: nGap:   3264 nFrame:    1043838 nG1sec:  3249 
nG5sec:  3258 nG15sec:  3262 nG1min:  3263 nProds:  6445749 nDups: 226277 
nPinfo:      0 nPtot:  6672026
uni14.unidata.ucar.edu MAC=00-06-76-05-02-a8 Qsize=100G
  uni14:: 20210226.213041: nGap:      9 nFrame:      58729 nG1sec:     4 
nG5sec:     5 nG15sec:     6 nG1min:     6 nProds:  6334693 nDups:  90558 
nPinfo:     14 nPtot:  6425251
  uni14:: 20210227.144201: nGap:      4 nFrame:          6 nG1sec:     2 
nG5sec:     2 nG15sec:     2 nG1min:     2 nProds:  6402274 nDups: 110865 
nPinfo:      0 nPtot:  6513139
  uni14:: 20210228.214732: nGap:    244 nFrame:       1042 nG1sec:   240 
nG5sec:   241 nG15sec:   242 nG1min:   242 nProds:  6465149 nDups: 114693 
nPinfo:      0 nPtot:  6579842
  uni14:: 20210301.180105: nGap:   4369 nFrame:     270576 nG1sec:  4296 
nG5sec:  4344 nG15sec:  4359 nG1min:  4367 nProds:  6585454 nDups: 219290 
nPinfo:      0 nPtot:  6804744
  uni14:: 20210302.225104: nGap:   2403 nFrame:     955039 nG1sec:  2375 
nG5sec:  2390 nG15sec:  2397 nG1min:  2400 nProds:  6295925 nDups: 415889 
nPinfo:      0 nPtot:  6711814
  uni14:: 20210303.212917: nGap:   2834 nFrame:    1197670 nG1sec:  2810 
nG5sec:  2824 nG15sec:  2830 nG1min:  2831 nProds:  6104113 nDups: 285407 
nPinfo:      0 nPtot:  6389520
  uni14:: 20210304.180325: nGap:   2054 nFrame:    1102815 nG1sec:  2030 
nG5sec:  2046 nG15sec:  2050 nG1min:  2052 nProds:  6067830 nDups: 602277 
nPinfo:      0 nPtot:  6670107

The salient things to not are:

nGap   - the number of Gap messages seen in all ingest log files for the day
nProds - the number of products successfully inserted into the LDM queue for 
the day
nDups  - the number of products _not_ inserted into the LDM queue because there
           is already products with the same MD5 signatures in the LDM queue

The nG1sec, nG5sec, nG15sec and nG1min values are counts of Gaps that
occur withing 1, 5, 15 seconds and 1 minute.  They are my way of
characterizing how the Gaps are experienced.  When the fraction of
Gaps are received withing 1 second, the "noise" (SI in this case) is
very bursty, etc.

The other thing listed for each machine is the size of its LDM queue.
A quick comparison of the nDups vs LDM queue size shows how the number
of duplicates detected and rejected is a positive function of the
size of the LDM queue on the ingest machine - the larger the LDM queue
size, the more duplicates that are detected and rejected.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CLZ-441499
Department: Support NOAAPORT
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.