[IDD #LDE-365303]: Duplicate(?) /pNCRVAX in IDD feed?

Hi Darrel, Brian and address@hidden,

As part of the investigations for duplicate NEXRAD Level III products that
we are seeing on our NOAAPort receive systems, I decided to create a summary
of how many "duplicate" and "retransmit" products we are seeing.

The summary created on one of our NOAAPort ingest machines spans the
time period of about 0Z to about 21Z today, Thursday, September 24.

I put the compressed tar file of the summary stats out on our RAMADDA
server for every to be able to grab:

motherlode.ucar.edu/repository
Unidata -> IDD -> NEXRAD Level III

The summary stats are in the file named:

NEXRAD Level III Duplicate Product and Retransmit Stats for 20150924

NB:

- I classify products as being "duplicates" if the date field in their
  Product IDs are the same, and their MD5 checksums are different

  In this case, each product is considered to be unique, and it is
  inserted in the local LDM queue and redistributed in our Internet
  Data Distribution (IDD) system.

  Observation: the difference in product creation time for "duplicate"
  products is typically very small, varying between a few milliseconds
  to a small number of seconds.

- I classify products as being "retransmits" if the date field in
  their Product IDs are the same, but their MD5 checksums are the
  same

  I this case, there already exists a product with the same MD5 checksum
  in the local LDM queue, so the newly ingest product is thrown away as
  being a duplicate; these duplicates are _not_ redistributed in the IDD.

  Observation: the time difference in product creation time for "retransmit"
  products is typically on the order of many seconds (e.g., 30+).

- our LDM product queue holds about an hour of products received in the
  NOAAPort SBN

  This means duplicate products sent more than an hour after the original
  will never be seen to be a "duplicate" or "retransmit".

The file in the compressed tar file named nexrad_dupstats.log contains the 
highest
level summary of received "duplicate" and "retransmit" products.  Here is a 
snippit
from the file:

ABC:ALL:: nDups:    18 nRtran:    44  <- ABC is the NEXRAD ID; ALL means all 
Level III products from the NEXRAD
AMX:ALL:: nDups:    41 nRtran:   126
BMX:ALL:: nDups:    20 nRtran:    42
CLX:ALL:: nDups:    43 nRtran:    83
DMX:ALL:: nDups:    38 nRtran:   136
ESX:ALL:: nDups:  2979 nRtran:   101
FTG:ALL:: nDups:    17 nRtran:    29
 ...
JAX:ALL:: nDups:  1903 nRtran:    65
LRX:ALL:: nDups:    10 nRtran:    32
MKX:ALL:: nDups:    15 nRtran:    41
MXX:ALL:: nDups:    24 nRtran:    68
POE:ALL:: nDups:    10 nRtran:    31
SJT:ALL:: nDups:    35 nRtran:    88
UEX:ALL:: nDups:    18 nRtran:   108
------------------------------------
ALL:ALL:: nDups: 43969 nRtran:  8325  <- all Level III products from all 
NEXRADs in the 0 - 21 Z time frame


Comment:

- _IF_ the way that I am classifying products as being "duplicate" or 
"retransmit"
  is correct (and I think that it is, but I am willing to be convinced 
otherwise),
  there are LOTS of "duplicates" and "retransmits" being sent in the NOAAPort 
SBN

  This seems like a very bad waste of bandwidth

I welcome comments by the recipients of this email!!!

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: LDE-365303
Department: Support IDD
Priority: High
Status: Closed