[IDD #CVI-315697]: Duplicate(?) products on the IDD feed from idd.unidata.ucar.edu

Hi Daryl,

Sorry for the slow reply; it is training workshop time here in the UPC
and time has been limited lately...

re:
> I feed from idd.unidata.ucar.edu and for the past few days, I have noticed
> what appears to be duplicate products coming down the feed.  I have
> attached an example NPWABR that came twice, but I don't have any
> additional metadata on this product to know pq insert times, etc.

The NWS has been rebroadcasting lots of the content in the NOAAPort
SBN lately.  We tried to minimize the effect of the rebroadcasts by
increasing the size of the LDM queue on our NOAAPort ingest machines
from 1.5 to 8 GB.  Given the volumes seen during times of rebroadcast,
the residency time in our LDM queues becomes decreasingly small (i.e.,
less than one hour) and duplicates get sent along to top level
IDD relay nodes like idd.unidata.ucar.edu.

Side comment: the quality of NOAAPort ingest (meaning the number of
missed frames) has degraded pretty much universally since the switch
over to the expanded NOAAPort broadcast... we have been fighting to
improve our ingestion for the past 6 weeks because of this.  We
contacted contact with Raytheon folks that deal with the NOAAPort
SBN, and they said that what we are seeing is pretty much what
all users are seeing.  The net result of this appears to be
more frequent rebroadcasts of data that are initiated when WFOs
are seen to miss data.

re:
> I have access to NWSTG's TOC FADD system, which denotes
> receipt/transmission times and it does not appear to show any duplicate
> receipt there.  I have attached a screenshot from it.

The data volume plots from NOAAPort ingest systems _strongly_
suggest that there have been a lot more rebroadcasts over the
past several weeks.

re:
> Do you have any insights in this?

Rebroadcast of data is all that we can think of.  Also, since more
than one NOAAport ingest site feeds into the IDD, and since at least
one of these sites (UW/SSEC) is using LDM queue sizes smaller than
what seems to be needed to detect and reject duplicate products
on their ingest machines, and since the volume in the IDD has grown
substantially with the increased volume in NOAAPort-derived 
(e.g., NEXRAD3, NGRID, NIMAGE, IDS|DDPLUS, HDS) and other (e.g.,
CONDUIT, NEXRAD2 and NEXRAD3) datastreams, it seems likely that
the LDM queue sizes on top level IDD relay machines are not enough
to detect and reject duplicate products.  This problem will get
much worse in the not too distant future when the 0.25 degree GFS
gets added to CONDUIT.  Data, data, data ...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CVI-315697
Department: Support IDD
Priority: Normal
Status: Closed