[IDD #SGN-156002]: SXXX91 KWBC product with \r\r\n\c within it

Hi Daryl,

re:
> My LDM PIPE ingestor uses '\r\r\n\c' as a string seperator between
> products.  This works well until this string is found within a product,
> like the attached!

Hmm...

re:
> Oct 01 20:33:14 pqcat INFO:      73786 20151001202533.335 IDS|DDPLUS
> 642833608  SXXX91 KWBC 012006 /pCRNH02
> 
> This is invalid formatting/content right?

I believe that the message is not formatted correctly.  I don't think that
the internal '\r\r\n\c' sequence is added by our NOAAPort ingest software,
but I will need to check to make sure.

re:
> Is this something you can let the NWS know about?

After verifying that the control sequence was not added by our NOAAPort
ingest software (and, again, I don't think that it is), I will certainly
send it along to the NWS for comment/possible action.

By the way, I have not heard back from the NWS NOAAPort folks about
the existence of LOTS of "duplicate" NEXRAD Level III products being
sent in the SBN.  I compiled some high level stats on how frequently
this occurred over the 5 day period of 20150924 - 20150928, and the
numbers are pretty interesting.   Even though I will be uploading
the gzipped tar files of the results for each day, I figure it might
be interesting to you to see the overall numbers:

ALL:ALL:: nDups: 46236 nRtran: 12277 nOther:     0 nSzdif:  6970
ALL:ALL:: nDups: 41228 nRtran: 12280 nOther:     0 nSzdif:  4668
ALL:ALL:: nDups: 41492 nRtran:  9936 nOther:     0 nSzdif:  5677
ALL:ALL:: nDups: 40078 nRtran:  6071 nOther:     0 nSzdif:  5295
ALL:ALL:: nDups: 41403 nRtran: 29169 nOther:     0 nSzdif:  5907

The meanings of the fields are:

ALL:ALL   - All Level III stations : All Level III products
nDups     - number of products sent by a Level III station that had the
            same Product ID year, month and day but different MD5
            signature
nRtran    - number of products sent by all Level III stations for a
            particular year, month and day that had the same MD5
            signature
nOther    - a catch all counter for products that didn't fit into
            nDups and nRtran; this number is always 0, but I included it
            as a sanity check
nSzdif    - number of nDups products that differed in product size
            So, nDups - nSzdif represents the number of "duplicate"
            products that had the same size (byte count) but different
            MD5 signature.  This is a very interesting number in my
            mind

As you can see, there are LOTS of "duplicate" products, and LOTS of
retransmits of the same products.  I understand the retransmits, but
I do not understand the "duplicate"s.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: SGN-156002
Department: Support IDD
Priority: Normal
Status: Closed