[LDM #OWC-960244]: pqing stripping byte

Carissa,

Thank you for the clear exposition of the problem. I was laboring under the 
assumption that the ESC character was outside the WMO bulletin rather than in 
the bulletin.

There is a line of code in pqing(1) that will delete every ESC character found 
in a WMO bulletin. This line of code dates from 1995 and has no explanation for 
its existence. I can only assume that an ESC character was used to pad WMO 
bulletins so that they didn't accidentally contain a control sequence. The 
webpage <http://www.nws.noaa.gov/tg/head.php> doesn't mention this, however.

Can you check this assumption with someone?

> Steve,
> 
> With the holiday on us know NCEP won't have the staff to meet until
> after the new year. But I would like to throw out some more information
> now that I have dug in more. Maybe clear up some confusion I might have
> caused from my first email and save a telecon with everyone.
> 
> Here is an example of SFPA99 KWBC 071426. When the flat file comes into
> us from the TOC gateway it has a header that looks like this -
> 
> 53 46 50 41 39 39 20 4b 57 42 43 20 30 37 31 34 32 36 0d 0d 0a 1b 6c
> 5c 50 8a 2d f6 2a d7 fa 3b...etc
> 
> The 1b is the first byte of "data" after the header. However, in my last
> email to you folks I missed one very important step. Before this file is
> piped into the pqing NCEP adds more header/footer information to it. To
> prove that this step wasn't stripping out the 1b I hijacked the NCEP
> code and was able to print out what would instead have gone into the
> pqing. This is what the pqing gets -
> 
> 01 0d 0d 0a 36 36 38 20 0d 0d 0a 53 46 50 41 39 39 20 4b 57 42 43 20
> 30 37 31 34 32 36 0d 0d 0a 1b 6c 5c 50 8a 2d f6 2a d7 fa 3b..etc
> 
> You see we add the appropriate SOH CR CR NL to the beginning. We also
> add the three digit random number followed by CR CR NL that the decoders
> use. And you can see that the 0x1b is way down later in the stream,
> still present.
> 
> We write the SFPA99 to a FILE and PIPE to a decoder via the pqact. The
> decoder chokes on this file, and the hexdump output of the FILE command
> is this -
> 
> 01 0d 0d 0a 36 36 38 20 0d 0d 0a 53 46 50 41 39 39 20 4b 57 42 43 20
> 30 37 31 34 32 36 0d 0d 0a 6c 5c 50 8a 2d f6 2a d7 fa 3b..etc
> 
> Now the 0x1b is missing from the data stream.
> 
> We spoke with the TOC/WMO experts asking if for some reason the 0x1b is
> not valid. They are not aware of any rules it violates, especially given
> this is in the "data" section, and they don't restrict anything there.
> 
> We are hoping you can again give this a look over and help us understand
> why the pqing would be removing this byte. If I run pqinsert on this
> attached file the byte is not removed and decoder is happy, so we have a
> workaround if necessary. But we would like understanding from you folks
> why this doesn't work with the pqing before we go down that road. I
> tested pqing on both rhel5 and rhel6. With ldm versions ldm-6.10.1 and
> 6.11.5.
> 
> If you still would like more clarity we would be happy to chat on the
> phone. I have also attached the text file that would be inserted into
> the pqing hoping you can test and replicate the problem. This file
> consists of the complete header/footer that we add to it before it gets
> inserted.
> 
> Thanks,
> 
> Carissa


Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: OWC-960244
Department: Support LDM
Priority: Normal
Status: Closed