[LDM #OWC-960244]: pqing stripping byte

Carissa,

pqing(1) was designed to create LDM data-products from input from a serial port 
such as the NWS telecommunications gateway. As a consequence, it does ensure 
that the WMO standard for such serial communications is followed.

Can you guarantee that the input bytes follow the WMO standard?

If your input is from a file and you want to turn that file into an LDM 
data-product, then a better choice might be the pqinsert(1) utility because it 
won't vet the data at all.

> We have come across an issue in testing our new lightning feed. We get the
> data via a bundled file with numerous other observations, then split up
> each bundle, and pqing each split file into LDM. The issue is that LDM is
> sometimes stripping off a byte at the beginning of the file, causing an
> error in the decoder log. We have seen this issue twice now, both times the
> stripped byte was hex 0x1B (1b). In looking at the pqing source code I see
> some manipulation with the first characters of the file. I am curious if
> this is inadvertently stripping this byte somehow? I ran this by hand using
> pqinsert, and the byte remained, so this is specific to the pqing process.
> 
> Attached is the split file ready to be ingested (orig_file_input) and the
> file after processed by LDM (ldm_file_output).
> 
> From the hexdump below I have highlighted in red where the problem occurs.
> You can see that once the file comes out of LDM the "1b" is missing. Again,
> this is happening infrequently, and so far only when the first byte after
> the returns/header is 1b. Do you have any thoughts? Are you aware of
> anything special with this hex byte that could cause this?
> 
> >hexdump -C < orig_file_input | head
> 00000000  53 46 50 41 39 39 20 4b  57 42 43 20 30 37 31 34  |SFPA99 KWBC
> 0714|
> 00000010  32 36 0d 0d 0a 1b 6c 5c  50 8a 2d f6 2a d7 fa 3b  |26....l\P.-?*?
> 00000020  57 90 c3 0c 18 8d 71 6d  9a 31 d7 ae cb 00 90 57
> |W.?...qm.1??..W|
> 
> >hexdump -C < ldm_file_output | head
> 00000000  01 0d 0d 0a 36 36 38 20  0d 0d 0a 53 46 50 41 39  |....668
> ...SFPA9|
> 00000010  39 20 4b 57 42 43 20 30  37 31 34 32 36 0d 0d 0a  |9 KWBC
> 071426...|
> 00000020  6c 5c 50 8a 2d f6 2a d7  fa 3b 57 90 c3 0c 18 8d
> |l\P.-?*??;W.?...|
> 
> We are running version ldm-6.11.5
> 
> The pqact entries look like this -
> 
> # Test lightning data
> HDS     ^SF(US|PA)99 KWBC
> FILE    -close
> /dcomdev/us007003/ldmdata/obs/upperair/lightning/%Y%m%d%H.lightning_test
> 
> HDS     ^SF(US|PA)99 KWBC
> PIPE    exec/decod_dcltng -v 2  -t 300 -d
> /dcomdev/us007003/decoder_logs/decod_dcltng.log
> fix/bufrtab.007
> 
> --
> Carissa Klemmer
> NCEP Central Operations
> Production Management Branch Dataflow Team301-683-3835


Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: OWC-960244
Department: Support LDM
Priority: Normal
Status: Closed