19990921: NOAAPORT SBN question on the NWSTG format

Greg,

It sounds like your prod.info.sz in the prod structure hasn't
been updated to reflect that you have memmoved the product
up into your allocated block. This is the size of the number of
bytes that you are inserting into the queue- and since you
get 20 extra bytes, it seems to coincide with the size
that you moved. Once you move the bytes, your
product size is 20 less.

When I read the noaaport stream here, I actually calculate the ccb length
from the data stream (just in case someday they decide to change its size),
and I load into the product heap the product at the appropriate offset
so I don't have to move things later. I also compute the running
MD5 checksum as each fragment is obtained and placed into the
product data pointer (not a big deal with nwstg, but it would
take considerable time to compute the MD5 for a 26MB visible
image if not doing it as the fragments arrive).

It sounds like you aren't computing the checksum which is used
for duplicate detection. Not a big deal.

If you are trying to make your products look "FOS" like with
the SOH \r\r\n for use with other decoders, I'll mention that
the Gempak source/bridge/dc/dcghdr.f routine is looking for
the FOS sequence number following the SOH sequence that you are
already adding. The realtime IDD feed that we send out from
the NOAAport creates this sequence number from the sbn modulo 1000
for a 3 digit number followed by \r\r\n. We did this to make things 
backwards compatible with the FOS stream so decoders wouldn't break.

Note that the Gempak gini reader imgi2gm.f for the imagery
products does not expect the SOH.... FOS stuff in the product
so no need to do that for the imager (GOES-E, GOES-W and gms/goes/meteo
composites). Also, our grib readers (dcgrib and gribtonc etc don't care
if the SOH is there or not).

Steve Chiswell
Unidata User Support









>From: Gregory Grosshans <address@hidden>
>Organization: .
>Keywords: 199909212202.QAA09104

>This is a multi-part message in MIME format.
>--------------A032417E9D9126CE52BDAC8D
>Content-Type: text/plain; charset=us-ascii
>Content-Transfer-Encoding: 7bit
>
>I'm injecting the NWSTG data stream from a NOAAPORT Receive System (NRS)
>into LDM and presently looking closely at the METAR data.  At the end of
>each product there are 20 erroneous bytes.  I'm wondering
>if its related to the structure 'prod'.  I'm curious if Unidata
>encountered this type of problem while developing your custom software
>to inject NRS data into LDM at Boulder?  
>
>Thanks,
>Gregg Grosshans
>Storm Prediction Center
>
>The following contains more specifics on what I've encountered.
>===============================================================
>
>Each message in the NWSTG stream has a 24 byte Communications Control
>Block (CCB) prior to the WMO header.  I'm striping the 24 bytes off and
>adding the SOH \r \r \n, and then placing the rest of
>the product into the LDM queue.  The product contains a \r \r \n ETX so
>I haven't had to add this.  The resulting product received in LDM looks
>ok except there are always 20 extra bytes at the end of each product,
>after the \r \r \n \ETX.
>
>The "INTERFACE CONTROL DOCUMENT (ICD) FOR THE NATIONAL WEATHER
>SERVICE TELECOMMUNICATIONS GATEWAY (NWSTG)", Nov. 2, 1998, (available at 
>http://www.nws.noaa.gov/noaaport/html/refs.shtml) only mentions the CCB
>and nothing about information following the product.
>
>When I enject the product into LDM the exact way its received
>from the NOAAPORT Receive System (NRS) the very end of the product is
>received and 'FILE'ed fine.  However, when I add the SOH \r\r\n
>and strip off the 24 Byte CCB using memmove I end up with 20
>bytes of erroneous characters at the end of the product when LDM
>'FILE's it.
>
>In summary the following is being done to the each product
>received from the NWSTG on the NRS:
>
>       -unsigned char pointer (msg) points to original
>        message.
>
>       -unsigned char pointer (formmsg) is calloc'ed (message
>         length -20)
>               24 bytes CCB not needed but adding 4 byte header
>       
>       -first four bytes of char pointer (formmsg) set to:
>               \001 \r \r \n
>       
>       -memmove(formmsg+4, msg+24, message_length - 20)
>               msg+24 skips 24 Byte CCB, message_length-20
>               just copies the actual product from msg, that
>               is the number of bytes for memove to copy
>               
>       -prod.data=(void *)formmsg
>       
>       -status=pq_insert(pq, &prod)
>       
>       -free(formmsg)
>
>If I print formmsg to stdout the product looks fine with no
>apparent errouneous characters at the end of the product, likewise
>when I fputs formmsg to a file.  When the product is received in
>LDM and written to a file everything appears in the product
>as desired except for 20 bytes after \r \r \n \ETX.
>--------------A032417E9D9126CE52BDAC8D
>Content-Type: text/x-vcard; charset=us-ascii;
> name="grosshans.vcf"
>Content-Transfer-Encoding: 7bit
>Content-Description: Card for Gregory Grosshans
>Content-Disposition: attachment;
> filename="grosshans.vcf"
>
>begin:vcard 
>n:Grosshans;Gregory
>tel;fax:405-579-0700
>tel;work:405-579-0720
>x-mozilla-html:TRUE
>url:www.spc.noaa.gov
>org:DOC/NOAA/NWS/NCEP/Storm Prediction Center
>adr:;;1313 Halley Circle                               ;Norman;OK;73069;
>version:2.1
>email;internet:address@hidden
>x-mozilla-cpt:;0
>fn:Gregory Grosshans
>end:vcard
>
>--------------A032417E9D9126CE52BDAC8D--
>