19990622: Matching PIL id's in pqact

Gregg,

I modified pqing to add the pil headers ingested from NOAAport.

In order to make NOAAport look like our previous FOS data stream,
we split the nwstg data stream into the HRS feed and the DDPLUS|IDS
feed on IDD.

When we were purchasing FOS, we had separate HRS, DDS, PPS, and IDS
data streams, where the DDPLUS was the sum of DDS and PPS.
The IDD feed from NOAAport uses the DDPLUS|IDS combination for
the text portion of nwstg so that our sites which previously
had pqact.conf patterns to match DDPLUS or IDS etc could
continue to use their patters without having to make a big
change from FOS to NOAAport.

The current receiver software we use with the data aquisition board
from SSEC dumps the nwstg channel into fifo files which 2 copies of
pqing are reading (one for HRS and one for DDPLUS|IDS).

I have recently been working here with a new data aquisition board
we purchased separately and have software directly read the
packets from the board and insert the products into the data queue without
having to run pqing externally. The advantage here is much better performance
with all four channels simultaneously on the pc. Also, pqing expects
FOS like control characters SOH and \r\r\nETX to delimit products.
The SSEC software adds the FOS like wrapping onto the products so
that pqing can read it. With my work now, that added step isn't necessary-
however many decoders expect the FOS (Or AFOS ZCZC NNNN). Eventually we'll
look to making NOAAport specific feeds, instead of trying to make everything
look like FOS headers. This would give us more flexibility of describing the
data contents instead of something like TIGW01 KNES to something that
tells what channel, satellite, sector, resolution, imager band, data format,
insertion time etc. However, for the present time, using FOS headers
from NOAAport has meant our sites really don't have to know whether the
data originated from NOAAport or FOS.

In pqing, I made small modifications to pqing.c, wmo_header.c and wmo_header.h.
This version will be part of the upcoming ldm5.0.8 distribution.
wmo_header.c looks at the line following the product identifier, and
if it is 5 or 6 characters, and all are alphanumeric, then the string
is used on the ldmproduct name. I chose not to use a table of valid pils
since that becomes a problem to maintain. As a result, products like
SAUS80 KWBC 221200
METAR
....
the METAR string is placed in the tag string, even though "METAR" really isn't
a PIL, and not all METAR bulletins use that format of report.

I also chose not to verify whether the pil represents a state or stationid
since bulletins like the Fiji forecast have "TONGA" as the second line- and
I didn't want to have to assume "GA" was from Georgia to match.

There is nothing that prevents the scheme with working on binary products 
like the SDUS97 bulletins, where the second line N0Rxxx, N0Vxxxetc could
be used to identify the product etc. At present however, I only
have pqing include the /p tag onto text bulletins we send
on the DDPLUS|IDS stream. The binary products all ge put in the
HDS stream for our scheme.

Steve Chiswell




>From: Gregory Grosshans <address@hidden>
>Organization: DOC/NOAA/NWS/NCEP/Storm Prediction Center
>Keywords: 199906221756.LAA26090

>This is a multi-part message in MIME format.
>--------------E90767404A79D2F64D4F5ADB
>Content-Type: text/plain; charset=us-ascii
>Content-Transfer-Encoding: 7bit
>
>I would like to obtain some additional information about the software
>change related to the pil-like identifiers.  It sounds like the software
>that changed was the software that receives the data from NOAAPORT
>and injects it into the LDM queue.  Is this correct?  Or was it a change
>to pqing?
>
>In regards to the pil-like change to the DDPLUS|IDS feed, is this a
>separate feed compared to the NOAAPORT stream or just the
>NOAAPORT gateway channel?
>
>The SPC uses Family Of Services and AFOS data in LDM and will
>soon inject NOAAPORT data from our NOAAPORT ground
>station.  We are also looking for a good solution to the non unique
>WMO headers, and this new feature UNIDATA is supporting for
>the IDD may also work for the SPC and the rest of NCEP.  Thus,
>any information you can share is appreciated.
>
>Thanks,
>Gregg Grosshans
>NWS/NCEP/Storm Prediction Center
>
>
>Steve Chiswell wrote:
>
>> Tom,
>>
>> Several weeks ago, we implemented a change to the ingest software
>> which now adds the pil-like identifier to the DDPLUS|IDS NOAAport
>> datastream product headers. Text products which contain a line that
>> conforms to a PIL have the string appended to the LDM product
>> identifier as "/pPIL_ID" Note that this change only affects the
>> product identifier as seen in the LDM, and does not modify the
>> data product in any way. Two of the three top tier sites have
>> already upgraded to the new ingest software, so almost all
>> products available on the IDD have the pil-like identifiers
>> appended. With the additional product identifier appended to the
>> end of the WMO identifier, we have been able to implement
>> this change basically without anyone noticing it. Of course,
>> we will notify everyone when all top tier ingest sites are
>> using the updated ingester so that all data on the IDD
>> will have the tag regardless of where it was ingested.
>>
>> In the example you show below, the product that begins with:
>>
>>  742
>>  SRUS51 KPHI 221222
>>  RR1PHL
>>  ...
>>
>> will be identified in the LDM data stream as:
>>
>>  SRUS51 KPHI 221222 /pRR1PHL
>>
>> So, you can use a pqact.conf line such as:
>>
>>  DDPLUS ^SRUS51 KPHI .* /pRR1PHL
>>
>> The "/p" identifier is analogous to the "/m" identifier which
>> is appended to the model data products in the HDS feed,
>> and can be used to differentiate between products issued under
>> common WMO headers. Note that WMO headers frequently have some
>> communitaction identifiers following the DDHHMM group, such as
>> RTDxx, COR, AAA, etc. which is generally not useful for most IDD
>> data processing- but in general, the IDD products will look like:
>>
>> TTAAii CCCC DDHHMM [COM] [/pPIL_ID]
>>
>> Since not all products have PILs, the "/p" tag will only be placed on
>> those products which contain a pil-like line. Some products, such
>> as some METAR bulletins, contain second lines that look like AFOS PILS
>> while other METAR bulletins do not. As a result, you will see some
>> METAR bulletins with /pMETAR appended, while some do not. While
>> the METAR string is not an actual PIL, it does conform to what
>> a PIL looks like, so the /p tag represents a "pil-like" tag.
>> Thus, for surface METAR decoding, you should still use the ^SA.*
>> pattern and not the /p tag.
>>
>> To date, nobody has complained about any problems caused by the addition
>> of the pil tags, so we plan to have the final top tier site upgrade
>> as soon as they can.
>>
>> We hope that the community finds this addition to be a valuable
>> resource for use in the IDD. If you want to observe the pil tags
>> in action, you can use the "ldmadmin watch" command or "notifyme"
>> command to show the datastream headers.
>>
>> As always, support questions can be addressed to address@hidden.
>>
>> Good luck-
>>
>> Steve Chiswell
>> Unidata User Support
>>
>> On Tue, 22 Jun 1999, Tom Grzelak wrote:
>>
>> >
>> > Hi!
>> >
>> > I would like to set up my pqact.conf file to route certain reports to
>> > specifc programs/people.  As long as the reports have unique headers, it i
> s
>> > straight-forward.  But I am not sure how to handle the following situation
> :
>> >
>> > I have a single header which is used for SEVERAL different reports.  It is
>> > the second and/or third line which makes the report unique.  How would I
>> > set up a pqact.conf line item to do pattern matching on more than one line
> ?
>> >
>> > For example:
>> > Here's a desired product with a header which is used for several other
>> > undesired products.
>> >
>> > 742
>> > SRUS51 KPHI 221222
>> > RR1PHL
>> > .BR PHL  0622 DH12/PPP
>> > PHL .03,ABE .01,ACY .14,MIV .06,ILG .05
>> > WRI .07,NXX .04,DOV .11,PNE .10,EWR T
>> > TTN .08,SMQ .03,MPO .01,GED .08,VAY .26
>> > 12N .02,RDG .02,BLM .05,WWD .05
>> > CDW .05
>> > .END
>> >
>> > In this example, a pattern match on the header line AND the second line
>> > would allow this DESIRED product to be sent to the right person/program.
>> >
>> > THE BIG QUESTION: How do you do a pattern match for the second line?
>> >
>> > DDPLUS  ^SRUS51 KPHI
>> >         PIPE
>> >         -close -strip /usr/bin/mailx address@hidden
>> >
>> > Thanks!
>> > Tom
>> > ===============================================================
>> > Tom Grzelak                        address@hidden
>> > Manager - Computing Services
>> >  ----> Center for Environmental Prediction
>> >  ----> Department of Environmental Sciences
>> >
>> > Rutgers University
>> > 14 College Farm Road                  Voice: 732-932-4923
>> > New Brunswick, NJ 08901-8551          Fax: 732-932-8644
>> >                 http://www.envsci.rutgers.edu
>> > ===============================================================
>> >
>
>--------------E90767404A79D2F64D4F5ADB
>Content-Type: text/x-vcard; charset=us-ascii;
> name="ggrosshans.vcf"
>Content-Transfer-Encoding: 7bit
>Content-Description: Card for Gregory Grosshans
>Content-Disposition: attachment;
> filename="ggrosshans.vcf"
>
>begin:vcard 
>n:Grosshans;Gregory
>tel;fax:405-579-0700
>tel;work:405-579-0720
>x-mozilla-html:FALSE
>org:DOC/NOAA/NWS/NCEP
>adr:;;Storm Prediction Center;Norman;OK;73069;
>version:2.1
>email;internet:address@hidden
>title:Software Analyst
>x-mozilla-cpt:;-9456
>fn:Gregg Grosshans
>end:vcard
>
>--------------E90767404A79D2F64D4F5ADB--
>