This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.
On Wed, 20 Sep 2000, Michael W Dross wrote: > > > Rob, > > I configured the ldm's pqing to use the -5 flag in the ldmd.conf file. This > system is feeding the WMO feedtype via socket from our NOAAPORT machine. > Its not feeding WMO data from any other sources. > > What I have noticed is it appears that the pqing is flagging products as " > Product already in queue" when it really isn't. It appears to be > ignoring the bulletin size, or at least it would appear. I have attached a > couple of hours of the pqing -v log. The bulletin I noticed most > often flagged is the "SXUS70 KWAL" since it comes across often. In particular > SXUS70 KWAL 200437 was > flagged and could find another one prior that was with the same time and/or > size > (604 bytes). > > Take a look and let me know what you think. Maybe I am not understanding > something... > Mike, Even though you are getting products from a single source there will still be duplicates. The NWS resends products all the time. Now with the -5 flag the difference of sequence numbers, ie 259 and 260 doesn't matter. I checked the log, the SNPS40 product was flagged as a duplicate but it's size is 97 bytes so it's a duplicate. I checked about 10 products it was the same scenario. @Sep 20 05:16:19 pqing[26561]: 97 20000920051619.499 WMO 259 SNPS40 AMMC 200400 RRF ^@Sep 20 05:16:19 pqing[26561]: 97 20000920051619.504 WMO 260 SNPS40 AMMC 200400 RRF ^@Sep 20 05:16:19 pqing[26561]: Product already in queue I did a test with pqing by using the same rawfile twice and it flagged all the duplicates correctly: pqing -vl - -q data/ldm.pq -f HDS -5 <ingest source or rawfile> You can also use pqcat to dump the LDM queue if you are thinking that a product is missing. Look at the pqcat man page. At this point I think the -5 flag is working correctly. Robb... > Thanks, > > (See attached file: pqing.log) > > Mike > > > |--------+--------------------------> > | | Robb Kambic | > | | <rkambic@unidata| > | | .ucar.edu> | > | | | > | | 09/18/2000 12:07| > | | PM | > | | | > |--------+--------------------------> > > >----------------------------------------------------------------------------| > | > | > | To: Michael W Dross/Cust/DukePower@DukePower > | > | cc: address@hidden, support-ldm | > | <address@hidden> | > | bcc: > | > | Subject: Re: Sequence number use in pqact.conf?? > | > > >----------------------------------------------------------------------------| > > > > > > > On Sun, 17 Sep 2000, Michael W Dross wrote: > > > > > > > I have searched and could not find the answer... So I turn to you guys :-) > > > > Is there a way in defining a pqact.conf entry that will use the sequence > > number?? I have tried with no luck. Consider the following example. > > > > > > DDPLUS ^WHXX04 (....) ([0-3][0-9])([0-2][0-9])([0-5][0-9]) > > FILE -overwrite > > data/tropical/GFDL_Hurricane_Guidance.(\2:mm)\2\3\4 > > > Mike, > > I looked at the pqact code about the feedtype and pattern matches, here's > a comment that states the sequence number is NOT included in the match. > > * If the feedtype matches AND ((the product ID matches the > * regular expression) OR (the pattern is "_ELSE_" AND nothing > * has been done to this product yet AND the first char of > * the ident isn't '_')) > > This is a problem because products coming from different NOAAport ingest > systems will look differnt because of the different sequence number. In > the ldm-5.1.2 release we added a -5 flag to pqing to NOT include the > sequence number in the md5 checksum. Therefore, the checksum will just be > on the data section of the product and it can designate these type of > products as duplicates. At this time, we have not implemented this > because it needs some synchronization between all the NOAAport ingest > source site. I suspect it will happen in the next couple of months. > > Robb... > > > > > > > Occasionally multiple bulletins with the same time/date heading are sent and > it > > gets appended to the file (-overwrite doesn't prevent it > > for some reason) the only difference is the sequence number. I thought this > > might work, but doesn't. > > > > DDPLUS ^(...) WHXX04 (....) ([0-3][0-9])([0-2][0-9])([0-5][0-9]) > > FILE -overwrite > data/tropical/GFDL_Hurricane_Guidance.\1(\3:mm)\3\4\5 > > > > > > It seems the entry starts with WHXX04 and the sequence number cannot be > > referenced?? > > > > Any ideas/suggestions would be greatly appreciated! > > > > Mike Dross > > > > > > =============================================================================== > Robb Kambic Unidata Program Center > Software Engineer III Univ. Corp for Atmospheric Research > address@hidden WWW: http://www.unidata.ucar.edu/ > =============================================================================== > > > > =============================================================================== Robb Kambic Unidata Program Center Software Engineer III Univ. Corp for Atmospheric Research address@hidden WWW: http://www.unidata.ucar.edu/ ===============================================================================