Re: pqcat attempt

andrew farnsworth wrote:
> 
> Hi again
> I did some checking on when we stopped getting the kind of temporal
> resolution in our data that we want, and it stopped on December 1st. Since
> that point we have only received hourly data, rather than the specified
> times of every frame. Perhaps that will help in locating the bug. . .
> 
> Best,
> Andrew


Hi Andrew,

First, are you sure that you're not filing the products that you think
are missing?  In looking at your pqact.conf, it looks as if you are
filing both the hourly and the nonhourly products under
/usr/local/ldm/data/nexrad/<mmdd>/<stn>r0/<mmddhh><something>.  Perhaps
you're thinking that all those products are hourly products when there
are other products mixed in.  Can you confirm that this is not the case?

Second, have you looked at the logs to see if there are any error
messages in them?  (Try 'ldmadmin log' or cd to ~ldm/logs and do 'more
ldmd.log'.)

Next, let me tell you about the results of my experiment.

I assume you're aware that the product ID for these products includes a
timestamp in the form of yyyymmddhhmm.  (This timestamp appears to be Z
time.)  So, for example, the pattern
..([0-9][0-9])(03)(0[0-9])(23|00)([0-6][0-9]) in your pqact.conf file
would match the string 200103002359.  (Do you have any experience with
regular expressions?)

I lifted some entries from your pqact.conf file and put them in my own
pqact.conf to see what happens.  First I lifted only the entries that
would match products having a March time stamp.  Here are the time
portions of these patterns:

..([0-9][0-9])(03|04|05|08|09|10)([0-3][0-9])(22|23)([0-6][0-9])
..([0-9][0-9])(03)(0[0-9])(22|23)([0-6][0-9])
..([0-9][0-9])(03)(0[0-9])(23|00)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(22|23)([0-6][0-9])
..([0-9][0-9])(03)(0[0-9])(00|01)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(00|01)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(01|02)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(01|02)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(02|03)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(00|01)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(00|01)([0-6][0-9])
..([0-9][0-9])(0[3-5])([0-3][0-9])(0[4-7])([0-6][0-9])

This produced no new files.  The reason for that became quickly apparent
- these patterns only match certain hours of the day.  So, the original
pattern, above,  ..([0-9][0-9])(03)(0[0-9])(23|00)([0-6][0-9]), will
only match products in March on days 1 through 9, hours 23 or 00.  Thus,
as you can see in the patterns above, only products having time stamps
of hour 22, 23, 00, 01, 02, or 03 would possibly match any of these
patterns.  (We are just about to start 22Z as I write this message.)

So I modified these entries to allow products having any hour time
stamp, e.g., the patterns became:

..([0-9][0-9])(03|04|05|08|09|10)([0-3][0-9])(..)([0-6][0-9])
..([0-9][0-9])(03)(0[0-9])(..)([0-6][0-9])
..([0-9][0-9])(03)(0[0-9])(..)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
..([0-9][0-9])(03)(0[0-9])(..)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
..([0-9][0-9])(0[3-5])([0-3][0-9])(..)([0-6][0-9])

Immediately products were being filed.  

(For your reference, the complete entries I used in my pqact.conf are at
the end of this message.  You'll see that I added strings "one, two,
..., twelve" to the paths just so I could see which entry generated
which files in my directory.)

This test tells me that these products are on the stream that we are
getting here at Unidata, and that these pqact.conf entries work.

From the pqcat output you sent me I was unable to verify whether or not
at that time you had received any of the products you are trying to
file.  Trying match those products IDs against every entry in your
pqact.conf file would be an impossible task - your pqact.conf is just
too big.  

At this point I suspect that either the products are being filed and
you've just missed them, or you are not receiving the products you are
trying to file.  Perhaps WSI is not transmitting them for some reason. 
I would strongly suspect this if things were working fine as of Dec 1,
then they stopped working but no LDM configuration files were changed. 
Have you asked WSI about this?

But, I would first try to confirm that you're not getting products at
your site.  I suggest moving your existing pqact.conf file to some other
name for the time being, and starting with a smaller pqact.conf file
that you can play with.  Start with just the hourly entries and make
sure that is working.  Then add in one March entry at a time, ensuring
each entry is working before adding another.  To start, don't limit the
hours - instead use the '..' pattern which will match any hour.  When
that's working, then you can put the limited hours back into the entry. 
At that point you'll know that if nothing is written it's because
nothing was received and not because something was wrong with the entry.

Keep in mind that all the fields in a pqact.conf entry must be separated
by tabs, not spaces.  And, if you break the entry into multiple lines,
each line must start with a tab, not spaces.  These invisible bugs can
drive people insane.

After you make a change to pqact.conf do 'ldmadmin pqactHUP' to have the
LDM reread the configuation file.  Then do 'ldmadmin tail' to view the
end of the log and ensure that the conf file was reread and there were
no problems.  If there was a syntax error in the file it should appear
in the log after the message "ReReading configuration file
/usr/local/ldm/etc/pqact.conf".

I hope this helps!  Please let me know if you have further questions. 
This can be complicated and confusing when you're first starting, and
you have a particularly complex pqact configuration to maintain.

Anne
-- 
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://www.unidata.ucar.edu/
****************************************************


# MARCH APRIL MAY AUGUST SEPTEMBER OCTOBER HOTSPOTS
WSI
^NEX/(JUA)/BREF1/..([0-9][0-9])(03|04|05|08|09|10)([0-3][0-9])(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/one/\3\4/\1r0/\3\4\5\6.nid

WSI
^NEX/(OKX|BOX|GYX|ENX|CXX|CBW)/BREF1/..([0-9][0-9])(03)(0[0-9])(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/two/\3\4/\1r0/\3\4\5\6.nid

WSI
^NEX/(BUF|DOX|DIX|CLX|LWX|CCX|MLB|AMX|BYX|JAX|VAX|JGX|MRX|RAX|DTX|GSP|JKL|ILN|AKQ|CLE|RLX|FCX|FFC|CAE|LTX|TBW|MHX|TLH|PBZ|BGM|TYX)/BREF1/..([0-9][0-9])(03)(0[0-9])(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/three/\3\4/\1r0/\3\4\5\6.nid

WSI ^NEX/(CBW)/BREF1/..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/four/\3\4/\1r0/\3\4\5\6.nid

WSI
^NEX/(DVN|DLH|ARX|MPX|LSX|NQA|LCH|POE|SHV|JAN|LZK|SGF|SRX|EAX|TWX|DMX|OAX|HGX|INX|LIX)/BREF1/..([0-9][0-9])(03)(0[0-9])(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/five/\3\4/\1r0/\3\4\5\6.nid

WSI
^NEX/(GRK|FDR|FWS|ICT|ABR|FSD|EWX|TLX|VNX|DFX|DYX|UEX|MVX|CRP|BRO)/BREF1/..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/six/\3\4/\1r0/\3\4\5\6.nid


WSI
^NEX/(BIS|LNX|GLD|MBX|AMA|LBB|SJT|UDX|CYS|FTG|PUX|MAF|GGW|RIW|BLX|FDX|EPZ|ABX|GJX|HDX|DDC)/BREF1/..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/seven/\3\4/\1r0/\3\4\5\6.nid

WSI
^NEX/(EMX|FSX|YUX|ICX|MTX|SFX|IWA|ESX|TFX)/BREF1/..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/eight/\3\4/\1r0/\3\4\5\6.nid

WSI
^NEX/(LRX|CBX|MSX|NKX|OTX|PDT|RGX|EYX|SOX|ATX|BHX|BBX|DAX|MUX|HNX|VBX|MAX|VTX|RTX)/BREF1/..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/nine/\3\4/\1r0/\3\4\5\6.nid

WSI
^NEX/(GRB|MKX|LOT|ILX|HPX|MOB|GWX|PAH|DVN|DLH|ARX|MQT|MPX|LSX|NQA|LCH|POE|SHV|JAN|LZK|SGF|SRX|EAX|TWX|DMX|OAX|HGX|INX|GRR|HTX|LVX|IWX|EVX|EOX|BMX|APX|IND|MXX|LIX)/BREF1/..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/ten/\3\4/\1r0/\3\4\5\6.nid


WSI
^NEX/(GRK|FDR|FWS|ICT|ABR|FSD|EWX|TLX|VNX|DFX|DYX|UEX|MVX|CRP|BRO)/BREF1/..([0-9][0-9])(03)(2[0-9]|30|31)(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/eleven/\3\4/\1r0/\3\4\5\6.nid

WSI
^NEX/(AHG|APD|AIH)/BREF1/..([0-9][0-9])(0[3-5])([0-3][0-9])(..)([0-6][0-9])
    FILE    /usr/local/ldm/data/nexrad/twelve/\3\4/\1r0/\3\4\5\6.nid