[IDD #BJM-438879]: Some Confusing regarding LDM pqact

Hi Ziv,

re:
> I have a ldm feed set up, and I want to turn my NEXRAD2 data products into
> files that have the following name format:
> 
> LLLL_YYYY_MM_DD_HH_MM_SS
> 
> where:
> L = location character
> Y = year character
> M = month character
> D = day character
> H = hour character
> M = month character
> S = second character

OK, but first you have to realize that the NEXRAD2 datastream products are 
pieces
of full volume scans that need to be reconstituted back into full volume scans
on one's local machine (sending out the volume scans in pieces was an NWS 
decision,
not ours).


> I have been reading the pqact documentation
> <http://www.unidata.ucar.edu/software/ldm/ldm-current/basics/pqact.conf.html#prodIdPat>
> closely,
> and I am *starting* to understand it. However, I cannot find what the
> NEXRAD2 header format is.
> 
> - Can give me the NEXRAD2 header format?

Are you asking for the format of the products themselves, or are you asking
for the ProductID format for the volume scan pieces that constitute the
NEXRAD2 datastream?  I think you are asking for the latter not the former.
If I am correct, then you can see the ProductIDs for products distributed
in the IDD easily using the LDM utility 'notifyme'.  Here is a simple example:

<as 'ldm'>
notifyme -vl- -f NEXRAD2

This will list out information on NEXRAD2 products that are being received
by one's LDM.

'notifyme' is great for doing a sanity check for what is being received
and what is available.  The sanity check in this case is comparing what
the upstream LDM has with what is being received:

<as 'ldm'>
notifyme -vl- -f NEXRAD2 -h <the upstream IDD host you want to interrogate>

notifyme -vl- -f NEXRAD2

Again, the second invocation shows what your LDM is receiving.  The first
invocation will return what the indicated upstream host is receiving
** as long as your LDM is ALLOWed to REQUEST data from that machine **.

Here is a small snippit from a 'notifyme' invocation that shows what the
indicated upstream is receiving, and the ProductIDs for those products:

$ notifyme -vl- -f NEXRAD2 -h idd.unidata.ucar.edu
Jan 06 19:25:48 notifyme[17011] NOTE: Starting Up: idd.unidata.ucar.edu: 
20160106192548.866 TS_ENDT {{NEXRAD2,  ".*"}}
Jan 06 19:25:48 notifyme[17011] NOTE: LDM-5 desired product-class: 
20160106192548.866 TS_ENDT {{NEXRAD2,  ".*"}}
Jan 06 19:25:48 notifyme[17011] INFO: Resolving idd.unidata.ucar.edu to 
128.117.140.3 took 0.000264 seconds
Jan 06 19:25:48 notifyme[17011] NOTE: NOTIFYME(idd.unidata.ucar.edu): OK
Jan 06 19:25:49 notifyme[17011] INFO:    21643 20160106192548.935 NEXRAD2 
393034  L2-BZIP2/KGSP/20160106191612/393/34/E/V06/0
Jan 06 19:25:49 notifyme[17011] INFO:    12543 20160106192548.896 NEXRAD2 
264010  L2-BZIP2/KLRX/20160106192505/264/10/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO:    19438 20160106192548.991 NEXRAD2 
807008  L2-BZIP2/KESX/20160106192510/807/8/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO:    19129 20160106192549.016 NEXRAD2 
649012  L2-BZIP2/KOKX/20160106192328/649/12/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO:     8536 20160106192549.197 NEXRAD2 
396025  L2-BZIP2/KOHX/20160106192042/396/25/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO:    55068 20160106192549.130 NEXRAD2 
710026  L2-BZIP2/KMAF/20160106192328/710/26/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO:    35998 20160106192549.261 NEXRAD2 38044 
 L2-BZIP2/KBYX/20160106192336/38/44/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO:    49516 20160106192549.511 NEXRAD2 
144007  L2-BZIP2/KICT/20160106192436/144/7/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    31618 20160106192549.366 NEXRAD2 
313055  L2-BZIP2/KDYX/20160106192300/313/55/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    48738 20160106192549.385 NEXRAD2 
718045  L2-BZIP2/KEPZ/20160106192342/718/45/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    45417 20160106192548.954 NEXRAD2 16019 
 L2-BZIP2/KHTX/20160106192215/16/19/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:     9292 20160106192550.261 NEXRAD2 
241022  L2-BZIP2/KILN/20160106192125/241/22/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    91018 20160106192549.896 NEXRAD2 
308027  L2-BZIP2/KARX/20160106192033/308/27/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    66000 20160106192550.055 NEXRAD2 
147031  L2-BZIP2/KEWX/20160106191806/147/31/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    14244 20160106192550.362 NEXRAD2 
987001  L2-BZIP2/KMUX/20160106192549/987/1/S/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    13664 20160106192550.044 NEXRAD2 
249019  L2-BZIP2/PAPD/20160106192146/249/19/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    75079 20160106192549.837 NEXRAD2 
157008  L2-BZIP2/KFDR/20160106192527/157/8/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    10369 20160106192550.706 NEXRAD2 
622024  L2-BZIP2/KAKQ/20160106192058/622/24/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    34552 20160106192550.319 NEXRAD2 
200015  L2-BZIP2/KMXX/20160106192253/200/15/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    12242 20160106192550.747 NEXRAD2 
299022  L2-BZIP2/KMKX/20160106192357/299/22/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO:    24000 20160106192550.710 NEXRAD2 
293004  L2-BZIP2/KJKL/20160106192513/293/4/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    21033 20160106192550.828 NEXRAD2 
771040  L2-BZIP2/KJAX/20160106192100/771/40/E/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:   102695 20160106192548.889 NEXRAD2 
936060  L2-BZIP2/KDAX/20160106192243/936/60/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:   154574 20160106192550.217 NEXRAD2 78055 
 L2-BZIP2/KHGX/20160106192233/78/55/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    49969 20160106192550.592 NEXRAD2 
377053  L2-BZIP2/KSJT/20160106192303/377/53/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    75103 20160106192550.348 NEXRAD2 
573031  L2-BZIP2/KPOE/20160106192410/573/31/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:     6412 20160106192550.943 NEXRAD2 
544001  L2-BZIP2/KDFX/20160106192550/544/1/S/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    10957 20160106192550.079 NEXRAD2 
995034  L2-BZIP2/PAKC/20160106191621/995/34/E/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    32463 20160106192550.904 NEXRAD2 
111020  L2-BZIP2/KEMX/20160106192451/111/20/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    37733 20160106192550.930 NEXRAD2 
829059  L2-BZIP2/KNKX/20160106192231/829/59/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:   183395 20160106192550.260 NEXRAD2 
447027  L2-BZIP2/KRGX/20160106192437/447/27/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    24984 20160106192551.093 NEXRAD2 
776052  L2-BZIP2/KYUX/20160106192239/776/52/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    37367 20160106192551.045 NEXRAD2 
625022  L2-BZIP2/KMAX/20160106192124/625/22/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    23613 20160106192551.294 NEXRAD2 
376034  L2-BZIP2/KLCH/20160106192410/376/34/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:   251208 20160106192550.182 NEXRAD2 
117038  L2-BZIP2/KVTX/20160106192406/117/38/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    38806 20160106192551.293 NEXRAD2 
359022  L2-BZIP2/KDVN/20160106192447/359/22/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    13586 20160106192551.415 NEXRAD2 
623024  L2-BZIP2/KCCX/20160106192115/623/24/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    11103 20160106192551.535 NEXRAD2 
542020  L2-BZIP2/KLTX/20160106192408/542/20/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO:    55954 20160106192550.559 NEXRAD2 
371069  L2-BZIP2/KVBX/20160106192143/371/69/I/V06/0
^CJan 06 19:25:51 notifyme[17011] NOTE: exiting

(NB: the ^C is the ctrl-C I issued to stop the 'notifyme' invocation)

The ProductID portion of the output lines above are the fields that begin
with 'L2-BZIP2' and end with '/0'.

Here are our pattern-action file actions that we use to process NEXRAD2 products
into volume scans:

#
# ---------------------------------
# - NEXRAD Level II Radar WSR-88D -
# ---------------------------------
#
# store first piece ie L2-BZIP2/KFTG/20091021190220/88/1/S/V03/0
#
NEXRAD2   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/1/S
        FILE    -close  
/data/ldm/pub/native/radar/level2/\1/\2/.tmp/Level2_\1_\2_\3.ar2v
#
# store internal pieces and end piece
# ie L2-BZIP2/KFTG/20091021190220/88/2/I/V03/0
# ie L2-BZIP2/KFTG/20091021190220/88/70/E/V03/0
#
NEXRAD2   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/([0-9]*)/(I|E)
        FILE    -flush  /data/ldm/pub/native/radar/level2/\1/\2/.tmp/\1_\2_\3
#
# execute script hhmmssRadarII.pl
#  -put good products in live directory
#  -put missing 1st piece products in badLevel2/piece1missing
#  -put maintenance products in badLevel2/maintenance
#
NEXRAD2   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/([0-9]*)/E
        EXEC    perl etc/TDS/util/hhmmssRadarII.pl \1 \2 \3 \4

Things to note in these actions:

1) certain white space in pattern-action file actions are <tab>s, not <space>s

   Writing the first action to show which what space are <tab>s:

NEXRAD2<tab>^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/1/S
<tab>FILE<tab>-close<tab>/data/ldm/pub/native/radar/level2/\1/\2/.tmp/Level2_\1_\2_\3.ar2v

2) the actions do the following:

   - the first action writes the first piece of a volume scan for a station
     into the hidden, temporary directory 
'/data/ldm/pub/native/radar/level2/\1/\2/.tmp'

   - the second action writes individual internal pieces of a volume scan to 
the same
     temporary directory

   - the third action is run when the final field of the ProductID indicates 
that the
     piece is the final one of a volume scan (the '/E' at the end of the 
ProductID

     This action runs a Perl script named 'hhmmssRadarII.pl' that will try to 
reassemble
     the volume scan pieces back into a full volume scan.

   Why is this so complicated?

   There is no guarantee that the pieces of a volume scan will be received in 
order.
   It is _very_ common to receive the "last" piece of a volume scan before one 
or
   more of the internal pieces are received.  Because of this fact, one has to 
be
   more clever on how to reconstitute the pieces back into a full volume scan.
   Our approach is to do it with the three actions that I listed above.

re:
> - Also, can you help me construct an argument (mostly just the *prodIdPat *
> and *pathname*) for a FILE action?

In order to speed things up for you, I am putting a pattern-action file
that have the actions above AND the Perl script that needs to be run on
our FTP server for you to grab.  Here are the specifics:

machine:   ftp.unidata.ucar.edu
user:      anonymous
pass:      your_full_email_address
directory: pub/nexrad2
files:     pqact.radars
           hhmmssRadarII.pl

Here is what to do:

1) create a couple of directories in ~ldm that will likely be useful

   <as 'ldm'>
   cd ~ldm
   mkdir decoders util

2) add the ~ldm/decoders and ~ldm/util directories to the end of the PATH
   for user 'ldm'

3) FTP the two files I made available:

   - move hhmmssRadarII.pl to ~ldm/util

     NB: make sure that hhmmssRadarII.pl is set to be executable

   - move pqact.radars to ~ldm/etc

4) edit the LDM configuration file and add the following line in the EXEC
   section near the top of the file:

   <as 'ldm'>
   cd ~ldm
   vi etc/ldmd.conf

   -- add
EXEC    "pqact -f HRS|FNEXRAD|NNEXRAD|NEXRAD2 /home/ldm/etc/pqact.radars"

5) stop and restart your LDM

BUT BEFORE YOU DO THE ABOVE:

- think about where you want the NEXRAD2 data products (pieces and reconstituted
  volume scans) to be written

  The example pattern-action file 'pqact.radars' actions will use the
  /data/ldm/pub/native/radar/level2 directory hierarchy.  If this does not fit
  with what you want, you will have to modify the actions in 'pqact.radars'.

After you get processing of the NEXRAD2 data setup, you will need to make sure
that you have data scouring turned on (we talked about this previously).

re:
> Thank you,

Please let me know when you have worked through the information above.


Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: BJM-438879
Department: Support IDD
Priority: Normal
Status: Closed