[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[IDD #BJM-438879]: Some Confusing regarding LDM pqact
- Subject: [IDD #BJM-438879]: Some Confusing regarding LDM pqact
- Date: Wed, 06 Jan 2016 13:22:24 -0700
Hi Ziv,
re:
> I have a ldm feed set up, and I want to turn my NEXRAD2 data products into
> files that have the following name format:
>
> LLLL_YYYY_MM_DD_HH_MM_SS
>
> where:
> L = location character
> Y = year character
> M = month character
> D = day character
> H = hour character
> M = month character
> S = second character
OK, but first you have to realize that the NEXRAD2 datastream products are
pieces
of full volume scans that need to be reconstituted back into full volume scans
on one's local machine (sending out the volume scans in pieces was an NWS
decision,
not ours).
> I have been reading the pqact documentation
> <http://www.unidata.ucar.edu/software/ldm/ldm-current/basics/pqact.conf.html#prodIdPat>
> closely,
> and I am *starting* to understand it. However, I cannot find what the
> NEXRAD2 header format is.
>
> - Can give me the NEXRAD2 header format?
Are you asking for the format of the products themselves, or are you asking
for the ProductID format for the volume scan pieces that constitute the
NEXRAD2 datastream? I think you are asking for the latter not the former.
If I am correct, then you can see the ProductIDs for products distributed
in the IDD easily using the LDM utility 'notifyme'. Here is a simple example:
<as 'ldm'>
notifyme -vl- -f NEXRAD2
This will list out information on NEXRAD2 products that are being received
by one's LDM.
'notifyme' is great for doing a sanity check for what is being received
and what is available. The sanity check in this case is comparing what
the upstream LDM has with what is being received:
<as 'ldm'>
notifyme -vl- -f NEXRAD2 -h <the upstream IDD host you want to interrogate>
notifyme -vl- -f NEXRAD2
Again, the second invocation shows what your LDM is receiving. The first
invocation will return what the indicated upstream host is receiving
** as long as your LDM is ALLOWed to REQUEST data from that machine **.
Here is a small snippit from a 'notifyme' invocation that shows what the
indicated upstream is receiving, and the ProductIDs for those products:
$ notifyme -vl- -f NEXRAD2 -h idd.unidata.ucar.edu
Jan 06 19:25:48 notifyme[17011] NOTE: Starting Up: idd.unidata.ucar.edu:
20160106192548.866 TS_ENDT {{NEXRAD2, ".*"}}
Jan 06 19:25:48 notifyme[17011] NOTE: LDM-5 desired product-class:
20160106192548.866 TS_ENDT {{NEXRAD2, ".*"}}
Jan 06 19:25:48 notifyme[17011] INFO: Resolving idd.unidata.ucar.edu to
128.117.140.3 took 0.000264 seconds
Jan 06 19:25:48 notifyme[17011] NOTE: NOTIFYME(idd.unidata.ucar.edu): OK
Jan 06 19:25:49 notifyme[17011] INFO: 21643 20160106192548.935 NEXRAD2
393034 L2-BZIP2/KGSP/20160106191612/393/34/E/V06/0
Jan 06 19:25:49 notifyme[17011] INFO: 12543 20160106192548.896 NEXRAD2
264010 L2-BZIP2/KLRX/20160106192505/264/10/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO: 19438 20160106192548.991 NEXRAD2
807008 L2-BZIP2/KESX/20160106192510/807/8/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO: 19129 20160106192549.016 NEXRAD2
649012 L2-BZIP2/KOKX/20160106192328/649/12/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO: 8536 20160106192549.197 NEXRAD2
396025 L2-BZIP2/KOHX/20160106192042/396/25/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO: 55068 20160106192549.130 NEXRAD2
710026 L2-BZIP2/KMAF/20160106192328/710/26/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO: 35998 20160106192549.261 NEXRAD2 38044
L2-BZIP2/KBYX/20160106192336/38/44/I/V06/0
Jan 06 19:25:49 notifyme[17011] INFO: 49516 20160106192549.511 NEXRAD2
144007 L2-BZIP2/KICT/20160106192436/144/7/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 31618 20160106192549.366 NEXRAD2
313055 L2-BZIP2/KDYX/20160106192300/313/55/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 48738 20160106192549.385 NEXRAD2
718045 L2-BZIP2/KEPZ/20160106192342/718/45/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 45417 20160106192548.954 NEXRAD2 16019
L2-BZIP2/KHTX/20160106192215/16/19/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 9292 20160106192550.261 NEXRAD2
241022 L2-BZIP2/KILN/20160106192125/241/22/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 91018 20160106192549.896 NEXRAD2
308027 L2-BZIP2/KARX/20160106192033/308/27/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 66000 20160106192550.055 NEXRAD2
147031 L2-BZIP2/KEWX/20160106191806/147/31/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 14244 20160106192550.362 NEXRAD2
987001 L2-BZIP2/KMUX/20160106192549/987/1/S/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 13664 20160106192550.044 NEXRAD2
249019 L2-BZIP2/PAPD/20160106192146/249/19/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 75079 20160106192549.837 NEXRAD2
157008 L2-BZIP2/KFDR/20160106192527/157/8/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 10369 20160106192550.706 NEXRAD2
622024 L2-BZIP2/KAKQ/20160106192058/622/24/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 34552 20160106192550.319 NEXRAD2
200015 L2-BZIP2/KMXX/20160106192253/200/15/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 12242 20160106192550.747 NEXRAD2
299022 L2-BZIP2/KMKX/20160106192357/299/22/I/V06/0
Jan 06 19:25:50 notifyme[17011] INFO: 24000 20160106192550.710 NEXRAD2
293004 L2-BZIP2/KJKL/20160106192513/293/4/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 21033 20160106192550.828 NEXRAD2
771040 L2-BZIP2/KJAX/20160106192100/771/40/E/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 102695 20160106192548.889 NEXRAD2
936060 L2-BZIP2/KDAX/20160106192243/936/60/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 154574 20160106192550.217 NEXRAD2 78055
L2-BZIP2/KHGX/20160106192233/78/55/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 49969 20160106192550.592 NEXRAD2
377053 L2-BZIP2/KSJT/20160106192303/377/53/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 75103 20160106192550.348 NEXRAD2
573031 L2-BZIP2/KPOE/20160106192410/573/31/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 6412 20160106192550.943 NEXRAD2
544001 L2-BZIP2/KDFX/20160106192550/544/1/S/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 10957 20160106192550.079 NEXRAD2
995034 L2-BZIP2/PAKC/20160106191621/995/34/E/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 32463 20160106192550.904 NEXRAD2
111020 L2-BZIP2/KEMX/20160106192451/111/20/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 37733 20160106192550.930 NEXRAD2
829059 L2-BZIP2/KNKX/20160106192231/829/59/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 183395 20160106192550.260 NEXRAD2
447027 L2-BZIP2/KRGX/20160106192437/447/27/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 24984 20160106192551.093 NEXRAD2
776052 L2-BZIP2/KYUX/20160106192239/776/52/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 37367 20160106192551.045 NEXRAD2
625022 L2-BZIP2/KMAX/20160106192124/625/22/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 23613 20160106192551.294 NEXRAD2
376034 L2-BZIP2/KLCH/20160106192410/376/34/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 251208 20160106192550.182 NEXRAD2
117038 L2-BZIP2/KVTX/20160106192406/117/38/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 38806 20160106192551.293 NEXRAD2
359022 L2-BZIP2/KDVN/20160106192447/359/22/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 13586 20160106192551.415 NEXRAD2
623024 L2-BZIP2/KCCX/20160106192115/623/24/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 11103 20160106192551.535 NEXRAD2
542020 L2-BZIP2/KLTX/20160106192408/542/20/I/V06/0
Jan 06 19:25:51 notifyme[17011] INFO: 55954 20160106192550.559 NEXRAD2
371069 L2-BZIP2/KVBX/20160106192143/371/69/I/V06/0
^CJan 06 19:25:51 notifyme[17011] NOTE: exiting
(NB: the ^C is the ctrl-C I issued to stop the 'notifyme' invocation)
The ProductID portion of the output lines above are the fields that begin
with 'L2-BZIP2' and end with '/0'.
Here are our pattern-action file actions that we use to process NEXRAD2 products
into volume scans:
#
# ---------------------------------
# - NEXRAD Level II Radar WSR-88D -
# ---------------------------------
#
# store first piece ie L2-BZIP2/KFTG/20091021190220/88/1/S/V03/0
#
NEXRAD2
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/1/S
FILE -close
/data/ldm/pub/native/radar/level2/\1/\2/.tmp/Level2_\1_\2_\3.ar2v
#
# store internal pieces and end piece
# ie L2-BZIP2/KFTG/20091021190220/88/2/I/V03/0
# ie L2-BZIP2/KFTG/20091021190220/88/70/E/V03/0
#
NEXRAD2
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/([0-9]*)/(I|E)
FILE -flush /data/ldm/pub/native/radar/level2/\1/\2/.tmp/\1_\2_\3
#
# execute script hhmmssRadarII.pl
# -put good products in live directory
# -put missing 1st piece products in badLevel2/piece1missing
# -put maintenance products in badLevel2/maintenance
#
NEXRAD2
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/([0-9]*)/E
EXEC perl etc/TDS/util/hhmmssRadarII.pl \1 \2 \3 \4
Things to note in these actions:
1) certain white space in pattern-action file actions are <tab>s, not <space>s
Writing the first action to show which what space are <tab>s:
NEXRAD2<tab>^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9][0-9][0-9])/[0-9]*/1/S
<tab>FILE<tab>-close<tab>/data/ldm/pub/native/radar/level2/\1/\2/.tmp/Level2_\1_\2_\3.ar2v
2) the actions do the following:
- the first action writes the first piece of a volume scan for a station
into the hidden, temporary directory
'/data/ldm/pub/native/radar/level2/\1/\2/.tmp'
- the second action writes individual internal pieces of a volume scan to
the same
temporary directory
- the third action is run when the final field of the ProductID indicates
that the
piece is the final one of a volume scan (the '/E' at the end of the
ProductID
This action runs a Perl script named 'hhmmssRadarII.pl' that will try to
reassemble
the volume scan pieces back into a full volume scan.
Why is this so complicated?
There is no guarantee that the pieces of a volume scan will be received in
order.
It is _very_ common to receive the "last" piece of a volume scan before one
or
more of the internal pieces are received. Because of this fact, one has to
be
more clever on how to reconstitute the pieces back into a full volume scan.
Our approach is to do it with the three actions that I listed above.
re:
> - Also, can you help me construct an argument (mostly just the *prodIdPat *
> and *pathname*) for a FILE action?
In order to speed things up for you, I am putting a pattern-action file
that have the actions above AND the Perl script that needs to be run on
our FTP server for you to grab. Here are the specifics:
machine: ftp.unidata.ucar.edu
user: anonymous
pass: your_full_email_address
directory: pub/nexrad2
files: pqact.radars
hhmmssRadarII.pl
Here is what to do:
1) create a couple of directories in ~ldm that will likely be useful
<as 'ldm'>
cd ~ldm
mkdir decoders util
2) add the ~ldm/decoders and ~ldm/util directories to the end of the PATH
for user 'ldm'
3) FTP the two files I made available:
- move hhmmssRadarII.pl to ~ldm/util
NB: make sure that hhmmssRadarII.pl is set to be executable
- move pqact.radars to ~ldm/etc
4) edit the LDM configuration file and add the following line in the EXEC
section near the top of the file:
<as 'ldm'>
cd ~ldm
vi etc/ldmd.conf
-- add
EXEC "pqact -f HRS|FNEXRAD|NNEXRAD|NEXRAD2 /home/ldm/etc/pqact.radars"
5) stop and restart your LDM
BUT BEFORE YOU DO THE ABOVE:
- think about where you want the NEXRAD2 data products (pieces and reconstituted
volume scans) to be written
The example pattern-action file 'pqact.radars' actions will use the
/data/ldm/pub/native/radar/level2 directory hierarchy. If this does not fit
with what you want, you will have to modify the actions in 'pqact.radars'.
After you get processing of the NEXRAD2 data setup, you will need to make sure
that you have data scouring turned on (we talked about this previously).
re:
> Thank you,
Please let me know when you have worked through the information above.
Cheers,
Tom
--
****************************************************************************
Unidata User Support UCAR Unidata Program
(303) 497-8642 P.O. Box 3000
address@hidden Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage http://www.unidata.ucar.edu
****************************************************************************
Ticket Details
===================
Ticket ID: BJM-438879
Department: Support IDD
Priority: Normal
Status: Closed