20040621: bigbird and KAMA Level II data?

>From:  Gerry Creager N5JXS <address@hidden>
>Organization:  Texas A&M University -- AATLT
>Keywords:  200406220437.i5M4beWb016431

Hi Gerry,

>I'd been seeing KAMA LII data for a while, but I don't even see a 
>subdirectory for it... suggesting it's been scoured out of existence (Of 
>course, just to confuse things, I just created one to see if there's 
>something unhappy in pqact, which I currently doubt).

Hmm...  I just logged onto bigbird and do see subdirectories for KAMA
for most days:

cd /data/ldm/gempak/nexrad/craft_all
ls */KAMA
20040608/KAMA:
20040609/KAMA:
20040610/KAMA:
20040611/KAMA:
20040612/KAMA:
20040613/KAMA:
20040614/KAMA:
20040615/KAMA:
20040616/KAMA:
20040617/KAMA:
20040618/KAMA:
20040619/KAMA:
20040620/KAMA:
20040621/KAMA:

However, a simple 'ls' does not show any files.  Closer inspection shows
that each subdirectory has all of the KAMA data files, but they are
all named with a '.' as the first character (i.e., hidden):

[ldm@bigbird 20040608]$ cd KAMA
[ldm@bigbird KAMA]$ ls
[ldm@bigbird KAMA]$ ls -alt
total 111384
-rw-rw-r--    1 ldm ldm 538043 Jun  8 19:06 .KAMA_20040608_2359
-rw-rw-r--    1 ldm ldm 544964 Jun  8 18:59 .KAMA_20040608_2353
-rw-rw-r--    1 ldm ldm 553216 Jun  8 18:53 .KAMA_20040608_2346
-rw-rw-r--    1 ldm ldm 557738 Jun  8 18:50 .KAMA_20040608_2340
-rw-rw-r--    1 ldm ldm 558830 Jun  8 18:42 .KAMA_20040608_2335

What this tells me is that the header of the last piece of the volume
scan does not have the indicator of being the last piece.  The
GEMPAK action for filing the products counts on the last piece of
the volume scan being marked as being the last piece:

cat ~ldm/etc/pqact.gempak_craft.tamu

# CRAFT stored as raw bz2 for GEMPAK
#
# file the raw data to a temporary file beginning with "." so that autoupdate 
GUIs don't
# get ugly partial volume plots
CRAFT   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9][0-9])
        FILE    data/gempak/nexrad/craft_all/\2/\1/.\1_\2_\3
#
# Done to move file after last record is received "/E" to prevent
# autoupdate from seeing partially received files (dccraft_move is a shell 
script copied from $NAWIPS/bin/scripts)
CRAFT   
^L2-BZIP2/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9])([0-2][0-9][0-5][0-9])([0-9][0-9]).*/E$
        EXEC    util/dccraft_move data/gempak/nexrad/craft_all/\2/\1/.\1_\2_\3 
data/gempak/nexrad/craft/\2/\1/\1_\2_\3

Notice that the second action, the one that renames the file .KMA... to
KAM...  relies on there being a '/E' in the product header.  If this is
missing, the file will never get renamed.  Given that all KAMA files
renamed as .KAMA..., we would have to conclude that the last piece
is always missing the termination indicator.

>Do you know if KAMA (Amarillo, TX) is off the air for LII data?

It isn't. On still closer inspection, none of the Level II files
are getting renamed.  After seeing this, I decided to see if any of
the products did, in fact, have the last piece indicator:

notifyme -vxl- -f CRAFT -o 3600
 ...
Jun 22 17:38:00 notifyme[19412]: c93ea14171852194788c89659da37330     6933 
20040622163811.090   CRAFT 145027  L2-BZIP2/KICX/20040622162852/145/27/E
 ...

But I don't see any renamed files in
/data/ldm/gempak/nexrad/craft_all/20040622/KICX.  This tells me that
the pattern included in the GEMPAK action file is incorrect.

Now, notifyme can be used to see all of the products whose headers match
a user-specified pattern:

notifyme -vxl- -f CRAFT -o 3600 -p "/E"
Jun 22 17:41:45 notifyme[21989]: Starting Up: localhost: 20040622164145.206 
TS_ENDT {{CRAFT,  "/E"}}
Jun 22 17:41:45 notifyme[21989]: Connected to upstream LDM-5
        NOTIFYME(localhost) returns OK
Jun 22 17:41:45 notifyme[21989]: NOTIFYME(localhost): OK
Jun 22 17:41:45 notifyme[21989]: e4eedf0077e81fb3cc9ced4eb4b8343b     8120 
20040622164308.001   CRAFT 135027  L2-BZIP2/KESX/20040622163242/135/27/E
Jun 22 17:41:45 notifyme[21989]: 88ca42c83c1b334e8f961e725ddb490e    11755 
20040622164155.943   CRAFT 190067  L2-BZIP2/KLZK/20040622163816/190/67/E
Jun 22 17:41:45 notifyme[21989]: b67a8e013a04f8c9edaa3cafd5d0272c     7558 
20040622164200.088   CRAFT 156043  L2-BZIP2/KGYX/20040622163609/156/43/E
 ...

As you can, you are getting lots -- and probably all -- of the last pieces of
volume scans.  So, the question is why the pqact.gempak_craft.tamu action
is failing?

<a few minutes later>  Found it!  I apparently did not change 'craft'
to 'craft_all' in the second action in 'pqact.gempak_craft.tamu', so
the action was incorrect.  My bad!!  I just changed the file and sent
a HUP to all pqacts, and now the products are being renamed after the
last piece is received.

Given that things are now working correctly, I went to each subdirectory
under /data/ldm/gempak/nexrad/craft_all and renamed the files from .K...
to K...  Sorry for the mistake!

>LIII 
>data's still flowing but I'm trying to get a service up for LII displays 
>and that's a bit troubling.

Sorry...

>Thanks, Gerry

Tom

>From address@hidden  Wed Jun 23 09:08:30 2004

Tom,

Cool.  Thanks for the excellent tutorial on notifyme (I learned 
something, AGAIN!) and for finding the problem.

We're getting data again!  Interesting piece of the gotcha:  I was 
getting a feed from bigbird (to bigfoot; we're experimenting with 
machines doing some specific tasks and bigfoot is doing all the L2 and 
L3 crunching for images right now) and was not seeing it for a while. 
I'd have thought, since I was seeing _some_ days, but not all or others, 
that the decoding on bigfoot was OK.  I'll examine it, now and see if 
it's confused, too.

You might want to look at http://mesonet.tamu.edu/LevelII.html

Just slammed it together and I'm soliciting comments.

Off to Austin to get back in the process of getting real funding for 
Mesonet.

Thanks, Gerry