20050408: zlibg2gini decoding of NIMAGE GINI data (cont.)

>From: Celia Chen <address@hidden>
>Organization: NCAR/RAP
>Keywords: 200504061932.j36JWtv2010902 LDM ldm-mcidas zlibg2gini

Hi Celia,

>The gini data decoding stopped working sometime after you helped me setup
>the pqact.conf on rasp. I turned off the CONDUIT data processing this morning 
>but zlibg2gini still didn't happen.  So, I moved gini feed and decoing to 
>level,
>RAL's other ldm host.  gini data has been decoded and saved for a few
>hours now.  
>
>I did switch the level-2 data FILE back to my original setup (64 individual
>files instead of one volume scan file), because our users prefer it that way. 

This is the most likely reason that the decoding for NIMAGE data stopped
working.  The reason I say this is that the action you had for the Level
II data will try to file a HUGE number of products: up to 64 for each radar
for each time it reports.  If every radar was operating in storm mode, this
could result in 124 * 64 * 12 = 95232 products being written each hour
for the NEXRD2 feed alone.  I a willing to bet that your system simply was
not able to keep up with the disk I/O that this would require, so it
processing came to a grinding halt.  The other thing about this is that
the FILE action you were (and are again) using did not close the file after
the piece was received.  At a minimum you should add the -close option
to your NEXRD2 FILE action:

        FILE    -close  ...

Note that the whitespace between the FILE and -close is a tab.

>However, I believe the gini data decoding had already stopped working before I
>did the switch. (The last gini file was decoded at ~21:46z or 3:46pm local
>time.  I talked to a user after 4pm and then modified the pqact.conf file
>after that.)

If GINI processing stopped before your pqact.conf reversion to the original
way of FILEing data, it would mean that there was still too much being requested
of a single invocation of pqact.  This is what I was referring to when I
said that there were no guarantees that more refinement of the split of
pqact processing would not be necessary.

>Speaking of level-2 data, do you (unidata) have yesterday's level-2 data
>from KFTG archived?

No, sorry.

>For some reason I don't have anything from FTG before
>19Z.  One of our users here needs the data - there was an accident at
>Jeffco yesterday around noon. If you don't have it do you know anybody
>around here might have it?   

Gerry Creager <address@hidden> at Texas A&M,
however, does keep 30 days of the Level II data online.  I would
contact him for the data you missed.  His archived Level II data
has all pieces written to a single file, not broken out by quadrant.

>So, as you can see, I have gave up on running zlibg2gini on rasp for now.
>I just hope there is no other serious problems on rasp.  

I think that additional looks at how you are processing the data could
be useful.  In particular, perhaps your users could adjust how they use
the data so that all data for a radar would be written into a single file.

>Thanks so much for your help!

No worries.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.