20040625: Perl redirect problems (cont.)

>From: Harry Edmon <address@hidden>
>Organization: University of Washington
>Keywords: 200406241954.i5OJsCWb010248 LDM PIPE Perl

Hi Harry,

>In your last messages you said:
>
>>OK.  My comment above would use this pair of actions if png2gif.pl
>>was modified to read the file /home/data/gempak/nport/RADAR/\1/\2/\2_\3_\4
>>produced by the previous action instead of trying to write that file
>>to disk and then process it.

Yes.

>I assume you would do this with a FILE and an EXEC action

Yes, exactly.

>- but how to you
>guarantee the EXEC happens after the file is completely written?

If an action reads the file created by a previous action, you could get
into a situation where the data destined for the file has not been
flushed to disk.  This possibility may need to be factored into the
processing action kicked off by EXEC (sleep?), but, then again, it may
not.  I talked with Russ about this, and he strongly suspects that data
that hasn't been flushed from OS mbufs would be included in a read by a
different process, so it should not matter that the OS hasn't finised
the flush.  This could well be OS-specific, however.

My instinct tells me that it may be important to specify the '-close'
flag to FILE so that the LDM buffer, at least, would be flushed, but
Russ feels that would likely decrease the number of buffers being used
and possibly decrease the performance on your system AND may not
be needed anyway.

>From address@hidden  Fri Jun 25 12:19:49 2004

>I am setting up a new ldm server.  What is your current "wisdom" on multiple
>pqact on the system - it is a dual Xeon server?

We are absolutely in favor for running multiple pqacts.  To prove this,
here is the recommendation ldmd.conf execs that Chiz is sending out
with his latest GEMPAK distribution:

exec "pqact -f CRAFT -p BZIP2/K[A-D] /local/ldm/etc/GEMPAK/pqact.gempak_craft"
exec "pqact -f CRAFT -p BZIP2/K[E-K] /local/ldm/etc/GEMPAK/pqact.gempak_craft"
exec "pqact -f CRAFT -p BZIP2/K[L-R] /local/ldm/etc/GEMPAK/pqact.gempak_craft"
exec "pqact -f CRAFT -p BZIP2/K[S-Z] /local/ldm/etc/GEMPAK/pqact.gempak_craft"

Chiz found that partitioning the CRAFT processing into 4 pqacts allows
us to process all of the CRAFT data with no losses.  One of the
culprits that this helps get around is the current limit of 32 open
file descriptors for each instance of pqact.  By the way, this limit
should go away in a future LDM release.

While you are on the line :-) ...

I have noticed that you are feeding all CONDUIT data to two machines
at UW:
marzipan and freshair:

Jun 15 21:20:04 thelma.ucar.edu marzipan(feed)[14176]: up6.c:331: Starting 
Up(6.0.14/6): 20040615210926.998 TS_ENDT {{CONDUIT,  "[18]$"}}
Jun 15 21:20:04 thelma.ucar.edu marzipan(feed)[14176]: topo:  
marzipan.atmos.washington.edu CONDUIT
Jun 15 21:20:05 thelma.ucar.edu marzipan(feed)[14177]: up6.c:331: Starting 
Up(6.0.14/6): 20040615210927.585 TS_ENDT {{CONDUIT,  "[36]$"}}
Jun 15 21:20:05 thelma.ucar.edu marzipan(feed)[14177]: topo:  
marzipan.atmos.washington.edu CONDUIT
Jun 15 21:20:15 thelma.ucar.edu marzipan(feed)[14209]: up6.c:331: Starting 
Up(6.0.14/6): 20040615210926.921 TS_ENDT {{CONDUIT,  "[09]$"}}
Jun 15 21:20:15 thelma.ucar.edu marzipan(feed)[14209]: topo:  
marzipan.atmos.washington.edu CONDUIT
Jun 15 21:21:50 thelma.ucar.edu marzipan(feed)[14306]: up6.c:331: Starting 
Up(6.0.14/6): 20040615210927.057 TS_ENDT {{CONDUIT,  "[27]$"}}
Jun 15 21:21:50 thelma.ucar.edu marzipan(feed)[14306]: topo:  
marzipan.atmos.washington.edu CONDUIT
Jun 15 21:21:51 thelma.ucar.edu marzipan(feed)[14310]: up6.c:331: Starting 
Up(6.0.14/6): 20040615210927.209 TS_ENDT {{CONDUIT,  "[45]$"}}
Jun 15 21:21:51 thelma.ucar.edu marzipan(feed)[14310]: topo:  
marzipan.atmos.washington.edu CONDUIT

Jun 17 22:30:58 thelma.ucar.edu freshair(feed)[27162]: up6.c:331: Starting 
Up(6.0.14/6): 20040617221627.623 TS_ENDT {{CONDUIT,  "[09]$"}}
Jun 17 22:30:58 thelma.ucar.edu freshair(feed)[27162]: topo:  
freshair.atmos.washington.edu CONDUIT
Jun 17 22:30:58 thelma.ucar.edu freshair(feed)[27163]: up6.c:331: Starting 
Up(6.0.14/6): 20040617221625.957 TS_ENDT {{CONDUIT,  "[45]$"}}
Jun 17 22:30:58 thelma.ucar.edu freshair(feed)[27163]: topo:  
freshair.atmos.washington.edu CONDUIT
Jun 17 22:30:58 thelma.ucar.edu freshair(feed)[27164]: up6.c:331: Starting 
Up(6.0.14/6): 20040617221625.305 TS_ENDT {{CONDUIT,  "[27]$"}}
Jun 17 22:30:58 thelma.ucar.edu freshair(feed)[27164]: topo:  
freshair.atmos.washington.edu CONDUIT
Jun 17 22:30:58 thelma.ucar.edu freshair(feed)[27165]: up6.c:331: Starting 
Up(6.0.14/6): 20040617221624.469 TS_ENDT {{CONDUIT,  "[18]$"}}
Jun 17 22:30:58 thelma.ucar.edu freshair(feed)[27165]: topo:  
freshair.atmos.washington.edu CONDUIT
Jun 17 22:30:58 thelma.ucar.edu freshair(feed)[27166]: up6.c:331: Starting 
Up(6.0.14/6): 20040617221625.580 TS_ENDT {{CONDUIT,  "[36]$"}}
Jun 17 22:30:58 thelma.ucar.edu freshair(feed)[27166]: topo:  
freshair.atmos.washington.edu CONDUIT

Would it be possible for you to feed freshair from marzipan or visa
versa?  If you need the separate requests for testing, can you switch
one of them from thelma.ucar.edu to emo.unidata.ucar.edu?  Thanks!

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.