20040629: potential LDM/pqact problem on OSF/1

Tom,

As far as I know, simple rearrangment of the order of actions in a
pqact(1) configuration-file shouldn't affect their behavior -- assuming
we're talking about single data-products here and the pipe is closed
every time.

Do we have a simple example with which we can duplicate the problem?

Regards,
Steve Emmerson

----- Original Message

>Date: Tue, 29 Jun 2004 15:39:33 -0600
>From: Unidata Support <address@hidden>
>Organization: UCAR/Unidata
>To: address@hidden
>Subject: 20040629: potential LDM/pqact problem on OSF/1

>From: David Ovens <address@hidden>
>Organization: University of Washington
>Keywords: 200406241954.i5OJsCWb010248 LDM PIPE Perl

Steve,

I have been working with David Ovens of the University of Washington on
a problem he is seeing when running a Perl script from a pqact.conf
action under OSF/1.  One of the first things his Perl script does is
write the product that is 'PIPE -close'd to it into a disk file.  David
has observed that the disk file is truncated on many occasions, while a
pqact FILE action for the same product is not.  He has also compared
the file(s) written on the OSF/1 machine with the same file(s)
processed on a Sun Solaris machine and sees that the ones written with
FILE are the same size.

My last test suggestion to David was to compare the size of a file
created by a Bourne shell scipt which I included in my last email to
him.

Interestingly, David has found that the size of files made by the pqact
FILE action and a 'PIPE -close' to the Bourne shell script were the
same when the Bourne shell script invocation was located immediately
after the FILE action and before the 'PIPE -close' action to his Perl
script in his pqact.conf file.  In this configuration, the Perl script
would produce truncated files while the other two actions would not.
His last test was to move the Perl action immediately before the Bourne
shell script action.  With this setup, he has seen both the Perl and
Bourne shell scripts produce truncated disk files while the pqact FILE
action produced full sized files.

So, the question is how moving the Perl action to before the Bourne
shell script action could cause truncated files.  If the problem was
strictly related to Perl -- my original assumption --, it should not
matter what order the actions were in the pqact.conf file.  Correct?
David is convinced that the problem is not related to Perl, but,
rather to the LDM running under OSF/1.

One last comment.  Harry Edmon made a comment to me that he changes
the number of open file descriptors a pqact can have in the LDM
he runs at UW.  It was not clear to me that Harry was saying that
he does this on every machine running the LDM, or on a subset
of the machines running the LDM.

Please send David any ideas you may have on his situation.

Thanks,

Tom

------- Forwarded Message

>To: Unidata Support <address@hidden>
>cc: David Ovens <address@hidden>,
>cc: Harry Edmon <address@hidden>
>Subject: Re: 20040628: Perl redirect problems (cont.)

On Tue, Jun 29, 2004 at 07:33:49AM -0600, Unidata Support wrote:
> How many actions are being processed by the pqact running on the
> OSF/1 machine?  The reason I ask is that each instance of pqact
> can have only 32 file descriptors open at one time.  Note: Harry
> commented that he routinely changes this limit from 32 to 64, but it
> is not clear to me that he does this on each machine at UW.

I do not know about the answer to this one, Harry is on vacation
today, but I do know that the only things this machine are processing
are the FNEXRAD and GINI files, nothing else (see below for this
machine's complete pqact.conf file).

> >Are there any other tests to do?
> 
> The only other thing I can think of is to write a Bourne shell
> script that files the same product as pqact's FILE and your Perl
> script.
> 
> Here is what I have in mind:
> 
> - create ldmfile.sh in /local/ldm/local/bin and make sure that it is
>   executable:
> 
> #!/bin/sh -f
> 
> #--------------------------------------------------------------------------
> #
> # Name:    ldmfile.sh
> #
> # Purpose: file a LDM product and log the receipt of the product
> #
> # History: 20030815 - Created for Zlib-compressed GINI image filing
> #
> #--------------------------------------------------------------------------
> 
> SHELL=sh
> export SHELL
> 
> # Set log file
> LOG=/local/ldm/logs/ldm-mcidas.log
> exec >>$LOG 2>&1
> 
> # Create directory structure
> fname=`basename $1`
> dirs=`echo $1 | sed s/$fname//`
> mkdir -p $dirs
> 
> # Write stdin to the designated file and log its filing
> echo `date -u +'%b %d %T'` `basename $0`\[$$\]: FILE $1
> cat > $1
> 
> # Done
> exit 0
> 
> - add a third action to your pqact.conf file:
> 
> FNEXRAD       ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
>       FILE
>       /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4
> FNEXRAD       ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
>       PIPE    -close
>       /local/ldm/local/bin/ldmfile.sh
>       /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4.2
> FNEXRAD       ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
>       PIPE    -close
>       /home/disk/ldm/local/bin/gini/png2gif.pl
>       /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4
> 
> If the Bourne shell script ldmfile.sh output is identical to the
> pqact FILE action, then it seems most reasonable that the problem
> is in the verion of Perl that you are using.

Ok, I did this, first as you suggested with my pqact.conf file looking
like this:

FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
        FILE    /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4
FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
        PIPE    -close
        /home/glacier/ldm/ldmfile.sh
        /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4.2
FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
        PIPE    -close
        /home/disk/ldm/local/bin/gini/png2gif.pl
        /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4

Unfortunately, after that each of the PERL-produced files was
truncated, though the Bourne-shell produced ones were fine!  Then, I
flipped the 'PIPE' order to do the PERL one first, here's my current,
complete pqact.conf file:

FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
        FILE    /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4
FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
        PIPE    -close
        /home/disk/ldm/local/bin/gini/png2gif.pl
        /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4
FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
        PIPE    -close
        /home/glacier/ldm/ldmfile.sh
        /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4.2
#PNG compressed
NIMAGE  ^(sat/ch[0-9]/.*/(.*)/([12][0-9][0-9][0-9][01][0-9][0-3][0-9])
([0-2][0-9][0-5][0-9])/(.*)/(.*km))/
        PIPE    -close
        /home/disk/ldm/local/bin/gini/png2gif.pl
        /home/glacier/ldm/nport/IMAGE/\5/\6/\2/\2_\3_\4 \1
#Zlib compressed
NIMAGE
^(satz/ch[0-9]/.*/(.*)/([12][0-9][0-9][0-9][01][0-9][0-3][0-9])
([0-2][0-9][0-5][0-9])/(.*)/(.*km))/
        PIPE    -close
        /home/disk/ldm/local/bin/gini/zlib2gif.pl
        /home/glacier/ldm/nport/IMAGE/\5/\6/\2/\2_\3_\4 \1

and this worked fine for several hours.  Until, 

-rw-r--r--   1 ldm      ldm      1445402 Jun 29 12:11 n0r_20040629_1901
-rw-r--r--   1 ldm      ldm       262144 Jun 29 12:11 n0r_20040629_1901.1
-rw-r--r--   1 ldm      ldm       262144 Jun 29 12:11 n0r_20040629_1901.2

Now, the Bourne-shell version and the PERL-version, for this one file,
are identical and are truncated (as confirmed by 'cmp' commands on
these)!  Hooray!  Consistency in the bug! This ought to help!

Sure looks like an OSF1 bug in the LDM to me, but maybe it has
something to do with those limits you were asking about.

David
-- 
David Ovens              e-mail: address@hidden
Research Meteorologist    phone: (206) 685-8108
Dept of Atm. Sciences      plan: Real-time MM5 forecasting for the
Box 351640                        Pacific Northwest
University of Washington          http://www.atmos.washington.edu/mm5rt
Seattle, WA  98195               Weather Graphics and Loops
                                  http://www.atmos.washington.edu/~ovens/loops

--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.

------- End of Forwarded Message

----- End of Original Message