20040628: Perl redirect problems (cont.)

>From: David Ovens <address@hidden>
>Organization: University of Washington
>Keywords: 200406241954.i5OJsCWb010248 LDM PIPE Perl

David,

>Back to the OSF1 LDM problem....

>> Are you comparing the size of the write by your Perl script on the OSF/1
>> machine with the size of a pqact FILE action on the Solaris machine?
>> 
>> If yes, I recommend setting up a FILE action on your OSF/1 machine and
>> comparing its size with the one on the Solaris box.

>I have modified my png2gif.pl script and my pqact.conf file on our
>OSF1 machine so that the pqact.conf has these:
>
>FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
>        FILE
>        /home/disk/ldm/local/bin/gini/png2gif.pl
>        /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4

I am not sure what the reference to png2gif.pl is doing in the FILE
action.  I will assume that this you meant:

FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
        FILE
        /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4

>FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
>        PIPE    -close
>        /home/disk/ldm/local/bin/gini/png2gif.pl
>        /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4
>
>and the png2gif.pl script now creates a ".1" version of the file using
>this: 
>
>binmode STDIN;
>$rec_term = $/;
>undef $/;
>$outfile = "$localfile.1";
>open(OUT,">$outfile");
>binmode OUT;
>while (<STDIN>) {
>    print OUT;
>}
>close STDIN;
>close OUT;
>$/ = $rec_term;
>
>and then it compares the file size to what the pqact 'FILE' action
>above produces.  Thus, now the $localfile and the $localfile.1 files
>are produced on the same machine.  Most of the time, the files are
>identical, but in the 7 hours in which this has been running, there is
>one file that ends up with two different sizes.
>
>-rw-r--r--    1 ldm      lpadmin   1109984 Jun 28 18:26 n0r_20040629_0117
>-rw-r--r--    1 ldm      lpadmin    262144 Jun 28 18:26 n0r_20040629_0117.1

How many actions are being processed by the pqact running on the
OSF/1 machine?  The reason I ask is that each instance of pqact
can have only 32 file descriptors open at one time.  Note: Harry
commented that he routinely changes this limit from 32 to 64, but it
is not clear to me that he does this on each machine at UW.

>The larger size file is identical to the file that is produced on our
>Solaris machine from a similar 'FILE' pqact entry.

This is reassuring and it indicates that the LDM/pqact is functioning
correctly.

>The output of 
>'cmp -l n0r_20040629_0117 n0r_20040629_0117.1' gives 
>   cmp: EOF on n0r_20040629_0117.1

>I also ran the 'od -c' comparisons that you suggested above and
>confirmed that the PERL version is simply truncated.

OK.

>Please note that
>I have tried all of the following methods in the png2gif.pl PERL
>script and have gotten the same type of random truncation behavior.
>  method 2)
>   open (PNG,"|cat > $localfile");
>   print PNG <STDIN>;
>   close PNG;
>
>  method 3)
>   open (PNG,"|cat > $localfile");
>   while (<STDIN>) {
>       print PNG;
>   }
>   close PNG;
>  
>  method 4)
>   system "cat > $localfile";
>   close STDIN;
>
>Is this a PERL bug or an LDM bug?

Since the FILE action produces consistent results that match the
same action on a different OS, I would be inclined to say that the
problem is related to Perl.  I will talk to others here at the UPC
about this later today.

>Are there any other tests to do?

The only other thing I can think of is to write a Bourne shell
script that files the same product as pqact's FILE and your Perl
script.

Here is what I have in mind:

- create ldmfile.sh in /local/ldm/local/bin and make sure that it is
  executable:

#!/bin/sh -f

#--------------------------------------------------------------------------
#
# Name:    ldmfile.sh
#
# Purpose: file a LDM product and log the receipt of the product
#
# History: 20030815 - Created for Zlib-compressed GINI image filing
#
#--------------------------------------------------------------------------

SHELL=sh
export SHELL

# Set log file
LOG=/local/ldm/logs/ldm-mcidas.log
exec >>$LOG 2>&1

# Create directory structure
fname=`basename $1`
dirs=`echo $1 | sed s/$fname//`
mkdir -p $dirs

# Write stdin to the designated file and log its filing
echo `date -u +'%b %d %T'` `basename $0`\[$$\]: FILE $1
cat > $1

# Done
exit 0

- add a third action to your pqact.conf file:

FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
        FILE
        /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4
FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
        PIPE    -close
        /local/ldm/local/bin/ldmfile.sh
        /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4.2
FNEXRAD ^rad/NEXRCOMP/(1km)/(n0r)_(........)_(....)
        PIPE    -close
        /home/disk/ldm/local/bin/gini/png2gif.pl
        /home/glacier/ldm/nport/RADAR/\1/\2/\2_\3_\4

If the Bourne shell script ldmfile.sh output is identical to the
pqact FILE action, then it seems most reasonable that the problem
is in the verion of Perl that you are using.

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.