20040624: Perl redirect problems

>From: David Ovens <address@hidden>
>Organization: University of Washington
>Keywords: 200406241954.i5OJsCWb010248 LDM PIPE Perl

David,

>I have PERL scripts called png2gif.pl and zlib2gif.pl that are invoked
>from the following pqact.conf lines by the LDM:
>
>FNEXRAD ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
>        PIPE    -close
>        local/gini/bin/png2gif.pl
>        /home/data/gempak/nport/RADAR/\1/\2/\2_\3_\4
>
>NIMAGE
>^(satz/ch[0-9]/.*/(.*)/([12][0-9][0-9][0-9][01][0-9][0-3][0-9])
>([0-2][0-9][\
>0-5][0-9])/(.*)/(.*km))/
>        PIPE    -close
>        local/gini/bin/zlib2gif.pl
>        /home/data/gempak/nport/IMAGE/\5/\6/\2/\2_\3_\4 \1
>
>Inside png2gif.pl and zlib2gif.pl we do this to write STDIN to a file:
>
>##
>## also tried:
>##      system("cat > $outfile");
>##      close STDIN;
>##
>##      new method:
>        binmode STDIN;
>        $rec_term = $/;
>        undef $/;
>        open(OUT,">$outfile");
>        binmode OUT;
>        while (<STDIN>) {
>            print OUT;
>        }
>        close STDIN;
>        close OUT;
>        $/ = $rec_term;
>
>Then, later on in the scripts, I've done this to check on the status of an
>Xvfb process that should be running:
>
>  $index = index(`ps auxww | grep 'Xvfb :$display' | grep -v grep`, "Xvfb");
>  $index2 = index(`ps auxww | grep 'Xvfb :$display' | grep -v grep`, "Xvfb");
>or,
>##      $oldfh = select(STDOUT); $| = 1; select($oldfh);
>##      $oldfh = select(STDERR); $| = 1; select($oldfh);
>##      open(PSOUT,"ps auxww |");
>##      @psoutput = <PSOUT>;
>##      close PSOUT;
>##      foreach (@psoutput) {
>##          if (/^\s*$ENV{LOGNAME}.*\d+:\d+\s+Xvfb :$display/) {
>##              $running_display = 1;
>##              last;
>##          }
>##      }
>
>
>
>These scripts have brought up 2 issues.
>
>  1) on an OSF1 machine, occasionally, the writing of the file
>     produces a much smaller RADAR file than is produces by a "FILE"
>     entry on our Solaris LDM server which is also processing the same
>     files.

Are you comparing the size of the write by your Perl script on the OSF/1
machine with the size of a pqact FILE action on the Solaris machine?

If yes, I recommend setting up a FILE action on your OSF/1 machine and
comparing its size with the one on the Solaris box.

>This is not a case of multiple transmissions, but is a
>     case of a truncation on the OSF1 side, as near as I can tell.

Did you do a binary comparison of the files?  I recommend using 'cmp'
to compare the files to see where they are different.  The other
thing that might be informative is to do an 'od -c' of both files
and pipe the output of a 'diff' of both into 'less' (or 'more'):

od -c solaris.out > solaris.od
od -c osf1.out > osf1.od
diff solaris.out osf1.out | less

>  2) On our new Linux LDM server, freshair, running Debian Linux
>     2.6.7, there is some strange truncation issue with the 'ps'
>     command -- but only as run by the LDM process:

You mean only as run by the Perl script that is run from an LDM
pqact action, yes?  If yes, I suspect some problem with the
implementation of Perl on your Linux box.

>it works fine on
>     the command line every time.

Do you mean that if you cat the product into STDIN of the Perl script
run from the Unix command line that the 'ps' output is always OK?  If
yes, was this test done by the user running the LDM or by a different
user?  If it was done by a different user, is that user using the
same shell as 'ldm'?

>What I am noticing is that
>     the output from the 'ps' command (either as taken from @psoutput
>     in the '##' commented out section or from the index sections) can
>     be either 0 bytes or 8K.  We've seen the values of $index and
>     $index2 be (63, -1) and (-1, -1) on numerous occasions when we
>     know that they should both be 63.   

Clint Rowe (who is sitting in my office at the moment) suggests that
your PATH may have the GEMPAK bin directory in it before system
directories.  If this is the case, you may be running the GEMPAK
'ps' command instead of the Unix command.

>Is there something obvious to you folks that might be causing us to
>experience this strange behavior with our LDM?

If you implement the same FILE action on OSF/1 as you have on Solaris
SPARC and you see a difference in products saved that way, then there
would seem to be a problem related to the LDM.  If the comparison
is between a Perl script write action and a FILE action, then
we would suspect Perl, not the LDM.

>Is "PIPE -close" causing this,

Don't think so.

>should we try "EXEC",

You could, but the PIPE should be fine.  I use it all of the time
in Bourne shell scripts.

>have you heard of this kind of thing before?

No.

>Sincerely baffled,

The 'ps' problem may well be related to your PATH.  The size issue
is puzzling, but we will wait until you get the results from an
equivalent FILE test on your Sun Solaris SPARC and OSF/1 machines.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.

>From address@hidden  Thu Jun 24 17:20:10 2004

Tom,

I don't have time right now to give a detailed response and am heading
off camping tomorrow.  But, I can address the main issues you raise.

First, the 'ps' command is definitely /bin/ps always.  I've seen that
in my detailed diagnostic outputs.  And, even with the correct
'/bin/ps' the index method with backticks invocation that filled
$index and $index2 shows that from time to time the same command
issued twice a row in the zlib2gif.pl script produced different
results.  This is the main puzzle. 

As to how I run the command, Harry Edmon and I modified ldmadmin to
'exec' a shell and then we ran the command with STDIN from one of
these zlib-compressed GINI files.  The command works every time,
flawlessly, run as user ldm.  We've tried as best can to mimic what
the LDM does, but, something must still be different....

> You mean only as run by the Perl script that is run from an LDM
> pqact action, yes?  If yes, I suspect some problem with the
> implementation of Perl on your Linux box.

Yes.  Perl could be faulty, but, again, the only manifestation of this
fault is when an LDM pqact action invokes that PERL script.  I cannot
reproduce the fault any other way.


Second, as to the OSF1/Solaris differences, I modify the pqact.conf 
on the OSF1 box to this:

FNEXRAD ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
        FILE
        /home/data/gempak/nport.compare/RADAR/\1/\2/\2_\3_\4
FNEXRAD ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
        PIPE    -close
        local/gini/bin/png2gif.pl
        /home/data/gempak/nport/RADAR/\1/\2/\2_\3_\4

This would eliminate possible PERL problems on the OSF1 side.  Then I
can test the rest of what you suggest.

Thanks for your ideas,


David




On Thu, Jun 24, 2004 at 04:56:49PM -0600, Unidata Support wrote:
> >From: David Ovens <address@hidden>
> >Organization: University of Washington
> >Keywords: 200406241954.i5OJsCWb010248 LDM PIPE Perl
> 
> David,
> 
> >I have PERL scripts called png2gif.pl and zlib2gif.pl that are invoked
> >from the following pqact.conf lines by the LDM:
> >
> >FNEXRAD ^rad/NEXRCOMP/(...)/(...)_(........)_(....)
> >        PIPE    -close
> >        local/gini/bin/png2gif.pl
> >        /home/data/gempak/nport/RADAR/\1/\2/\2_\3_\4
> >
> >NIMAGE
> >^(satz/ch[0-9]/.*/(.*)/([12][0-9][0-9][0-9][01][0-9][0-3][0-9])
> >([0-2][0-9][\
> >0-5][0-9])/(.*)/(.*km))/
> >        PIPE    -close
> >        local/gini/bin/zlib2gif.pl
> >        /home/data/gempak/nport/IMAGE/\5/\6/\2/\2_\3_\4 \1
> >
> >Inside png2gif.pl and zlib2gif.pl we do this to write STDIN to a file:
> >
> >##
> >## also tried:
> >##      system("cat > $outfile");
> >##      close STDIN;
> >##
> >##      new method:
> >        binmode STDIN;
> >        $rec_term = $/;
> >        undef $/;
> >        open(OUT,">$outfile");
> >        binmode OUT;
> >        while (<STDIN>) {
> >            print OUT;
> >        }
> >        close STDIN;
> >        close OUT;
> >        $/ = $rec_term;
> >
> >Then, later on in the scripts, I've done this to check on the status of an
> >Xvfb process that should be running:
> >
> >  $index = index(`ps auxww | grep 'Xvfb :$display' | grep -v grep`, "Xvfb");
> >  $index2 = index(`ps auxww | grep 'Xvfb :$display' | grep -v grep`, "Xvfb");
> >or,
> >##      $oldfh = select(STDOUT); $| = 1; select($oldfh);
> >##      $oldfh = select(STDERR); $| = 1; select($oldfh);
> >##      open(PSOUT,"ps auxww |");
> >##      @psoutput = <PSOUT>;
> >##      close PSOUT;
> >##      foreach (@psoutput) {
> >##          if (/^\s*$ENV{LOGNAME}.*\d+:\d+\s+Xvfb :$display/) {
> >##              $running_display = 1;
> >##              last;
> >##          }
> >##      }
> >
> >
> >
> >These scripts have brought up 2 issues.
> >
> >  1) on an OSF1 machine, occasionally, the writing of the file
> >     produces a much smaller RADAR file than is produces by a "FILE"
> >     entry on our Solaris LDM server which is also processing the same
> >     files.
> 
> Are you comparing the size of the write by your Perl script on the OSF/1
> machine with the size of a pqact FILE action on the Solaris machine?
> 
> If yes, I recommend setting up a FILE action on your OSF/1 machine and
> comparing its size with the one on the Solaris box.
> 
> >This is not a case of multiple transmissions, but is a
> >     case of a truncation on the OSF1 side, as near as I can tell.
> 
> Did you do a binary comparison of the files?  I recommend using 'cmp'
> to compare the files to see where they are different.  The other
> thing that might be informative is to do an 'od -c' of both files
> and pipe the output of a 'diff' of both into 'less' (or 'more'):
> 
> od -c solaris.out > solaris.od
> od -c osf1.out > osf1.od
> diff solaris.out osf1.out | less
> 
> >  2) On our new Linux LDM server, freshair, running Debian Linux
> >     2.6.7, there is some strange truncation issue with the 'ps'
> >     command -- but only as run by the LDM process:
> 
> You mean only as run by the Perl script that is run from an LDM
> pqact action, yes?  If yes, I suspect some problem with the
> implementation of Perl on your Linux box.
> 
> >it works fine on
> >     the command line every time.
> 
> Do you mean that if you cat the product into STDIN of the Perl script
> run from the Unix command line that the 'ps' output is always OK?  If
> yes, was this test done by the user running the LDM or by a different
> user?  If it was done by a different user, is that user using the
> same shell as 'ldm'?
> 
> >What I am noticing is that
> >     the output from the 'ps' command (either as taken from @psoutput
> >     in the '##' commented out section or from the index sections) can
> >     be either 0 bytes or 8K.  We've seen the values of $index and
> >     $index2 be (63, -1) and (-1, -1) on numerous occasions when we
> >     know that they should both be 63.   
> 
> Clint Rowe (who is sitting in my office at the moment) suggests that
> your PATH may have the GEMPAK bin directory in it before system
> directories.  If this is the case, you may be running the GEMPAK
> 'ps' command instead of the Unix command.
> 
> >Is there something obvious to you folks that might be causing us to
> >experience this strange behavior with our LDM?
> 
> If you implement the same FILE action on OSF/1 as you have on Solaris
> SPARC and you see a difference in products saved that way, then there
> would seem to be a problem related to the LDM.  If the comparison
> is between a Perl script write action and a FILE action, then
> we would suspect Perl, not the LDM.
> 
> >Is "PIPE -close" causing this,
> 
> Don't think so.
> 
> >should we try "EXEC",
> 
> You could, but the PIPE should be fine.  I use it all of the time
> in Bourne shell scripts.
> 
> >have you heard of this kind of thing before?
> 
> No.
> 
> >Sincerely baffled,
> 
> The 'ps' problem may well be related to your PATH.  The size issue
> is puzzling, but we will wait until you get the results from an
> equivalent FILE test on your Sun Solaris SPARC and OSF/1 machines.
> 
> Cheers,
> 
> Tom
> --
> **************************************************************************** <
> Unidata User Support                                    UCAR Unidata Program <
> (303)497-8643                                                  P.O. Box 3000 <
> address@hidden                                   Boulder, CO 80307 <
> ---------------------------------------------------------------------------- <
> Unidata WWW Service              http://my.unidata.ucar.edu/content/support  <
> ---------------------------------------------------------------------------- <
> NOTE: All email exchanges with Unidata User Support are recorded in the
> Unidata inquiry tracking system and then made publically available
> through the web.  If you do not want to have your interactions made
> available in this way, you must let us know in each email you send to us.

-- 
David Ovens              e-mail: address@hidden
Research Meteorologist    phone: (206) 685-8108
Dept of Atm. Sciences      plan: Real-time MM5 forecasting for the
Box 351640                        Pacific Northwest
University of Washington          http://www.atmos.washington.edu/mm5rt
Seattle, WA  98195               Weather Graphics and Loops
                                  http://www.atmos.washington.edu/~ovens/loops