[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
20041214: LDM nids_inflate
- Subject: 20041214: LDM nids_inflate
- Date: Tue, 14 Dec 2004 11:05:03 -0700
Rhesa,
I'm afraid that I'm not competent to comment on the perl-script.
I am curious, however, about the LDM crashing because that should never
happen. Would you please give me more details.
Regards,
Steve Emmerson
------- Original Message
Date: Tue, 14 Dec 2004 10:00:01 -0700
From: Unidata Support <address@hidden>To: "Rhesa Freeman" <rf
address@hidden>
cc: address@hidden, address@hidden,
address@hidden
Subject: 20041214: LDM nids_inflate
Rhesa,
Sounds like your perl script has a bug, and may be running
in an infinite loop or running out of memory on your system.
I'll see if Steve Emmerson has any other suggestions about LDM crashing if
your system is out of resources.
Since the script was from Pete Pokrandt, you may want to
consult with him on any problems he might have encountered with
the use of his script.
In general, trying to uncompress nids products is going to be very
expensive in real time if you are receiving the full suite of
NIDS sites. Do you need to uncompress the products from the LDM?
The Unidata display packages can real the products in the zlib
compressed format.
Steve Chiswell
Unidata User Support
>From: "Rhesa Freeman" <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200412141549.iBEFnolI021151
>This is a multi-part message in MIME format.
>
>------=_NextPart_000_0010_01C4E1C2.3DFAAD00
>Content-Type: text/plain;
> charset="iso-8859-1"
>Content-Transfer-Encoding: quoted-printable
>
>We are currently running the ldm 6.1.0 and are having a problem with the =
>nids_inflate.pl script. =20
>
>Here is the script:
>
>#!/usr/bin/perl
>#
># Based on example code available from
># http://www.perldoc.com/cpan/Compress/Zlib.html#DEFLATE
>#
># Modified for Unidata IDD-NIDS use by
># Pete Pokrandt, University of Wisconsin, Nov 2000.
># address@hidden
>#
>use Compress::Zlib ;
>use File::Path ;
>$input =3D '' ;
>binmode STDIN;
>binmode STDOUT;
>binmode OUTFILE;
>
># Read in the whole file, 1 Mb should cover it. It'll end w/o error
># when it hits the end of stdin
>
>read (STDIN, $input, 1000000);
>close STDIN;
>
># strip off the uncompressed header. Don't need the swap in variable
># but with my limited perl knowledge I didn't want to mess with the
># original prog much
>
>$tempin =3D substr($input,41);
>$input=3D$tempin;
>
># Read in the first chunk of data. This one has to have a header
># stripped off after decompression
>
>$x =3D inflateInit()
> or die "Cannot create a inflation stream\n" ;
>($output, $status) =3D $x->inflate(\$input) ;
>if ($status < 0 or $status > 2) {
>die "Not a zlib compressed file";
>}
># print "Argv0 is @ARGV[0]\n";
># print FILEHANDLE LIST
># check to see if path exists, create it if not. Ugly code alert!
># fix this later
># The length in substr (last number) must match the length of the
># path to the subdir, minus the length of the acutal file name.
>$where =3D substr(@ARGV[0], 0, 34);
># printf (STDERR "Where: $where\n");
>if (-d $where) {
> } else {
> mkpath("$where", 0, 0777);
>}
>open(OUTFILE, ">@ARGV[0]") or die "Can't open output file @ARGV[0]. =
>Error =3D $!\n";
>
>$stripped =3D substr($output,54);
>print OUTFILE $stripped;
>
>
># While the input string still has stuff in it - this works
># because the inflate function modifies input to have whatever
># is left in it after the first complete chunk is uncompressed
># (the 4000 byte section Dan mentioned)
>
>while (length($input) > 0 )
>{
># Initialize a new structure - this needs to be done for each call
># to inflate. die gracefully if it fails from lack of memory, whatever
>
>$x =3D inflateInit()
> or die "Cannot create a inflation stream\n" ;
>
># call inflate. It returns
># $output - the decompressed stream, which will be max 4000 bytes
># $status - a return code. 0 is success,
># 1 is end of data (i.e. you didn't send
># a complete stream, and it expected
># more data
># $input - contains the remainder of $input after the stream that
># got decoded into $output
>#
>
>($output, $status) =3D $x->inflate(\$input) ;
>
># write $output to STDOUT. each write gets appended to the previous, so
># you eventually build up the whole file. The first write still needs
># to be modified to strip off the header that is in the file after
># decompression
>
>print OUTFILE $output;
>
># Gracefully exit if the decode failed. This means something bad
># happened. This is how I discovered that I had both the compressed
># and encrypted data in one file. The compressed stuff worked but the
># encrypted didn't.
>
>if ($status < 0 or $status > 2) {
>print OUTFILE "\n";
>close OUTFILE;
>exit 0;
>#die "bogus second header";
>}
>}
>print OUTFILE "\n";
>close OUTFILE;
>
>exit 0;
>
>
>
>Here is the log file:
>
>Dec 13 18:43:43 pqact[27120]: Starting Up
>Dec 13 18:44:12 pqact[27120]: child 27329 exited with status 32
>Dec 13 18:44:35 pqact[27120]: pbuf_flush 3: time elapsed 2.666694
>Dec 13 18:45:44 pqact[27120]: child 28068 exited with status 32
>Dec 13 18:47:21 pqact[27120]: pbuf_flush 3: time elapsed 2.455863
>Dec 13 18:49:28 pqact[27120]: pbuf_flush 3: time elapsed 2.905513
>Dec 13 18:49:32 pqact[27120]: pbuf_flush 3: time elapsed 2.231726
>Dec 13 19:01:57 pqact[27120]: child 35710 exited with status 32
>Dec 13 19:08:04 pqact[27120]: child 38341 exited with status 32
>Dec 13 19:08:20 pqact[27120]: child 38451 exited with status 32
>Dec 13 19:16:45 pqact[27120]: child 41379 exited with status 32
>
>Then the ldm crashes. Any suggestions? Thanks.
>
>Rhesa Freeman
>Data Manager
>Regional Weather Information Center
>University of North Dakota
>
>
>
>------=_NextPart_000_0010_01C4E1C2.3DFAAD00
>Content-Type: text/html;
> charset="iso-8859-1"
>Content-Transfer-Encoding: quoted-printable
>
><!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
><HTML><HEAD>
><META http-equiv=3DContent-Type content=3D"text/html; =
>charset=3Diso-8859-1">
><META content=3D"MSHTML 6.00.2900.2523" name=3DGENERATOR>
><STYLE></STYLE>
></HEAD>
><BODY bgColor=3D#ffffff>
><DIV><FONT face=3DArial size=3D2>We are currently running the ldm 6.1.0 =
>and are=20
>having a problem with the nids_inflate.pl script. </FONT></DIV>
><DIV><FONT face=3DArial size=3D2></FONT> </DIV>
><DIV><FONT face=3DArial size=3D2>Here is the script:</FONT></DIV>
><DIV><FONT face=3DArial size=3D2></FONT> </DIV>
><DIV><FONT face=3DArial size=3D2>#!/usr/bin/perl<BR>#<BR># Based on =
>example code=20
>available from<BR># <A=20
>href=3D"http://www.perldoc.com/cpan/Compress/Zlib.html#DEFLATE";>http://ww=
>w.perldoc.com/cpan/Compress/Zlib.html#DEFLATE</A><BR>#<BR>#=20
>Modified for Unidata IDD-NIDS use by<BR># Pete Pokrandt, =
>University=20
>of Wisconsin, Nov 2000.<BR># <A=20
>href=3D"mailto:address@hidden";>address@hidden</A><BR>#<BR>u=
>se=20
>Compress::Zlib ;<BR>use File::Path ;<BR>$input =3D '' ;<BR>binmode=20
>STDIN;<BR>binmode STDOUT;<BR>binmode OUTFILE;</FONT></DIV>
><DIV> </DIV>
><DIV><FONT face=3DArial size=3D2># Read in the whole file, 1 Mb should =
>cover it.=20
>It'll end w/o error<BR># when it hits the end of stdin</FONT></DIV>
><DIV> </DIV>
><DIV><FONT face=3DArial size=3D2>read (STDIN, $input, 1000000);<BR>close =
>
>STDIN;</FONT></DIV>
><DIV> </DIV>
><DIV><FONT face=3DArial size=3D2># strip off the uncompressed header. =
>Don't need the=20
>swap in variable<BR># but with my limited perl knowledge I didn't =
>want to=20
>mess with the<BR># original prog much</FONT></DIV>
><DIV> </DIV>
><DIV><FONT face=3DArial size=3D2>$tempin =3D=20
>substr($input,41);<BR>$input=3D$tempin;</FONT></DIV>
><DIV> </DIV>
><DIV><FONT face=3DArial size=3D2># Read in the first chunk of data. This =
>one has to=20
>have a header<BR># stripped off after decompression</FONT></DIV>
><DIV> </DIV>
><DIV><FONT face=3DArial size=3D2>$x =3D inflateInit()<BR> or =
>die "Cannot=20
>create a inflation stream\n" ;<BR>($output, $status) =3D =
>$x->inflate(\$input)=20
>;<BR>if ($status < 0 or $status > 2) {<BR>die "Not a zlib =
>compressed=20
>file";<BR>}<BR># print "Argv0 is @ARGV[0]\n";<BR># print FILEHANDLE =
>LIST<BR>#=20
>check to see if path exists, create it if not. Ugly code =
>alert!<BR># =20
>fix this later<BR># The length in substr (last number) must match =
>the=20
>length of the<BR># path to the subdir, minus the length of =
>the=20
>acutal file name.<BR>$where =3D substr(@ARGV[0], 0, 34);<BR># printf =
>(STDERR=20
>"Where: $where\n");<BR>if (-d $where)=20
>{<BR> } else=20
>{<BR> mkpath("$where", 0,=20
>0777);<BR>}<BR>open(OUTFILE, ">@ARGV[0]") or die "Can't open output =
>file=20
>@ARGV[0]. Error =3D $!\n";</FONT></DIV>
><DIV> </DIV>
><DIV><FONT face=3DArial size=3D2>$stripped =3D =
>substr($output,54);<BR>print OUTFILE=20
>$stripped;</FONT></DIV>
><DIV> </DIV><FONT face=3DArial size=3D2>
><DIV><BR># While the input string still has stuff in it - this =
>works<BR>#=20
>because the inflate function modifies input to have whatever<BR># is =
>left in it=20
>after the first complete chunk is uncompressed<BR># (the 4000 byte =
>section Dan=20
>mentioned)</DIV>
><DIV> </DIV>
><DIV>while (length($input) > 0 )<BR>{<BR># Initialize a new structure =
>- this=20
>needs to be done for each call<BR># to inflate. die gracefully if it =
>fails from=20
>lack of memory, whatever</DIV>
><DIV> </DIV>
><DIV>$x =3D inflateInit()<BR> or die "Cannot create a =
>inflation=20
>stream\n" ;</DIV>
><DIV> </DIV>
><DIV># call inflate. It returns<BR># $output - the decompressed =
>stream,=20
>which will be max 4000 bytes<BR># $status - a return code. 0 is=20
>success,<BR># =
> &=
>nbsp; =20
>1 is end of data (i.e. you didn't=20
>send<BR># &nbs=
>p;  =
>; =20
>a complete stream, and it=20
>expected<BR># =
> &=
>nbsp; =20
>more data<BR># $input - contains the remainder of $input =
>after the=20
>stream=20
>that<BR># &nbs=
>p; got=20
>decoded into $output<BR>#</DIV>
><DIV> </DIV>
><DIV>($output, $status) =3D $x->inflate(\$input) ;</DIV>
><DIV> </DIV>
><DIV># write $output to STDOUT. each write gets appended to the =
>previous,=20
>so<BR># you eventually build up the whole file. The first write still =
>needs<BR>#=20
>to be modified to strip off the header that is in the file after<BR>#=20
>decompression</DIV>
><DIV> </DIV>
><DIV>print OUTFILE $output;</DIV>
><DIV> </DIV>
><DIV># Gracefully exit if the decode failed. This means something =
>bad<BR>#=20
>happened. This is how I discovered that I had both the compressed<BR># =
>and=20
>encrypted data in one file. The compressed stuff worked but the<BR># =
>encrypted=20
>didn't.</DIV>
><DIV> </DIV>
><DIV>if ($status < 0 or $status > 2) {<BR>print OUTFILE =
>"\n";<BR>close=20
>OUTFILE;<BR>exit 0;<BR>#die "bogus second header";<BR>}<BR>}<BR>print =
>OUTFILE=20
>"\n";<BR>close OUTFILE;</DIV>
><DIV> </DIV>
><DIV>exit 0;<BR></FONT></DIV>
><DIV><FONT face=3DArial size=3D2></FONT> </DIV>
><DIV><FONT face=3DArial size=3D2></FONT> </DIV>
><DIV><FONT face=3DArial size=3D2>Here is the log file:</FONT></DIV>
><DIV><FONT face=3DArial size=3D2></FONT> </DIV>
><DIV><FONT face=3DArial size=3D2>Dec 13 18:43:43 pqact[27120]: Starting =
>Up<BR>Dec 13=20
>18:44:12 pqact[27120]: child 27329 exited with status 32<BR>Dec 13 =
>18:44:35=20
>pqact[27120]: pbuf_flush 3: time elapsed 2.666694<BR>Dec 13 =
>18:45:44=20
>pqact[27120]: child 28068 exited with status 32<BR>Dec 13 18:47:21 =
>pqact[27120]:=20
>pbuf_flush 3: time elapsed 2.455863<BR>Dec 13 18:49:28 =
>pqact[27120]:=20
>pbuf_flush 3: time elapsed 2.905513<BR>Dec 13 18:49:32 =
>pqact[27120]:=20
>pbuf_flush 3: time elapsed 2.231726<BR>Dec 13 19:01:57 =
>pqact[27120]:=20
>child 35710 exited with status 32<BR>Dec 13 19:08:04 pqact[27120]: child =
>38341=20
>exited with status 32<BR>Dec 13 19:08:20 pqact[27120]: child 38451 =
>exited with=20
>status 32<BR>Dec 13 19:16:45 pqact[27120]: child 41379 exited with =
>status=20
>32<BR></FONT></DIV>
><DIV><FONT face=3DArial size=3D2>Then the ldm crashes. Any =
>suggestions? =20
>Thanks.</FONT></DIV>
><DIV><FONT face=3DArial size=3D2></FONT> </DIV>
><DIV><FONT face=3DArial size=3D2>Rhesa Freeman</FONT></DIV>
><DIV><FONT face=3DArial size=3D2>Data Manager</FONT></DIV>
><DIV><FONT face=3DArial size=3D2>Regional Weather Information =
>Center</FONT></DIV>
><DIV><FONT face=3DArial size=3D2>University of North Dakota</FONT></DIV>
><DIV><FONT face=3DArial size=3D2></FONT> </DIV>
><DIV><FONT face=3DArial size=3D2></FONT> </DIV>
><DIV><FONT face=3DArial size=3D2> </DIV></FONT></BODY></HTML>
>
>------=_NextPart_000_0010_01C4E1C2.3DFAAD00--
>
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web. If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.
------- End of Original Message