Re: I solved the Nexrad mystery! (was: Can rad handle new stuff...) (fwd)

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================

---------- Forwarded message ----------
Date: Tue, 07 Nov 2000 23:08:01 -0600
From: Pete Pokrandt <address@hidden>
To: address@hidden
     address@hidden, address@hidden,
     address@hidden, address@hidden,
     address@hidden, address@hidden
Subject: Re: I solved the Nexrad mystery! (was: Can rad handle new stuff...) 


Dan and all,

Well, It's pretty ugly still, I wrote some comments to what's
going on, but here's what I used. This script *does* strip
off the uncompressed header to begin with, but does *not* strip off the
header that shows up at the beginning of the uncompressed data after the
first call to inflate.  I did that by hand, but another call to 
substr prior to the first print to STDOUT would do it.

Pete

#------- Cut Here --------------
#!/usr/bin/perl
use Compress::Zlib ;
$input = '' ;
binmode STDIN;
binmode STDOUT;

# Read in the whole file, 1 Mb should cover it. It'll end w/o error
# when it hits the end of stdin

read (STDIN, $input, 1000000);

# strip off the uncompressed header. Don't need the swap  in variable
# but with my limited perl knowledge I didn't want to mess with the
# original prog much

$tempin = substr($input,41);
$input=$tempin;

# While the input string still has stuff in it - this works
# because the inflate function modifies input to have whatever
# is left in it after the first complete chunk is uncompressed
# (the 4000 byte section Dan mentioned)

while (length($input) > 0 )
{
# Initialize a new structure - this needs to be done for each call
# to inflate. die gracefully if it fails from lack of memory, whatever

$x = inflateInit()
   or die "Cannot create a inflation stream\n" ;

# call inflate. It returns
#  $output - the decompressed stream, which will be max 4000 bytes
#  $status - a return code. 0 is success,
#                           1 is end of data (i.e. you didn't send
#                             a complete stream, and it expected
#                             more data
#  $input  - contains the remainder of $input after the stream that
#            got decoded into $output
#

($output, $status) = $x->inflate(\$input) ;

# write $output to STDOUT. each write gets appended to the previous, so
# you eventually build up the whole file. The first write still needs
# to be modified to strip off the header that is in the file after
# decompression

print $output;

# Gracefully exit if the decode failed. This means something bad
# happened. This is how I discovered that I had both the compressed
# and encrypted data in one file. The compressed stuff worked but the
# encrypted didn't.

if ($status < 0 or $status > 2) {
die "bogus second header";
}
}
#------- Cut Here --------------

In a previous message to me, you wrote: 

 >I have also been able to uncompress the ARX NIDS data but I needed to do some 
 >tricks to get it to work.  This is what I did... its pretty ugly!!
 >
 >It turns out that NWS has split the NIDS file into 4000 byte blocks and then 
 >compressed the data using the zlib compress utility.  Then the compressed 
 >blocks are sent out sequentially as part of the whole product.  This means 
 >you have separate the blocks to decode.  
 >
 >The problem is that the uncompress utility doesn't work that effectively.  
 >The problem is you don't know the size of the compressed blocks.  The block 
 >sizes are buried in the SBN header which is masked off by the LDM.  So you 
 >have to guess what the block sizes are.  The uncompress utility will return 
 >the size of the uncompressed data but won't reveal the size of the compressed 
 >data. 
 >
 >What I've done is to forward through the data 1000 bytes at a time and then 
 >try to uncompress the data.  If the function returns an error, I increment 
 >through the data a byte at a time until I get a valid return on the 
 >uncompression.  This is just plain ugly....
 >
 >What I'm looking for is a simpler solution!!  If there is another utility 
 >that gives me the compressed block sizes, this would make my job much easier!
 >
 >BTW, I read where the uncompress utility will uncompress all the data if the 
 >file is memory mapped???  Otherwise, it only works a block at a time.
 >
 >So Pete, can you forward the Perl script so I can check what functions are 
 >used??
 >
 >Dan.... from cloudy Chicago!!


--
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
^ Pete Pokrandt                    V 1447  AOSS Bldg  1225 W Dayton St^
^ Systems Programmer               V Madison,         WI     53706    ^
^                                  V      address@hidden       ^
^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (Phone/voicemail) ^
^ University of Wisconsin-Madison  V       262-0166 (Fax)             ^
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+