IDD NIDS decompression script ready to go

All,

A follow up to the success I had last night. I now have the
decompression script running, stripping off all unused headers, 
and able to be PIPE'd into from pqact.conf.  The resulting
file that is written to disk looks identical to those
currently broadcast using the WSI data feed type to those
who subscribe to it.

Here's the lowdown. Remember the tabs between pieces here will
likely show up as spaces in this email.

You need to have the zlib libraries installed on your system.
Most linux boxes already have them, but others may need to
install from source (http://www.info-zip.org/pub/infozip/zlib/)

You need to have perl installed somewhere on your system, and you
need to install (if it's not already) the perl interface to
zlib, also available from the above zlib web site.

Then call the following script from pqact.conf. Make sure the first
line of the script is the #!/usr/bin/perl one, and change the path
on that line to point to where your perl lives.

#--- BEGIN nids_inflate.pl CUT HERE ---
#!/usr/bin/perl
use Compress::Zlib ;
$input = '' ;
binmode STDIN;
binmode STDOUT;
binmode OUTFILE;

# @ARGV is an array of command line arguments
# @ARGV[0] is the first one
# HDS 333  SDUS53 KARX 071918 /pNTPARX
# DDS ^SD.* .... ([0-3][0-9]) FILE Radar.(\1:yy)(\1:mm)(\1:dd)

# Read in the whole file, 1 Mb should cover it. It'll end w/o error
# when it hits the end of stdin

read (STDIN, $input, 1000000);

# strip off the uncompressed header. Don't need the swap  in variable
# but with my limited perl knowledge I didn't want to mess with the
# original prog much

$tempin = substr($input,41);
$input=$tempin;

# Read in the first chunk of data. This one has to have a header
# stripped off after decompression

$x = inflateInit()
   or die "Cannot create a inflation stream\n" ;
($output, $status) = $x->inflate(\$input) ;
if ($status < 0 or $status > 2) {
die "Not a zlib compressed file";
}
# print "Argv0 is @ARGV[0]\n";
# print FILEHANDLE LIST
open(OUTFILE, ">@ARGV[0]") or die "Can't open output file";

$stripped = substr($output,54);
print OUTFILE $stripped;


# While the input string still has stuff in it - this works
# because the inflate function modifies input to have whatever
# is left in it after the first complete chunk is uncompressed
# (the 4000 byte section Dan mentioned)

while (length($input) > 0 )
{
# Initialize a new structure - this needs to be done for each call
# to inflate. die gracefully if it fails from lack of memory, whatever

$x = inflateInit()
   or die "Cannot create a inflation stream\n" ;

# call inflate. It returns
#  $output - the decompressed stream, which will be max 4000 bytes
#  $status - a return code. 0 is success,
#                           1 is end of data (i.e. you didn't send
#                             a complete stream, and it expected
#                             more data
#  $input  - contains the remainder of $input after the stream that
#            got decoded into $output
#

($output, $status) = $x->inflate(\$input) ;
# write $output to STDOUT. each write gets appended to the previous, so
# you eventually build up the whole file. The first write still needs
# to be modified to strip off the header that is in the file after
# decompression

print OUTFILE $output;

# Gracefully exit if the decode failed. This means something bad
# happened. This is how I discovered that I had both the compressed
# and encrypted data in one file. The compressed stuff worked but the
# encrypted didn't.

if ($status < 0 or $status > 2) {
close OUTFILE;
die "bogus data";
}
}
close OUTFILE;
#--- END   nids_inflate.pl CUT HERE ---


My ldmd.conf REQUEST line looks like:

request HDS     "^SDUS53 KARX.*" sunset.meteor.wisc.edu

I'm only pulling in the radar portion of the HDS stream to this
particular machine, not the whole enchilada of HDS. 

My pqact.conf line for this radar data looks like:

HRS     ^SDUS53 K(ARX) (..)(....) /p(...).*     
        PIPE    
        -close /usr/local/ldm/util/nids_inflate.pl 
/usr/local/ldm/data/nids/\1/(\2:yy)(\2:mm)\2\3.\1.\4

This results in the following type file name being created 

/usr/local/ldm/data/nids/ARX/0011DDHHMM.TTT

where DD == Day 01-31
      HH == Hour00-23
      MM == Minute 00-53
      TTT== Data type from /p section, i.e. /pN0RARX would give TTT==N0R

The whole string after 'nids_inflate.pl' is just an argument to the
script, which is the file name that the data gets written to, so you
can change that to suit your needs.

I have developed and tested this under RedHat linux 6.1/6.2, so your
mileage may vary on other platforms, but perl is pretty platform
independent, so there shouldn't be any issues.

Performance-wise, I don't see any hit at all, it runs so fast, but
that's just a few products from one station. I would imagine sending all
108 or whatever radars and a full suite of products might cause it to be
more of an issue.

One other thing, the WSI data seems to come out with the time
rounded to the nearest 5 minutes, whereas the IDD NIDS data does
not.  This was only an issue for me, because I had wxp's rad
command looking for files with the time rounded to 5 minutes - 
something like 

nids-ARX          %D/ARX/%y%m%d%h%5n.ARX.%e

in the ~wxp/etc/ymdh file.

since many of the file names are not rounded to the nearest 5 minutes,
it didn't find them, even when you'd specify the correct file name/time.

I changed that to read:

nids-ARX          %D/ARX/%y%m%d%h%1n.ARX.%e

and all is well.

Good luck, hope this helps everyone out, feel free to email me with
questions, problems, successes, etc.

Pete

--
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
^ Pete Pokrandt                    V 1447  AOSS Bldg  1225 W Dayton St^
^ Systems Programmer               V Madison,         WI     53706    ^
^                                  V      address@hidden       ^
^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (Phone/voicemail) ^
^ University of Wisconsin-Madison  V       262-0166 (Fax)             ^
+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+