I solved the Nexrad mystery! (was: Can rad handle new stuff...) (fwd)

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================

---------- Forwarded message ----------
Date: Tue, 07 Nov 2000 21:37:07 -0600
From: Pete Pokrandt <address@hidden>
To: David Wojtowicz <address@hidden>
     address@hidden, address@hidden,
     address@hidden, address@hidden,
     address@hidden
Subject: I solved the Nexrad mystery! (was: Can rad handle new stuff...)



All,

I have succeeded in uncompressing the zlib compressed ARX Nexrad files
coming in via the HDS feed. I was able to display the final product with
rad (a fairly old version at that), now I just need to synthesise some
of the steps involved to get to a single program that the files can be
piped into via the ldm, which results in an uncompressed, rad/gempak
readable file being written to disk.

Here's the long version of the story..

It was a combination of a few things that threw me off.

Bob and David were right, there is an uncompressed header on the files
that must be stripped off first. BUT, the remaining file is not in
a gzip or bzip or compressed or whatever format. Apparently those
utilities use the zlib algorithms, but encapsulate the data somehow.
Since the radar data does not have the identifying info that gzip needs,
it refuses to try to uncompress it.

I followed a tip from Mike and located a perl sample program, which
would uncompress files compressed by the zlib routines, and modified it
somewhat, and was able to uncompress the data.

Turns out that the uncompressed version also has a header on it,
slightly different than the uncompressed one from earlier, but not
included in the WSI NIDS files which work with wxp's rad and the various
gempak programs, so it has to be stripped off also.

One other wrench that threw me off for a while.. the files I were
working with had two separate uncompressed headers in them, followed by
binary data, which made no sense whatsoever based on the documentation I
had read. Usually, the data after one header was un-compressible, while
the other was not..

It turns out that the *encrypted* versions of the ARX files are also
still being sent out, in addition to the compressed ones. As far as I
can see, there is no way to differentiate between the compressed ones
(which we can read) and the encrypted ones (which we can't) in the
pqact.conf file. So both of these were being written into the one file I
was testing my routines with.

In practice, this shouldn't be a problem. I'll just modify the script
to die gracefully when it gets piped a file it can't decode (the
encrypted one) and to create a file when it can decode. That way the
encrypted one will be basically ignored, while the compressed version
will be uncompressed to a file on disk. Given the return codes from the
un-zlib'ing routines, this should be a piece of cake.

I'm going to refine my code and make sure it truly works with data piped
in from the ldm, but it looks like it should work. I'll make the perl
code available when it's done. It requires perl (obviously) and the
perl interface to zlib, there's a link to it from the zlib home page
(http://www.info-zip.org/pub/infozip/zlib/)

I'm a little concerned about the performance issues of firing up perl
every time a nids product comes in, especially once the whole stream
kicks in in January, so in the longer run I may use the perl code as
a basis for a C program that does the same thing. Otherwise, maybe a
seasoned C programmer could do it too, based on the logic in the perl
code < grovel grovel hint hint :) >.

It's been interesting, kinda like trying to get that grib decoder
working the first time.. Hex, decimal, binary, bloodshot eyes.. 
Thank God for od (octal dump) and hexl-mode in emacs!

Pete

--
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
^ Pete Pokrandt                    V 1447  AOSS Bldg  1225 W Dayton St^
^ Systems Programmer               V Madison,         WI     53706    ^
^                                  V      address@hidden       ^
^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (Phone/voicemail) ^
^ University of Wisconsin-Madison  V       262-0166 (Fax)             ^
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+