20000621: Unidata-Wisconsin datastream additions: CIMSS products (cont.)

>From: "D. J. Raymond" <address@hidden>
>Organization: New Mexico Tech
>Keywords: 200006210029.e5L0TGT14952 Unidata-Wisconsin

Dave,

>My recollection is that the delta encoding is embedded in the old
>area file format.  Is this true?

No.  SSEC runs a routine that takes an AREA file and delta encodes it
for transmission.

We have been speculating that you are either running WXP and using the
option that Dan added to read delta-encoded AREA files directly (I
always thought that this was a bad thing to do, by the way) or that you
have your own code that reads the delta encoded image.  If you are
using WXP, then you can decode the imagery back to AREA file form and
things will keep working (WXP supplied by Unidata supported AREA file
display).

My experimentation with a variety of compression techniques (none,
delta, compress, gzip, bzip2 and PNG) showed that PNG was the overall
best technique for compressing the images in the UW datastream (just
slightly better than bzip2, but I was bzip2ing the entire image in
comparison to PNG compressing just the image portion).  Applications
like ImageMagic and web browsers know how to display PNG compressed
imagery directly.  All they have to do is skip the header block bytes
of the compressed image and they are ready to use.  This turns out to
be very useful for visually checking imagery that is saved in its
compressed form.

>Also, is there any document that gives the detailed specifications
>of the old and new area file formats, or do I have do reverse
>engineer this from the software?

The delta encoded image is not well documented at all (another reason
to get rid of it);  I had to read code to figure it out (ugly).
I can provide you with specifics if needed.

The AREA file format is documented extensively in a couple of files
(area.doc and area2.doc) that can be FTPed from the pub/mcidas
directory of anonymous FTP on ftp.unidata.ucar.edu.  These describe the
area directory entries that are needed to figure out where other
directories/codicils are located.

The AREA file format is also described in the McIDAS-X Programmer's
Reference Manual in the page concerning the AREA file format.  Please
refer to:

McIDAS Programmer's Manual
http://www.ssec.wisc.edu/mug/prog_man/prog_man.html

Chapter 7 - Format of the Data Files
http://www.ssec.wisc.edu/mug/prog_man/Current/progman-116.HTML#11044

The PNG compressed file format I came up with was designed to be as
simple as possible:

             AREA file                  PNG file
       +--------------------+   +----------------------+
       |  area directory    |   |  area directory      |
       +--------------------+   +----------------------+
       +--------------------+   +----------------------+
       | navigation codicil |   |  navigation codicil  |
       +--------------------+   +----------------------+
       +--------------------+   +----------------------+
       | calibration codicil|   | calibration codicil  |
       +--------------------+   +----------------------+
       +--------------------+   +----------------------+
       |    aux block       |   |      aux block       |
       +--------------------+   +----------------------+
       +--------------------+   +----------------------+
       |                    |   |  AREA file comment   |
       |                    |   |       cards          |
       |    image lines     |   +----------------------+
       |                    |   +----------------------+
       |                    |   |                      |
       +--------------------+   |    PNG compressed    |
       +--------------------+   |       image          |
       | AREA file comment  |   |                      |
       |      cards         |   |                      |
       +--------------------+   +----------------------+

The area directory contains the byte offsets of where each of the other
directories/blocks begin.

You should note that the calibration block, aux block, and comment
cards do not necessarily exist in the AREA file.  The UW imagery that
is being sent right now is pre-calibrated, so there is no calibration
block.  They also do not have aux blocks, but they do have comment
cards at the end (the comment cards record the McIDAS commands that had
been run on the original image to create the product).  Entries in the
area directory tell you this (by way of where they begin or where a
different block begins).

The PNG compression is only done on the image line portion of the
AREA file.  The comment cards, if they exist, are moved to just
after all of the other directories/codicils in the PNG compressed file.

The ldm-mcidas source contains PNG interface routines that make it very
easy to read/write the PNG compressed portion of the image.  I don't
think it will take you more than a minute or two of looking at
pnga2area.c to figure out the calling sequence.

I will be more than happy to lend a hand to get you going as quickly as
possible if you like.  I am quite sure, however, that you won't need this.

>Regards,
>Dave

Tom