[Support #YEF-379651]: McIDAS AREA files with PNG compression

HI Bas,

re:
> Mr. Valentijn Venus has asked me to adjust the McIDAS files in order to
> decompress PNG-compressed AREA files on-the-fly. At the end of this
> e-mail, I have included an e-mail with previous communication between
> Unidata support and Mr. Valentijn Venus.

OK, thanks for including the previous email.

> After spending quite some time learning Fortran (I have a C/C++
> background, and it was the first time that I really needed to analyse
> fortran code),

Luckily, the code that needs to be enhanced is in C, not Fortran :-)

> and understanding the necessary parts of the McIDAS
> source code, I have come up with the following solution:
> 
> 1. Make some adjustments in file mcread.c so that it decompresses
>    PNG-compressed AREA files (I still have to think about how to let mcread
>    detect whether a file is a compressed PNG or not)

The PNG-compressed portion of the images in question are recognizable by
a "magic word" located at the beginning of the compressed portion of the
image.  Also, the AREA file header has been modified to put the comment
blocks after the header components instead of at the end of the image.
Either/both of these clues can be used to alert the reader that there is
a need to uncompress the image.

> 2. If needed, make adjustments to the corresponding Fortran files that
>    call the mcread-functions (like RDTRKA.for, REDARA.for and AGETSERV.fp)
>    - however, I think if mcread is intelligent enough, no adjustments are
>    needed in other files.

I agree.  There should be no need to modify the Fortran routines.  The C read
routine can be made intelligent enough to recognize the need to uncompress
the PNG-compressed data portion of the AREA file.  The only reason I did not
do this was sites were already used to including uncompression actions in
their ~ldm/etc/pqact.conf processing file.  Please don't misinterpret this
comment:  I think adding uncompression on the fly would be a fantastic
enhancement to the ADDE serving of the data!

> 3. mcread will call a still-to-be-implemented PNG-decompress function,
>    that looks similar to the main function in pnga2area.c.

My comment:  instead of making the function look similar to the main function
in pnga2area.c, why not make it the same function?  In order to add the
ability to decompress the PNG portion, we will need to add the PNG library
to the standard McIDAS build, and it will be very easy to simply include
the routines we already wrote into the McIDAS library.

>    Most important
>    difference is that the output is a buffer (not a file). I did not decide
>    yet where I will implement this function (in mcread.c, in pnga2area.c or
>    in a new file).

OK.  I included zlib uncompression in the ADDE server service routine for
the GINI ADDE server.

> As you may understand, I have chosen for the easiest possible solution
> (with the knowledge that I currently have). However, I am a bit
> concerned about the performance of my proposed solution. It would work
> well if AREA files are read once and then buffered in memory by McIDAS.
> What I am worried about is that currently AREA files are repeatedly
> opened and closed on disk, while McIDAS reads them line-by-line. In this
> case, with the PNG-decompression included, when McIDAS requests a line,
> a file will be opened, read entirely, and then decompressed into a large
> memory buffer, of which only one line is used.
> 
> My questions:
> Could you please tell me if McIDAS avoids repeatedly reading files from
> disk, by buffering files that are recently read? If not, I would have to
> reconsider my solution. 

It does _not_.  Each time a file is read, it will need to be uncompressed.

> Could you also inform me if you have a better solution, or do you have
> any suggestions to improve upon the solution mentioned in this e-mail?

One implementation of the GINI ADDE server called the uncompress procedure
to uncompress only the lines requested by the client request.  This mandated
that I be able to calculate how much of the compressed portion of the image
to skip over before beginning uncompression.  The problem I ran into was
there was no foolproof way (or at least I did not find one) to jump to
the appropriate portion of the compressed image to start.  After working on
this for over a week, I decided to implement a different scheme where I
would start decompressing at the beginning of the compressed portion
and continue only as long as needed.  This approach is efficient for client
requests for portions of the image near the beginning, and very inefficient
for client requests for portions of the image near the end.  I really worried
about the overhead of doing this until I exercised the server code thoroughly.
After testing I became convinced that overhead -- while significant -- was
not a "show stopper" (i.e., it was something that I could live with).

So, I would recommend the same approach:  only uncompress enough of the image
to satisfy the client's request for data.  Uncompressing the entire image
at the beginning would most likely be OK for small images, but it would surely
be unacceptable for large ones (like full resolution, full disk visible
images from GOES or METEOSAT which can be upwards of 300 MB).  If, however,
we can figure out exactly where to jump into the PNG-compressed portion
of the image before starting to uncompress, we would be able to only uncompress
the portion of the image that was requested.  Even this approach has its
downside:  if the user wants to display the entire image (or a "blowdown" of
the entire image), the entire image will have to be uncompressed.  This doesn
not to be done before hand, however, it can be done incrementally so that the
size of the uncompress buffer can be kept to a minimum.

> Thanks in advance,

Question:  are you actively working on this now, or are you in a thinking
phase?  You may or may not know that I am in the process of putting together
the Unidata McIDAS-X v2006 distribution.  It would be wonderful if an 
enhancement
to the AREA ADDE server could be incorporated in v2006's initial release, but
I do not think that this will be possible as I need to have something for
sites to download by the end of July/beginning of August, and I have not yet
started on the huge task of converting SSEC McIDAS documentation to use
in our website.  It seems to me, therefore, that this code modification (and
the code added by you (or Tyn?) to deal with the wavelet-compressed METEOSAT
images would be best added in the first addendum to the code.

Aside:  I am also waiting on some code from CPTEC (a division of Brazil's INPE)
that might be used in an ADDE server for imagery data in TeraScan (tm) TDF
format.  Review of this code (when I get it), turning it into a server, and
testing will surely take at least a couple of weeks, so I do not expect it
to be included in the initial v2006 release.

We will surely continue this discussion in the near future.  Please don't 
hesitate
to contact me (address@hidden) to discuss strategies and coding.

> Bas.
> --
> Ir. V. (Bas) Retsios
> Software Developer
> IT Department, Sector Remote Sensing & GIS
> International Institute for Geo-information Science and Earth Observation 
> (ITC)
> P.O. Box 6,  7500 AA Enschede, The Netherlands
> Phone +31 (0)53 4874 573, telefax +31 (0)53 4874 335
> E-mail address@hidden, Internet http://www.itc.nl

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: YEF-379651
Department: Support McIDAS
Priority: Normal
Status: Closed