20051020: zlibg2gini for NOAAPORT sounder images (cont.)

>From: "Jordan Gerth" <address@hidden>
>Organization: SSEC
>Keywords: 200510191601.j9JG1O7s009290 NOAAPORT AWIPS GINI 

Hi Jordan,

>We are using RedHat Enterprise.  We were able to get a 
>build of zlibg2gini compiled this morning.  However, we 
>continue to notice that the size being reported in the 
>zlib debug logs is greater than the size of the output 
>file according to the system (via 'ls -lart').

I just verified this on two different operating systems:  Sun Solaris
5.9 SPARC and Fedora Core 4 Linux.

>This size 
>reporting difference is enough to cause some products 
>(such as satellite sounder and northern hemisphere 
>imagery) to be rejected from our ingest queue for AWIPS.

OK.

>Do you know of any differences between the satellite data 
>for the sounder and northern hemisphere as opposed to 
>other products (such as those for CONUS, TIGE04, etc.) 
>that would be causing our trouble?

I see that the big difference between the sounder and non-sounder
images is that the last zlib-compressed chunk for sounder images
uncompresses into a number of lines that when added to the cumulative
total number of lines uncompressed exceeds the number of lines in the
GINI image header.  zlibg2gini.c contains code that keeps track of the
number of lines uncompressed and exits when the running total exceeds
the number indicated in the header.  This has the effect of throwing
away the last few lines in an image.

>Any solutions?

My first inclination would be to exit the uncompression look in
zlibg2gini after writing the last uncompressed chunk to the output file
even though the number of lines written will exceed the number
indicated in the header.  Since you now have the ability to build the
executable, you could try the following:

in zlibg2gini.c change:

    do {

      intot  += d_stream.total_in;
      outtot += d_stream.total_out;

      nline   = d_stream.total_out / elems;
      totlin += nline;

      if ( totlin > lines ) break;

      write( fd_out, uncompr, d_stream.total_out );

      udebug( "bytes in: %4d, bytes out: %4d, nlines = %2d, total lines = %4d",
               d_stream.total_in, d_stream.total_out, nline, totlin );


to:

    do {

      intot  += d_stream.total_in;
      outtot += d_stream.total_out;

      nline   = d_stream.total_out / elems;
      totlin += nline;

      write( fd_out, uncompr, d_stream.total_out );

      udebug( "bytes in: %4d, bytes out: %4d, nlines = %2d, total lines = %4d",
               d_stream.total_in, d_stream.total_out, nline, totlin );

      if ( totlin > lines ) break;


Rebuild the zlibg2gini executable, uncompress some NOAAPORT-ingested
GINI images, and compare the output size on disk to that reported.
Finally, see if AWIPS "likes" the image even though it contains more
lines than are indicated in the header.

My second thought would be to change the glibg2gini.c code to write
out just the portion of the last uncompressed record that is enough
to make an image with the same number of lines indicated in the
image header.  In this case, the code change should look something
like:

change:

    do {

      intot  += d_stream.total_in;
      outtot += d_stream.total_out;

      nline   = d_stream.total_out / elems;
      totlin += nline;

      if ( totlin > lines ) break;

      write( fd_out, uncompr, d_stream.total_out );

      udebug( "bytes in: %4d, bytes out: %4d, nlines = %2d, total lines = %4d",
               d_stream.total_in, d_stream.total_out, nline, totlin );


to:

    do {

      intot += d_stream.total_in;
      nline  = d_stream.total_out / elems;

      if ( (totlin+nline) > lines ) {

        nline               = lines - totlin;
        d_stream.total_out  = nline * elems;
        outtot             += d_stream.total_out;
        totlin              = lines;

        write( fd_out, uncompr, d_stream.total_out );
        udebug( "bytes in: %4d, bytes out: %4d, nlines = %2d, total lines = 
%4d",
                 d_stream.total_in, d_stream.total_out, nline, totlin );
        break;

      } else {

        outtot += d_stream.total_out;
        totlin += nline;

        write( fd_out, uncompr, d_stream.total_out );

        udebug( "bytes in: %4d, bytes out: %4d, nlines = %2d, total lines = 
%4d",
                 d_stream.total_in, d_stream.total_out, nline, totlin );
      }


The real question is why the last bit uncompresses to more lines than
it should!?

>Thanks,

No worries.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.