20000208: More on ADDE from MSFC and imagery in the UW stream

>From: Gilbert Sebenste <address@hidden>
>Organization: NIU
>Keywords: 200001311730.KAA02738 McIDAS ADDE accounting

Gilbert,

I am responding to your last question about loading loops of images.

>Thanks again for letting me have access to the images! They are WAY cool!
>
>I was wondering if somehow, they can be shared with the UNIDATA community.
>Now that the data is "compressed" on the fly, how much of a bandwidth
>issue is it? I would love to have other Universities share in what you've
>given me...to give students an opportunity to do some mesometeorology
>using satellite data. On the regular UNIDATA datastream, you can't do it
>well, since the images only come across once an hour, and the resolution
>is 4 KM for visible images.
>
>Of course, I don't want to annhilate your server or network either...but I
>would assume that they wouldn't be used continuously. And even if they
>are, how much of a bandwidth issue is it, either way?
>
>Hopefully the compression will help on your end with your users, in
>bringing up the data faster. It definitely helps here...when I turn
>compression off, there is a HUGE difference in download times for each
>image.
>
>Second question...is there a way to bring up, say, the last 6 images for a
>loop using IMGDISP? Is there a command like, say, "IMGLOOP" to do this?

Yes, IMGDISP has the ALL= keyword that allows you to load several images.
As an example:

IMGDISP RTIMAGES/GE-IR STA=KMIA MAG=2 EU=IMAGE SF=YES REFRESH='EG (GRA);MAP H 
GRA=(GRA)' ALL=1 10

This command will load 10 images from the RTIMAGES/GE-IR dataset starting
in frame 1 and continuing through frame 10.  The load will be centered
on Miami, FL and blown up by a factor of two.  The IMAGE enhancement will
be used for each frame and a high resolution map will be drawn on top of
the image in each frame.

Some important items in the above command line:

o ALL=n m  - check out the online McIDAS help for IMGDISP or review the
  manual entry for it in the online McIDAS-X Users Guide

o SF=YES   - force flipping to the frame after the image is displayed;
  if you didn't do this; and if you forgot to tell the EG and MAP commands
  where to do their thing, then the erasure and map drawing would be done
  on the frame that you are currently looking at AND that frame would
  not change

o EG (GRA) - says to erase the current graphic frame; this changes as
  each image gets loaded by virtue of the SF=YES keyword

o MAP H GRA=(GRA) - tell MAP explicitly which frame to draw the map on

The shortcomings of IMGDISP for loading loops are:

o the loop bounds do not get set like they did in ALOOP

o since the loop bounds do not get set, the dwell rates also do not
  get set

o if you have fewer than 10 images, then only the first 'n' (where 'n'
  is the number you actually have) will get loaded; this makes it difficult
  to make a generic construct like 'ALL=1 10;LB=1 10;DR 9*3 10' since
  you don't know apriori that all of frames 1-10 should be included in a loop

We have at various times considered adding logic to IMGDISP to be aware
of how many frames were loaded and to set loop bounds and dwell rates.
The reason that we have not is that we are trying to decrease the number
of changes we have to make to the SSEC distribution each time we get one.
Given that there is now only me doing McIDAS support at Unidata, I have
to be a lot more careful on how much I hack into the SSEC code since each
new hack hardly ever goes away (read this to mean that SSEC hardly ever
picks up the mods that we make to McIDAS (a bit of an overstatement, but
reasonably true)).
        
>Muchas gracias and keep up the great work! I greatly appreciate it!!!

So, the big carrot for you in the MSFC imagery is the temporal
resolution for all imagery and higher spatial resolution for visible
imagery?  It must be since the IR and WV imagery that we send the
broadcast are at full resolution (modulo the removal of the 1.7 times
oversampling in the element dimension).

How many sites, with their current internet capabilities, do you think
could ingest twice the number of image products than they are currently
getting?  Also, how many sites do you think could ingest visible images
that are 4 (2 km res.) or 16 (1 km res.) times the size of the current
images?  I do not ask this to be argumentative, but, rather, to get one
user's perspective how much data can effectively be handled in the
current IDD.

Our feeling has been that the great majority of sites are having
problems getting the images in its current size.  To put some numbers
on these products, a current VIS GOES East of West image in the
broadcast is anywhere from 1.2 to 1.8 MB (assuming that it is not dark;
those images compress really well).  Is uncompressed size is on the
order of 2.3 MB.  From these numbers, you can see that we are only
getting a compression ratio of 75%.  Some experimentation at the UPC by
Steve Chiswell has shown that use of PNG compression can give us much
better compression numbers; broadcast products on the order of 35-40%
of their original size.

Even with these savings, this would translate into VIS products of the
following size:

resolution   product size   Comments
------------+--------------+---------
4 km         840 KB         current resolution
2 km         3.3 MB         double LINe and ELEment resolution
1 km        13.3 MB         quadruple LINe and ELEment resolution

Of course, this is the worst case scenario.  IR imagery is already being
sent out at its maximum resolution, and it compresses better than VIS
imagery.

So, what to do?

The first steps that we will be taking are:

o PNG compress new imagery as it is added to the Unidata-Wisconsin
  datastream

o after sites get PNG uncompression routines in place (e.g, a new ldm-mcidas
  decoder), switch the old images from delta encoding (the old standard
  McIDAS comprssion) to PNG compression

o figure out which is more important to the typical user:

  o the same images at twice the temporal resolution
  o bigger images at the same temporal resolution
  o bigger images at twice the temporal resolution
  o something different (ADDE access to high resolution imagery
    at top tier IDD sites?)

The ADDE option is attractive from the standpoint of allowing users to
go get as much imagery as they can handle, but has the disadvantage of
forcing sites not using McIDAS to use it to get desired imagery.  This
may be a bitter pill for some sites to swallow, but I don't know for
sure.

OK, enough for now.  Please let me know what you think about the above
issues.  The will be useful for the next User's Committee Meeting.

Tom

>From address@hidden  Tue Feb  8 13:34:00 2000
 
> So, the big carrot for you in the MSFC imagery is the temporal
> resolution for all imagery and higher spatial resolution for visible
> imagery?  It must be since the IR and WV imagery that we send the
> broadcast are at full resolution (modulo the removal of the 1.7 times
> oversampling in the element dimension).

You are correct! IF, and I repeat, IF we had to have a choice between 
temporal resolution and spatial resolution, temporal would be first,
followed by spatial. That said, however, spatial is a VERY close second.

> How many sites, with their current internet capabilities, do you think
> could ingest twice the number of image products than they are currently
> getting?  Also, how many sites do you think could ingest visible images
> that are 4 (2 km res.) or 16 (1 km res.) times the size of the current
> images?  I do not ask this to be argumentative, but, rather, to get one
> user's perspective how much data can effectively be handled in the
> current IDD.

I understand, and you have valid points. We could go 1 KM, but not at 15
min intervals. We could if it were 4 KM. But that is *current*. When we
get our OC-3 connection, we'll take 1 KM vis *global* :-)

COD is about to upgrade to a full T3, and will get a DS-3 by the end of
the year. By demand, I think in the next year, most schools will upgrade
(we did by adding the equivalent of two T1's at NIU just before Christmas,
just to stay with the curve) to do everything from Internet broadcasting,
etc. 

Just as the NCEP feed is for high-bandwidth folks like you, SSEC, UIUC...I
forsee, in the next year, a ton of schools going high-bandwidth as well to
support distance learning (which is why NIU is getting the OC-3). 

I think that a 4 KM vis every 15 minutes would be taken aby a number of
schools...and more would be added in the next year or two.

 > Our feeling has been that the great majority of sites are having
> problems getting the images in its current size.

Ask. Put out a blanket request on the mcidas users email groups. Of course
there will be those who are having trouble...but if you do this:

> Even with these savings, this would translate into VIS products of the
> following size:
> 
> resolution   product size   Comments
> ------------+--------------+---------
> 4 km         840 KB         current resolution
> 2 km         3.3 MB         double LINe and ELEment resolution
> 1 km        13.3 MB         quadruple LINe and ELEment resolution

I think you'd see people wanting 2 KM vis every 15 minutes. That's my
guess.
 
> Of course, this is the worst case scenario.  IR imagery is already being
> sent out at its maximum resolution, and it compresses better than VIS
> imagery.

Right.
 
> So, what to do?
> 
> The first steps that we will be taking are:
> 
> o PNG compress new imagery as it is added to the Unidata-Wisconsin
>   datastream

Great.
 
> o after sites get PNG uncompression routines in place (e.g, a new ldm-mcidas
>   decoder), switch the old images from delta encoding (the old standard
>   McIDAS comprssion) to PNG compression

OK..
 
> o figure out which is more important to the typical user:
> 
>   o the same images at twice the temporal resolution
>   o bigger images at the same temporal resolution
>   o bigger images at twice the temporal resolution
>   o something different (ADDE access to high resolution imagery
>     at top tier IDD sites?)

Ask, ask, ask. Just do a straw poll to get a "feel" for what they want.
Don't ask to say what they think others want; I know COD would drool over
1 KM 15 minute imagery, but that doesn't mean they can have it in AREA
files over the IDD. Let the masses speak! If nothing else, ADDE access to
high-res imagery is my solution for here, for now, until the OC-3 hits.
And even then, guess what? I can't have a dish, so the only way I could be
a relay then is if it could be transported via the LDM. IE, send the other
NOAAPORT channels my way. As I said, my solution now is to grab it using
ADDE.
 
> The ADDE option is attractive from the standpoint of allowing users to
> go get as much imagery as they can handle, but has the disadvantage of
> forcing sites not using McIDAS to use it to get desired imagery.  This
> may be a bitter pill for some sites to swallow, but I don't know for
> sure.

Try it. This supports YOUR position, and may prolong the life of McIDAS
as a UNIDATA supported platform in the long run. Could an ADDE-type system
be built into the new Java stuff you all are working on? I would assume
so, and would hope so. It's supposed to be the next great software from
UNIDATA, so it must do everything McIDAS and WXP can, and more...and
more conveniently!

> OK, enough for now.  Please let me know what you think about the above
> issues.  The will be useful for the next User's Committee Meeting.

That's my feeling. Ask the entire community. I can only speak for NIU, and
I know that the met program here wants 1 KM vis. They saw it in my office,
and nearly freaked. When they get their new system in, they'll want an
install from you...right now, they (along with COD) have no sysadmin...and
they'll want access to the ADDE servers at UNIDATA and NASA. My HUNCH: 4
KM vis, every 15 minutes. Their DESIRE: 1 KM res, every 15 minutes, and
every 5 minutes in RSO.

BTW, how exactly ARE you feeding that stuff to ADDE.UNIDATA.UCAR.EDU?
And how can McIDAS read it? I thought it could only read AREA files...

*******************************************************************************
Gilbert Sebenste                                                     ********
Internet: address@hidden    (My opinions only!)                     ******
Staff Meteorologist, Northern Illinois University                      ****
E-mail: address@hidden                                 ***
web: http://weather.admin.niu.edu                                      **
Work phone: 815-753-5492                                                *
*******************************************************************************