20000414: satellite survey file size information (cont.)

>From: Gilbert Sebenste <address@hidden>
>Organization: NIU
>Keywords: 200004131753.LAA27350 McIDAS-X MCPATH REDIRECT BATCH

Gilbert,

(still clearing out old emails :-)

re: other sites possibly running NOAAPORT ingest systems
>At this point, I don't know of anyone else planning to get a NOAAPORT
>server running a McIDAS ADDE server, but if I do, I'll spread the word. 

OK, thanks.

>BTW, instead of hitting you all the time, I'd like to spread the load a
>little. How can I contact Jim?

As I pointed out in a previous email, in ADDE you point at one server
at a time for a praticular dataset.  The only way of spreading loads
is to have multiple sites for the same data, and have sets of sites
access different servers.

Satellite survey:

>> Datastream Options:
>> 
>> 1) Status quo: same spatial and temporal resolution of products we have
>>    now
>> 
>>    For a typical daylight hour, we would expect to see a reduction of
>>    about 2.3 MB in the hourly volume of imagery.
>> 
>> 2) Doubling temporal resolution of current GOES-East/West VIS, IR, H2O,
>>    Floater I, and Floater II products
>> 
>>    We would expect to see the hourly volume of imagery rise to about
>>    11.9 MB.  This is an increase of about 3.5 MB over what is being
>>    sent now.
>> 
>> 3) Doubling the spatial resolution of GOES-East/West VIS sectors while
>>    keeping all other products the same
>> 
>>    We would expect to see the hourly volume of imagery rise to about 15
>>    MB.  This is an increase of about 7 MB over what we are currently
>>    sending.  The compressed GOES-East/West VIS sector products sent
>>    through the IDD would each be about 6 MB.  After the VIS sectors are
>>    converted to AREA files, they would each be just over 9.6 MB in size.
>> 
>> 4) Doubling temporal resolution of current GOES-East/West VIS, IR, H2O,
>>    Floater I, and Floater II products, AND doubling the spatial
>>    resolution of the GOES-East/West VIS products
>> 
>>    We would expect to see the hourly volume of imagery rise to about 29
>>    MB.  This is an increase of about 21 MB over what we are currently
>>    sending.

>I propose option #5 or #6:
>
>5) 1 or 2 KM visible GOES-8 imagery every 15 minutes. 1 or 2 KM GOES-10
>visible every hour. Eliminate the two floater sectors UNLESS the area of
>interest is outside of the domain of either satellite. I thought the IR
>and WV were sent at full resolution already?

The IR and WV are at full spatial resolution.  The are not at full temporal
resolution.

Sending out 2 km VIS images every 15 minutes for one of the GOES
satellites would increase the volume of data in the Unidata-Wisconsin
datastream by about 22 MB per daylight hour.  Doing both satellites
would increase the volume by 44 MB per daylight hour.

Sending out 1 km VIS images every 15 minutes for one of the GOES
satellites would increase the volume of data in the Unidata-Wisconsin
datastream by about 86 MB per daylight hour.  Doing both satellites
would increase the volume by 172 MB per daylight hour.

>6) 1 KM visible from GOES-8/10 every 30 minutes. Eliminate the
>floaters. Keep everything else the same.

Same concern as above: if a 4 km VIS image will compress down to 1.5 MB,
then a 1 km VIS image would be 16 x 1.5 MB = 22 MB.  Two of these per
hour for each of the GOES satellites would increase the volume in the
Unidata-Wisconsin datastream by 84 MB (2 x (2 x 22 - 2)).

AND, the individual products are LARGE.  They will each uncompress into
AREA files that are 35 MB in size.  If a site wanted to continue to
keep 10 hours of VIS imagery for a single satellite, that site would
need 700 MB just for the VIS image!  Keep both East and West, and the
site would need 1.4 GB just for the VIS images!

>7) Pipe dream: Go ALL ADDE. Eliminate most products except 1 KM vis and 4
>KM IR from G-8 and G-10 from the datastream. Isn't that what you told me
>you wanted sometime down the line?

So, you _almost_ came around to my way of thinking.  I suggest that
the best course of action is to:

o double the temporal resolution of VIS, IR and WV images; if the
  spatial resolution of the VIS images is not enough, double it.

o keep the floaters the same or do away with them

o provide ADDE access so that sites wanting to could create their own
  sector where they want, when they want

>Frankly, the IR and WV are good enough as is, IMO.(Of course, your
>mileage may vary including those who think I can take a long walk off a
>short pier! :-) ).

Temporal resolution wise?  Or, are you talking about spatial resolution.
I don't think so since you knew that the IR and WV are already at max
resolution.

>I'm not launching a crusade against Millersville, mind you. But their
>purpose is rapidly becoming obsolete. Their goal is to provide
>high-resolution temporal data by picking areas of interest as a result of 
>limited bandwidth or, moreso these days, data access.

The original concerns were: cost, bandwidth, and processing and storage
capabilities at individual sites.  Millersville has been kind enough
to do the selections for many years.  We have had no site that was willing
to provide this service with the consistency that Millersville has been
able to provide, so we all owe thanks to those folks manning the weather
lab at MU.

>Now that we
>have advanced the Internet and McIDAS so much...even small schools could
>take the extra hit, I would think.

Unfortunately, there are still a lot of sites out there that don't
have good bandwidth or enough disk space to receive/store a lot of
data.

>With CenturyNet coming online in
>Illinois this year, even College of DuPage will get a DS-3 (I see the
>general trend; that doesn't mean smaller schools won't be as fast to get
>high bandwidth. But I surmise it won't be long). All my McIDAS stuff
>right now is going through the T1 here. With the higher res satellite,
>you could make your OWN floater sectors of anywhere you want (and that
>spills over to NIDS on 10/1/2000). After many years of hard work, my hunch
>is it will be time by early next year to thank them profusely for a job
>well done, and then retire their floater services. I assume this scenario
>saves you money that goes to SSEC too.

>I like them, but what do you think about my suggestions?

I think that for sites with LOTS of bandwidth and disk, they would work
just fine.  For "normal" sites (:-), LARGE products would be a big
problem.

>I'm not trying to
>ruffle feathers (although normally, I guess I do! :-) ); I just want to
>use what we have to it's maximum potential.

ADDE access to imagery looks to be a viable option.  I am working with
SSEC to setup access to lots of data from their realtime databases
for Unidata sites, but that will take some time.

Tom