20011102: ADDE access to GINI imagery

>From: "Paul L. Sirvatka" <address@hidden>
>Organization: COD
>Keywords: 200110292153.f9TLrs128399 McIDAS-X ADDE server

Paul,

>In attempting to automate everything I have run into some problems. I have
>a good handle on the scripts, but have noticed pretty strong
>inconsistencies in the availability of images.
>
>ON RTGINI, when I use ADDE.UCAR.EDU I sometimes get nothing when I run
>DSINFO IMAGES RTGINI.

This would happen right after 0Z.

>Also...do you scour the images at 00Z? If so, the 23:45 image goes away
>pretty rapidly.

The images are not scoured, but the set of images in the dataset is
changed.  We have 15 days of GINI images online on adde.ucar.edu stored
in daily directories.  In order to make access quick, we setup the
GINICOMP, GINIEAST, GINIWEST, and RTGINI datasets so that they contain
images from the "current" day only (done by changing a pointer at 0Z to
the directory containing the current day's images).  So, at 0Z, the
number of images for the "current" day drops to zero as the pointer
switches to a directory that will be filled by the data ingestion
process.  Now, one can ask to search through the full set of GINI
images (through another dataset that is the set of all days data), but
those searches use alot of machine resources, so we discourage their
use.

On a machine like papagayo.unl.edu, the number of images in each of the
above datasets is kept small and fixed.  This way, the set of images in
RTGINI, GINICOMP, GINIEAST, and GINIWEST, is always the same size.
Since you are trying to keep things as realtime as possible, I would
strongly encourage you to point to papagayo.unl.edu since the set of
images will be more like what you are looking for.

>If you have any info on why there is inconsistent availability on other
>sites as well I would appreciate it.

How a dataset is setup is a site-specific option.  Since papagayo.unl.edu
gets all of its images from adde.ucar.edu, and since the attempt on papagayo
is to keep the most recent 32 images -regardless of day/time-, you would do
better to point at it for your automated processes.  This is what I was
getting at in a previous email.

>Should I expect (if no local
>errors) the other ADDE sites to have all images in RTGINI?

No.  Again, the name of the dataset is the same, but the contents can
be very different (numbers of images).  The number of images kept
on a particular server depends on how much disk space that server has
and how many images the administrator of that server wants to keep.

>And with GINIEAST and GINIWEST?

The exact same comment(s) apply.  As far as datasets go, GINIEAST is
the set of images that a site has from NOAAPORT channel 1; GINIWEST is
the set of images a site has from NOAAPORT channel 2; GINICOMP is the
set of composite images that a site has from either or both of NOAAPORT
channels 1 and 2; RTGINI is the union of the sets GINICOMP, GINIEAST,
and GINIWEST.

>Should I use one over the other in the commands to
>get the images?

Yes, I think that papagayo.unl.edu's holdings would better suit your
purposes.

>In a related question: will the McIDAS data stream have more than an
>hourly image? 

As you know, it doesn't right now.  It _could_ have more images, but
given how easy it is to access imagery remotely (with ADDE and DODS),
and given that remote access methods do not require that full images be
sent, it is much more efficient to simply have sites use remote access
methods to get the portions of images that they are interested in from
machines that can handle the load.  The point is that a large number of
sites simply do not have the network bandwidth or storage facilities to
receive and save the images that are currently available on NOAAPORT.
Each VIS image (and there are 8 per hour: 4 from GOES-East and 4 from
GOES-West) is about 26 MB in size.  If a site was to ask for just these
two image sectors, they would have to be able to handle (receive and
store) 200 MB of images each hour.  And, this is on top of all of the
other data that they want to receive.  And, the amount of satellite
data that will be available is increasing by many orders of magnitude.
Again, the number of sites that can handle this kind of data load is
small and will be getting smaller as the data volume goes up.

Tom