[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
20000416: McIDAS GINI ADDE server installation at Plymouth State
- Subject: 20000416: McIDAS GINI ADDE server installation at Plymouth State
- Date: Sun, 16 Apr 2000 18:16:30 -0600
>From: Jim Koermer <address@hidden>
>Organization: Unidata Program Center
>Keywords: Plymouth NOAAPORT GINI ADDE GINI.CFG DSINFO IMGLIST IMGDISP how it
>works
Jim,
I am CCing this go Gilbert since he might be interested in how ADDE actuall
works.
As promised, the following is a blow-by-blow of how the McIDAS remote
ADDE server is setup on your machine, snow.plymouth.edu. I present the
information in what seems like the most logical way to me: a synopsis
of what happens in an ADDE transaction:
o a site learns that it may feed GINI imagery by McIDAS ADDE off or your
system. A good example of this is Gilbert's request to you and your
granting him access privilege. One part of the granting process is you
providing the end user with the name of your machine running the
McIDAS remote ADDE server; the other part is you providing that user
with the name of the dataset that he is allowed to access. In
this case, you have provided Gilbert with these two pieces of information:
machine: snow.plymouth.edu
dataset: RTGINI
ADDE has no "discovery mechanism". This means that a site can not
interrogate your system to find out what datasets you have available
through your remote ADDE server. In addition, there is no general
protocol for "pinging" random machines on the net to see if they are
running a McIDAS remote ADDE server. This is inability to discover
information can be viewed as one level of security.
o that site sets up their McIDAS or MetAPPS application to "point" to
your machine. This means that the user does the following in McIDAS:
DATALOC ADD RTGINI snow.plymouth.edu
What this means to McIDAS is that when a user runs any command that
wants to access data from a dataset called RTGINI, his McIDAS process
first asks the following question: where do I look for the dataset
RTGINI? The answer to this question is controlled in the user's own
McIDAS session through a table that is created and maintained by
McIDAS DATALOC commands. The name of this table is up to the user,
but is the one that is declared in his/her MCTABLE_WRITE environment
variable. Each McIDAS user should have this environment variable
defined to be the file MCTABLE.TXT which should exist in his/her
McIDAS-X working directory:
setenv MCDATA /home/user/mcidas/data
setenv MCTABLE_WRITE "${MCDATA}/MCTABLE.TXT"
Again, this is the table that the user 'user' can write.
To complicate things a little, however, McIDAS has the notion of user
and site defined DATALOCation tables. The set of DATALOCation tables
is defined by each user in the environment variable MCTABLE_READ in
his/her session. MCTABLE_READ can contain one or more DATALOCation
tables, and they will be searched in order, from left to right, when
trying to find the machine to request data from when the user asks
for data from a dataset. The typical setup for MCTABLE_READ is as
follows:
setenv MCTABLE_READ "${MCDATA}/MCTABLE.TXT;/home/mcidas/data/ADDESITE.TXT"
The user's own DATALOCation table is searched first, and then the one
setup for a site by the McIDAS superuser, mcidas
(~mcidas/data/ADDESITE.TXT).
o the user asks his/her McIDAS session to list back the contents of
the dataset RTGINI:
DSINFO ALL RTGINI
Data Set INFO (DSINFO) makes the request: list back ALL types of data
in the dataset called RTGINI. The first thing that happens in this
query is a search through files listed in the user's MCTABLE_READ for
the location of the dataset RTGINI. Since this was specified by the
user to be snow.plymouth.edu, your system is contacted and asked.
Now, which port is contacted on your system is a function of the
definition of an environment variable named MCCOMPRESS in the user's
McIDAS-X session. If this environment variable exists (is defined to
be anything), then the request goes to port 503 on your machine. If
MCCOMPRESS does not exist in the user's session, the request goes to
port 500.
The difference in the port access is as you might expect: requests
sent to port 500 will result in transfer of information that is not
compressed. Requests sent to port 503 results in transfers of
invormation that are compressed on-the-fly. The great, great
majority of requests will go to port 503 as the transfer time is
greatly reduced when the data sent back is compressed.
o your system receives a request on port 503 (assuming MCCOMPRESS
is defined in the requesting user's session) which inetd has
to react to.
o inetd looks in /etc/inetd.conf to see what it is supposed to do with
the request on port 503. Since the remote ADDE server was setup on
your machine, and since TCP wrappers were setup on the McIDAS
inetd services, your inetd.conf will have lines looking like:
mcserv stream tcp nowait mcadde /usr/local/libexec/tcpd
/home/mcidas/bin/mcservsh -H /home/mcidas
mccompress stream tcp nowait mcadde /usr/local/libexec/tcpd
/home/mcidas/bin/mcservsh -H /home/mcidas
These lines would look a little different on different machines since
the location of the TCP wrapper will be different.
The TCP wrapper routine, tcpd, is called to evaluate the request on
port 503. If the IP number of the machine making the request is not
allowed in the TCP wrapper configuration file,
hosts.allow/hosts.deny, nothing more is done. If the machine making
the request is NOT denied, then the TCP wrapper runs the McIDAS ADDE
Bourne shell script, /home/mcidas/bin/mcservsh.
o /home/mcidas/bin/mcservsh tests for the existence of the file .mcenv
in the McINST_ROOT directory on the machine. If .mcenv exists, it
is read so that McIDAS environment variable definitions can become
active. All McIDAS routines need certain environment variables set
in order to run correctly. .mcenv on your system is located in the
HOME directory of the user 'mcadde' (which just so happens to be
the same as the HOME directory of the user 'mcidas'; this is by design).
On your system, this looks like:
umask 002
MCDATA=/home/mcidas/workdata
MCPATH=${MCDATA}:/home/mcidas/data:/home/mcidas/help
MCGUI=/home/mcidas/bin
MCTABLE_READ="/home/mcidas/mcidas/data/MCTABLE.TXT"
MCTABLE_WRITE="/home/mcidas/mcidas/data/MCTABLE.TXT"
PATH=${MCGUI}:/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin:/sbin:/usr/sbin
export MCDATA MCPATH MCGUI MCTABLE_READ MCTABLE_WRITE PATH
cd $MCDATA
o /home/mcidas/bin/mcservsh then 'exec's $McINST_ROOT/bin/mcserv, a compiled
McIDAS executable. 'mcserv' then interprets the request that was sent
from the user's machine to see what kind of McIDAS data it is asking
for. In the example we are working on, the request is to list back
the types of data in the RTGINI dataset. So, if RTGINI is not defined
on your system, an error message will be sent back and the transation
will be terminated. On your system, RTGINI is defined, so the return
will look like:
Dataset Names of Type: IMAGE in Group: RTGINI
Name NumPos Content
------------ ------ --------------------------------------
GAN8KIR 9999 GINI 8 km 10.7 um Alaska National
GAN8KVIS 9999 GINI 8 km Visible Alaska National
GAN8KWV 9999 GINI 8 km WV Alaska National
GAR16KWV 9999 GINI 16 km WV Alaska Regional
GAR2KVIS 9999 GINI 2 km VIS Alaska Regional
GAR8K12 9999 GINI 8 km 12.0 um Alaska Regional
GAR8K39 9999 GINI 8 km 3.9 um Alaska Regional
GAR8KIR 9999 GINI 8 km 10.7 um Alaska Regional
GE1KVIS 9999 GINI 1 km VIS East CONUS
GE2KVIS 9999 GINI 2 km VIS East CONUS
GE4K12 9999 GINI 4 km 12.0 um East CONUS
GE4K39 9999 GINI 4 km 3.9 um East CONUS
GE4KIR 9999 GINI 4 km 10.7 um East CONUS
GE4KVIS 9999 GINI 4 km VIS East CONUS
GE8KWV 9999 GINI 8 km WV East CONUS
GHN14KIR 9999 GINI 14 km IR Hawaii National
GHN14KVIS 9999 GINI 14 km VIS Hawaii National
GHN14KWV 9999 GINI 14 km WV Hawaii National
GHR1KVIS 9999 GINI 1 km VIS Hawaii Regional
GHR4K12 9999 GINI 4 km 12.0 Hawaii Regional
GHR4K39 9999 GINI 4 km 3.9 um Hawaii Regional
GHR4KIR 9999 GINI 4 km 10.7 um Hawaii Regional
GHR8KWV 9999 GINI 8 km WV Hawaii Regional
GNC24K12 9999 GINI 24 km 12.0 um Nhem-Composite
GNC24K39 9999 GINI 24 km 3.9 um Nhem-Composite
GNC24KIR 9999 GINI 24 km 10.7 um Nhem-Composite
GNC24KVIS 9999 GINI 24 km VIS Nhem-Composite
GNC24KWV 9999 GINI 24 km WV Nhem-Composite
GPN8KIR 9999 GINI 8 km 10.7 um Puerto Rico National
GPN8KVIS 9999 GINI 8 km VIS Puerto Rico National
GPN8KWV 9999 GINI 8 km WV Puerto Rico National
GPR1KVIS 9999 GINI 1 km VIS Puerto Rico Regional
GPR4K12 9999 GINI 4 km 12.0 um Puerto Rico Regional
GPR4K39 9999 GINI 4 km 3.9 um Puerto Rico Regional
GPR4KIR 9999 GINI 4 km 10.7 um Puerto Rico Regional
GPR8KWV 9999 GINI 8 km WV Puerto Rico Regional
GSN8K12 9999 GINI 8 km 12.0 um Super-National
GSN8K39 9999 GINI 8 km 3.9 um Super-National
GSN8KIR 9999 GINI 8 km 10.7 um Super-National
GSN8KVIS 9999 GINI 8 km VIS Super-National
GSN8KWV 9999 GINI 8 km WV Super-National
GW1KVIS 9999 GINI 1 km VIS West CONUS
GW2KVIS 9999 GINI 2 km VIS West CONUS
GW4K12 9999 GINI 4 km 12.0 um West CONUS
GW4K39 9999 GINI 4 km 3.9 um West CONUS
GW4KIR 9999 GINI 4 km 10.7 um West CONUS
GW4KVIS 9999 GINI 4 km VIS West CONUS
GW8KWV 9999 GINI 8 km WV West CONUS
No Datasets found of Type: POINT in Group: RTGINI
No Datasets found of Type: VPROF in Group: RTGINI
No Datasets found of Type: GRID in Group: RTGINI
No Datasets found of Type: TEXT in Group: RTGINI
No Datasets found of Type: NAV in Group: RTGINI
DSINFO -- done
The only kind of data that was found to be defined in your RTGINI was
data of type IMAGE. No POINT, VPROF, GRID, TEXT, or NAV data was
found in RTGINI.
The listing shows not only that there was data of type image
in RTGINI, but it shows the group names of that data: GAN8KIR, GAN8KVIS,
GAN8KWV, GAR16KWV, etc.
The datasets defined on your system and the groups in those datasets
are defined in the file RESOLV.SRV, which can exist in any directory
in the MCPATH set of directories defined for the server processes.
MCPATH is one of the things defined in the .mcenv file that was
read by the Bourne shell script, mcservsh. RESOLV.SRV is typically
found in the first directory in MCPATH. That directory for Unidata
sites is defined by the environment variable MCDATA. This would
be the /home/mcidas/workdata directory in your case (and 99.99% of
all others).
o since the user found that:
o you really are running an ADDE server (i.e., the DSINFO request
did not time out with no response)
o you have a dataset named RTGINI defined on your system
o there is a group named GW1KVIS in RTGINI on your system
S/he might decide to take a look at what is in that dataset/group.
S/he could list the contents of that dataset/group:
IMGLIST RTGINI/GW1KVIS.ALL
(the .ALL says to list ALL of the contents of RTGINI/GW1KVIS)
or s/he might decide to display the most recent item in the
RTGINI/GW1KVIS dataset/group:
IMGDISP RTGINI/GW1KVIS STA=KDEN EU=IMAGE SF=YES REFRESH='EG;MAP h'
o what actually happens when the user asks for a listing of data that
is available or a display of the latest item in the dataset is:
o his/her DATALOC tables are searched to find out which machine to
send the request to
o the serving machine is contacted
o inetd either fires up /home/mcidas/bin/mcservsh directly, or it
fires up a TCP wrapper that, if the requesting machine is not
denied/is allowed, fires up /home/mcidas/bin/mcservsh.
o mcservsh execs /home/mcidas/bin/mcserv
o mcserv interprets the request to see what kind of data is being
asked for. It does this by searching RESOLV.SRV to find the
requested dataset/group. If it finds it, it will know what
type of data is being asked for. In this case, the type of the
data is IMAGE
o mcserv execs either /home/mcidas/bin/adirserv (for lists) or
/home/mcidas/bin/agetserv (for displays)
o adirserv/agetserv look at the request to see if the type of
IMAGE data being requested is of type AREA or something else.
In this case, it is of type GINI, so adirserv/agetserv know
that a different process must handle the request
o adirserv/agetserv execs either giniadir (for lists) or giniaget
(for displays) since the type of IMAGE data requested was GINI.
The name of the routine that is execed is put together on the
fly from the type of data (lower cased) plus adir/aget. So,
the name of the routine that is being run is giniadir or giniaget
depending on the function requested
o giniadir/giniaget startup and interpret the request that was made of
them. One part of the request is information from the requesting
process (i.e., from the user asking for a list/display), the other
part of the request is information from the dataset definition
file on the server machine, RESOLV.SRV. For GINI imagery, one
piece of the information in RESOLV.SRV is the name of a configuration
file that giniadir/giniaget must read in order to figure out
how to satisfy the user's request.
The configuration file specified in your remote ADDE GINI server is
/home/mcidas/workdata/GINI.CFG. GINI.CFG contains the following
kinds of information:
o where are data files to be found (DIRMASK)
o how are data files named (FILEMASH)
o who is allowed to request data (IPMASK)
The format of the Unidata ADDE configuration files of which GINI.CFG
is one example is:
DIRMASK= <regular expression>
FILEMASK= <regular expression>
IPMASK= <regular expression>
If the location and naming of data files that giniadir/giniaget is
simple enough, the above three variables would be all that is
needed. On your system, for example, the location of GOES-East files
is /data/noaaport2/sat; the location for GOES-West files is
/data/noaaport3/sat (I may have these reversed). The files locations
can, therefore, be expressed by the regular expression:
DIRMASK=/data/noaaport[23]/sat
I can also specify the set of IP addresses of machines allowed to
make requests by specifying a simple or complex regular expression
for IPMASK. Right now, IPMASK on your system is defined as:
IPMASK=*
All requestors are allowed.
The names of the files that represent the various groups in the RTGINI
dataset are not nearly so simple, however. GOES-East 1 km Visible
imagery files look like 0004162115_gecvi.sat, GOES-West 3.9 um
IR images look like 0004162045_gwci3.sat, and so on.
You may think that one could simply specify FILEMASK as '*' and
find the appropriate files (since '*' matches everything) and you
would be correct. The object, however, is to search as few files
as possible in order to fulfill a user's request. For this reason,
I added some shortcuts that can be used in the configuration
file. One of the shortcuts is the use of the group portion of
the dataset/group name of the data requested in the definitions.
Here is an example:
GW1KVIS_FILE=*_gwcvi.sat
This is of the form: \TYPE_FILE where \TYPE represents the group
portion of the dataset/group name. This form of file name specification
takes precedence over the FILEMASK= specifier in the configuration
file.
To generalize things, I added the capability to define group specific
locations, names, and access privilege. These definitions all take
precedence over their counterparts DIRMASK, FILEMASK, IPMASK. What I
mean by this is the following: If you chose to store GOES-West 3.9
um IR images in the /data/gini/goeswest/3.9 directory, and you chose
to name them YYMMDDHHMM_goesW_3.9.sat, and you chose to make these
files only available to a machine with IP address 128.117.140.56,
then you would define the following in GINI.CFG to locate the files:
GW1KVIS_DIR=/data/gini/goeswest/3.9
GW1KVIS_FILE=*_goesW_3.9.sat
GW1KVIS_IP=128.117.140.56
This will be used even though you might have defined:
DIRMASK=/data/noaaport[23]/sat
FILEMASK=*
IPMASK=*
since the \TYPE_DIR, \TYPE_FILE, and \TYPE_IP forms take precedence
over the DIRMASK, FILEMASK, and IPMASK forms, respectively.
With the setup of GINI.CFG on your system, when the request from the
user comes in to display the most recent RTGINI/GW1KVIS image, the
GINI ADDE server, giniaget, reads GINI.CFG and finds that:
o it should look in either /data/noaaport2/sat or /data/noaaport3/sat
(DIRMASK=/data/noaaport[23]/sat)
o it should look for files named *_gwcvi.sat (from GW1KVIS_FILE=*_gwcvi.sat)
A regular expression describing the fully qualified name of the desired
data is then formed from the combination of DIRMASK and FILEMASK
settings:
name == /data/noaaport[23]/sat/*_gwcvi.sat
Unix globbing is then used to generate a list of files matching this
expression in the exact same way that an 'ls' does. The files in
this list are then opened, headers read, and date/time sorted so
that the most recent image can be returned back to the user.
Phew! Seems kinda complicated, but the entire process is incredibly
flexible.
To answer a simple question like 'how do I limit ADDE access to my
machine' is at least two fold:
1) define the set of IP address allowed to request data from you
ADDE server. This will limit this set of IP addresses for ALL
datasets.
2) define the set of IP addresses allowed to request data from
your RTGINI dataset through the IPMASK setting in the GINI.CFG
configuration file. This is much more flexible that the use
of TCP wrappers since the control is on a per-dataset basis.
I sincerely hope that the presentation above has not made the entire
process more complicated that it really is. I felt, however, that it
was necessary to give you the complete overview so you would have
a feel for exactly what is going on behind the scenes on your machine
as users come in and browse your GINI data.
Please let me know if/when you have any questions about the process.
Tom
>From address@hidden Sun Apr 16 22:22:51 2000
>Subject: Re: 20000416: McIDAS GINI ADDE server installation at Plymouth State
>Tom,
>Thanks! Interesting. But that means that, if I read it correctly, I can
>only be pointed to one RTGINI server at a time. If UNIDATA is faster than
>Plymouth, say...I will have to delete the IP address and add the new one
>(or name). No problem, I was just curious.
>Thanks again. 3 ADDE servers...hmmm. I will check on UIUC, but I doubt if
>they will get one (their own NOAAPORT feed).
>Gilbert
>From address@hidden Wed Apr 19 09:55:40 2000
>Subject: Re: 20000416: McIDAS GINI ADDE server installation at Plymouth State
Tom,
Thanks for all the info about what you did to install ADDE. Hopefully,
I'll have time to digest all of it. When you do announce this to the
Unidata community, you may want to mention our web site at the following
address:
http://snow.plymouth.edu/mcidas_sat.html
so that they can get an idea of the all GINI data that are available. It
may entice some non-ADDE sites to get going on ADDE.
Here, we just finished a successful search for a new synoptic person.
I'll be glad when he gets here for the fall semester. It should provide
us with a person that will have a students more involved with the
various weather visualization packages that we have available.
Jim
--
James P. Koermer E-Mail: address@hidden
Professor of Meteorology Office Phone: (603)535-2574
Natural Science Department Office Fax: (603)535-2723
Plymouth State College WWW: http://vortex.plymouth.edu/
Plymouth, NH 03264