19991119: A quick overview of how ADDE works

>From: "Jennie L. Moody" <address@hidden>
>Organization: University of Virginia
>Keywords: 199911182049.NAA16853 McIDAS ADDE

Jennie,

re: my comments on the use of ADDESITE.TXT

>I find this confusing, I still struggle to follow client/server,
>server side and client routing tables, etc. Damn.

I give what I feel to be an illustrative example (with explanations)
below.  I hope that this will help to give you an overview of how
ADDE actually works.

re: web page listing of what .mcenv should contain
>right I saw this last night.  But I couldn't figure out.  I mean the
>words are simple enough:

>> Here, ... should be replace by the same directories you added to PATH above.
>  
>> 
>> Also, MCTABLE_READ and MCTABLE_WRITE are being set to non-existent
>> files so that the remote server being run by mcadde will always access
>> data locally (otherwise a circular definition would occur)."
>
>but I don't know what it *means* for a remote server to "always access 
>data locally"......

OK, this comment helps me focus my example so that you will have a
good feeling for how things work.

>I am starting to gather that what this must
>mean is that since these environment variables don't exist, the
>data definitions (ie:  the names of datasets defined on this
>server) will be found because the MCDATA environment includes a
>search PATH that includes /home/mcidas/data (which points to 
>mcidas76/data, which now included ADDESITE.TXT, because I moved it
>there).  

Almost, but not quite.  The dataset definitions (i.e. the correspondence
between an ADDE data set like RTIMAGES/GE-IR and the image files that
it is composed of) are contained in the file RESOLV.SRV.  This file
will be located in one of the directories contained in the MCPATH for
the user that is the remote ADDE server (i.e. 'mcadde').  More on this
below.

>So, let take an example.  If you are the client (you issue a command
>on your mcidas session to display something in a dataset that is
>defined on my machine, lets imagine for simplicity, that its
>a dataset that only exist on my machine, so there isn't more than
>one place you can find it (eg., its not RTGRIDS, by Jennie/GRIDS
>or something).

OK, let's take a concrete example.  You tell me that you have a dataset
with a group name of OZONE that is accessible on your machine,
windfall.evsc.virginia.edu.  In order for me to access the grids
in this dataset, I will need to tell my McIDAS session to look on
your machine when I request data from it.  I do this by running
the DATALOC command in my session:

DATALOC ADD OZONE windfall.evsc.virginia.edu

This information is written into the file that is specified in the
environment variable MCTABLE_WRITE in my session.

The example in the online documentation has the user 'mcidas' define
this to be:

  setenv MCTABLE_WRITE "/home/mcidas/data/ADDESITE.TXT"

So, if I was the user 'mcidas', I would have just written some information
into /home/mcidas/data/ADDESITE.TXT.

If I was any user except 'mcidas', the online instructions specify
MCTABLE_WRITE to be:

  setenv MCTABLE_WRITE "${MCDATA}/MCTABLE.TXT"

where MCDATA was defined to be:

  setenv MCDATA ${HOME}/mcidas/data

So, if I was the user 'yoksas', I would have just written into the file
/home/yoksas/mcidas/data/MCTABLE.TXT.

Now, when I run a McIDAS command that requests data from this dataset, McIDAS
applications in my session try to figure out where they should go look for
the data in the dataset whose group name is OZONE.  Where McIDAS looks
is defined by the environment variable MCTABLE_READ.

MCTABLE_READ is a semi-colon list of fully qualified pathnames of files
in which to look.

For the user 'mcidas', the online documentation recommends setting
MCTABLE_READ as:

  setenv MCTABLE_READ "${MCDATA}/MCTABLE.TXT;/home/mcidas/data/ADDESITE.TXT"

For any other user, the online documentation gives this as an example
MCTABLE_READ search path list:

  setenv MCTABLE_READ "${MCDATA}/MCTABLE.TXT;/home/mcidas/data/ADDESITE.TXT"

Notice that these definitions are the same!  They say that for any user,
McIDAS applications should look in the files ${MCDATA}/MCTABLE.TXT
and /home/mcidas/data/ADDESITE.TXT to see where the dataset requested is
to be found.  McIDAS routines look first in ${MCDATA}/MCTABLE.TXT and
if they don't find the dataset group name they are looking for, the
next look in /home/mcidas/data/ADDESITE.TXT.  If they don't find the
requested data set in either file, then they assume that the dataset
is local (i.e. LOCAL-DATA).

The idea behind the list of files in MCTABLE_READ is so that a user
can have his/her own set of DATALOCations and there can be a global
one that is setup by the McIDAS system administrator (i.e. the user
'mcidas').  All users are allowed to only write into one DATALOCation
table: The non-'mcidas' user is allowed to write only into
~user/mcidas/data/MCTABLE.TXT, and 'mcidas' is allowed to only write into 
/home/mcidas/data/ADDESITE.TXT.

Does the above make sense?  If not, continuing will not be beneficial.

>You (the client) send out some request to
>use a grid in this dataset, and you find (resolve?) on your own 
>(client side routing table?) end that the dataset is sitting on 
>windfall.

Yup.  Again, that information comes from one of the files in your
MCTABLE_READ environment variable.

>So, you (the client) send out an inetd process that runs 
>a mcservsh?

If the dataset is on a remote machine, then yes, a request is sent
to the remote machine.  The request is sent on either port 500 or
port 503.  The decision as to which port to send the request to
is governed by the MCCOMPRESS environment variable that may or may
not be defined in the user's environment.  If MCCOMPRESS is defined,
the request is sent on port 503 and the data that is sent is compressed
by the server and uncompressed by the client on-the-fly.  Otherwise,
the request goes to port 500 on the remote machine and the data is
not compressed.

>on my (server) end,

Yup.  The process is that a request is set to either port 500 or 503
(see above).  inetd on your machine then looks in a table (/etc/inetd.conf)
to see what it should do with this request.  It will see entries in
/etc/inetd.conf that look like:

mcserv          stream  tcp     nowait  mcadde  /home/mcidas/bin/mcservsh      
mcservsh -H /home/mcidas
mccompress      stream  tcp     nowait  mcadde  /home/mcidas/bin/mcservsh      
mcservsh -H /home/mcidas

If you look at these entries (and read the man page on inetd.conf, you will
see that the 6th entry (/home/mcidas/bin/mcservsh) is the executable that is
to be run and the 5th entry (mcadde) is the user to run the executable as.
The last entry above specifies a flag and its value to send to the
executable when invoked.

>it reads .mcenv on my machine and 

/home/mcidas/bin/mcservsh is a Bourne shell script.  In it the decision
is made to read in the values from .mcenv IF it exists:

TZ=GMT
export TZ

McINST_ROOT='/home/mcidas/mcidas76'

case $1 in
-H)     HOME="$2"
        shift
        shift

        [ -f "$HOME/.mcenv" ] &&
           . "$HOME/.mcenv"
        ;;
esac

exec $McINST_ROOT/bin/mcserv

The 'case' statement says that if the first passed parameter to the
command was '-H', then pickup the second value (which is specified to
be '/home/mcidas' in the inetd.conf lines above).  So, if .mcenv exists
([ -f "$HOME/.mcenv" ] asks whether or not the file exists), it is read
in ('. "$HOME/.mcenv").  You should be familiar with this syntax since
you are a Korn shell user.

>trys to resolve the named dataset locally?,

Yes.

>and you find the 
>definitions through the search path which includes /home/mcidas/data
>(linked to /home/mcidas/mcidas76/data) which is where we copied
>the ADDESITE.TXT definitions.

No, the definitions for the dataset are found in the file RESOLV.SRV
which will be in one of the MCPATH directories on your machine.  Here
MCPATH is the one that is defined in .mcenv.

>And so find the unique grids on my machine. 

Right.

>So what did changing the MCDATA directory in .mcenv to point 
>to /home/mcidas/uvaworkdata have to do with any of this?

RESOLV.SRV is found there.

>ADDESITE.TXT is not in uvaworkdata (its not supposed to be, right?).  

It is not there, and should not be there.

>I was going to call and get clarification on this, but....
>its Friday afternoon, and I am afraid I would talk to long
>and end up late picking up Jack from school (its happened
>before).  

Good move.

>If I still have this messed up (conceptually), and it would be easier 
>to address with a conversation, I can wait til next week. 

Does the above make sense?

Now, MCTABLE_READ and MCTABLE_WRITE are defined in .mcenv to point to
non-existent files.  This way, the datasets are deduced to be local
since there is no MCTABLE.TXT or ADDESITE.TXT telling the command
where to go to resolve the server.  If there was a valid MCTABLE_READ
in effect, then request would be forwarded to the machine specified 
in the MCTABLE.TXT or ADDESITE.TXT that contained the information on
where to look for the dataset.  This _might_ be a cool way to relay
requests, but it would be disasterous if the locations in MCTABLE.TXT
or ADDESITE.TXT pointed to the machine on which the remote server was
running.  This is the circular problem I referred to in the previous
email.

re: answering questions after midnight
>Yeah, I know what you mean.  Its not the time to try to read and
>comprehend mcidas documentation either !

That is even worse!

>(Sorry, its not the
>words, but the concepts that are hard to follow.) If it makes 
>sense to define a prototypical example of what goes on, like I was 
>trying to above, that might help.  

I hope that the above helped.

>Right, ADDESITE.TXT is in /home/mcidas/mcidas76/data, and thats the
>only place it should be, right.

Right, that is the only place it should be.

Tom