20020219: XCD decode of AVN; user RESOLV.SRV (cont.)

>From: Eirh-Yu Hsie <address@hidden>
>Organization: Aeronomy Laboratory/NOAA/DOC
>Keywords: 200202141802.g1EI2kx04733 McIDAS-XCD AVN

Hsie,

re: ADDE setup of RTGRIDS/AVN serving was using old GRID file range

>Your diagnostic is correct.

OK, its good to understand the problem.

>I have a follow up question.  A user
>copies ~mcidas/workdata/RESOLV.SRV to his home directory
>(~/mcidas/data) and add his own definition to it a while back.  My
>question is how can a user define his own data set on top of the site
>wide data definition (in ~mcidas/workdata/RESOLV.SRV) without copy or
>modify RESOLV.SRV?

Quick review of the purpose of RESOLV.SRV (for the tracking system):

RESOLV.SRV is the file that defines the correspondence between ADDE
dataset group/descriptor pairs and the files that this dataset
represents.  A RESOLV.SRV file in a user's McIDAS working directory
(~/workdata for the user 'mcidas' and ~/mcidas/data for other users)
will be used for LOCAL-DATA datasets.

For the user to be able to use elements of LOCAL-DATA datasets two
things must be setup correctly:  the user's RESOLV.SRV entries for the
dataset's group/descriptor elements AND the McIDAS applications run by
the user for those datasets must be able to find the files that
comprise the datasets.  RESOLV.SRV entries are usually maintained
(added, modified, deleted) by the McIDAS DSSERVE command.  How McIDAS
finds data files is controlled by two separate McIDAS systems:  file
REDIRECTions and the list of directories set in the user's MCPATH
environment variable.

OK, so much for the review.

Now comes the long answer to your short question (mostly for future
information in the tracking system).

1) the McIDAS administrator at a site (the user 'mcidas') should setup
   ADDE serving of all generally available datasets.  This is done by
   'mcidas' making a copy of the distribution file,
   ~mcidas/data/DSSERVE.BAT into ~mcidas/data/LSSERVE.BAT.  After
   making this local copy, 'mcidas' needs to edit LSSERVE.BAT to setup
   the datasets as they will be found on his/her system.  After making
   all of the edits, 'mcidas' will then run BATCH LSSERVE.BAT from a
   McIDAS session (or batch.k LSSERVE.BAT from the ~mcidas/workdata
   directory).

2) the ADDE remote server should be setup to be run as the user 'mcadde'.
   My recommendation is to make the HOME directory for 'mcadde' the
   same as that for the user 'mcidas'.  This way all of the work done
   by 'mcidas' to setup generally available datasets can be easily
   used by the remote server (i.e., as the user 'mcadde').  Recall
   that one of the setup steps is the creation of the file ~mcidas/.mcenv
   that defines the needed McIDAS environment when McIDAS applications
   run from the remote server

3) each user at a site should setup pointing (DATALOCs) to the datasets
   setup by 'mcidas' and available through the ADDE remote server.
   For instance, if 'mcidas' setup the dataset RTIMAGES, and if the
   remote server is configured for a machine named stratus.al.noaa.gov,
   then each user would "point" at the remote server for that dataset
   by running a DATALOC command:

   DATALOC ADD RTIMAGES stratus.al.noaa.gov

   To simplify this process, I included a McIDAS BATCH file template
   that can be used to setup a number of DATALOCs at one go:
   ~mcidas/data/DATALOC.BAT.  It is the responsibility of the
   McIDAS system administrator at a site to make a local copy of
   DATALOC.BAT, ~mcidas/data/LOCDATA.BAT, and edit that copy to
   setup pointing for all of the datasets that will be generally
   available on his/her system.  The user then makes those definitions
   active by running:

   BATCH LOCDATA.BAT

   from his/her McIDAS session (or by running 'batch.k LOCDATA.BAT'
   from his/her ~/mcidas/data directory)

   At this point (assuming that everyting was done correctly), each user
   should be able to access all of the datasets "published" by the
   user 'mcidas' and made available through the remote ADDE interface

4) after doing the above, each user can setup his/her own datasets
   in his/her own account.  This is done by running DSSERVE commands
   to define those datasets and either adding file REDIRECTions so
   the files in the datasets can be found or including the directory(ies)
   where the data files in the datasets live in his/her MCPATH.

   NOTE:  in order to redefine a dataset, the DATALOC for that dataset
   must be LOCAL-DATA.  What I mean by this is that the non-'mcidas'
   user will not be allowed to define the RTIMAGES dataset (or any
   other defined/accessible through the remote server) while his/her
   DATALOC points to a remote ADDE server.  In general, it is better
   for the end user to choose a name for his/her dataset that does
   not match one that they have a DATALOC to a non-local dataset for.

   So, in order for the user 'hsie' wants to define a new dataset, he
   must determine the following things:

   a) the group name for the dataset
   b) the types of data that are to be included in this dataset
   c) the descriptor names for this 'group' (the set of data referenced
      by a group/descriptor pair will all be of the same type:  GRID,
      IMAGE, POINT, or TEXT
   d) the location of the actual data files that will be included in each
      group/descriptor set
      
   After deciding on thise things, the user needs to construct DSSERVE
   commands to define the datasets in their session.  ~mcidas/data/DSSERVE.BAT
   is a good example of doing this.

   Since datasets typically have multiple group/descriptor pairs of
   definitions, I strongly recommend that the user create a McIDAS
   BATCH file that holds all of the DSSERVE commands that he/she will
   need to run to define his/her datasets.  NOTE: this file should
   not be named LSSERVE.BAT as this will clash with the system definition
   file ~mcidas/data/LSSERVE.BAT.

So, my recommendation is NOT for the user to copy the 'mcidas' RESOLV.SRV
file, but, rather to create his/her own by running DSSERVE commands
for each group/descriptor pair.

Let's look at a quick example.  Suppose I am the user 'hsie', and suppose
that the McIDAS administrator for my site has already defined a series
of datasets that I can access by a remote ADDE server on the machine
stratus.al.noaa.gov.  The McIDAS administrator, should have created the
file ~mcidas/data/LOCDATA.BAT as an edited copy of ~mcidas/data/DSSERVE.BAT.
The edited ~mcidas/data/LOCATA.BAT will look something like:

DATALOC ADD CIMSS     stratus.al.noaa.gov
DATALOC ADD GINICOMP  stratus.al.noaa.gov
DATALOC ADD GINIEAST  stratus.al.noaa.gov
DATALOC ADD GINIWEST  stratus.al.noaa.gov
DATALOC ADD RTGRIDS   stratus.al.noaa.gov
DATALOC ADD RTIMAGES  stratus.al.noaa.gov
DATALOC ADD RTNIDS    stratus.al.noaa.gov
DATALOC ADD RTNEXRAD  stratus.al.noaa.gov
REM DATALOC ADD RTNOWRAD  stratus.al.noaa.gov
DATALOC ADD RTPTSRC   stratus.al.noaa.gov
DATALOC ADD RTWXTEXT  stratus.al.noaa.gov
DATALOC ADD TOPO      LOCAL-DATA

DATALOC ADD AMRC      uwamrc.ssec.wisc.edu
DATALOC ADD ME7       io.sca.uqam.ca

DATALOC ADD BLIZZARD adde.unidata.ucar.edu

DATALOC ADD MYDATA LOCAL-DATA

The contents listed above assume that stratus.al.noaa.gov has and is
willing to serve through the remote ADDE interface all of the datasets
that are not commented out (a comment in a McIDAS BATCH file begins
with the sequence 'REM ').

'hsie' should have access to ~mcidas/data/LOCDATA.BAT through his
MCPATH definition, which would be defined in his ~/.cshrc (or ~/.profile,
etc.) shell definition file.  This would look something like:

MCPATH=/home/hsie/mcidas/data:/home/mcidas/data:/home/mcidas/help

(your setup is slightly different, but you should get the idea).

The user can verify that he can see this file by using the DMAP command
from his McIDAS session:

DMAP LOCDATA.BAT

Given that he can see ~mcidas/data/LOCDATA.BAT, he will make those
DATALOCs active by running:

BATCH LOCATA.BAT

Now (assuming tha the remote ADDE interface is correctly setup on
stratus), he should be able to access all of the data in all of the
datasets defined in LOCDATA.BAT.

Next, 'hsie' wants to define a dataset for his research.  Let's assume
that he has several different kinds of satellite images (GOES VIS, IR, WV)
and some model output in McIDAS GRID file format (say AVN, ETA, and
RUC).  Perhaps these data were from the COMEX project (whatever name
you like).  Let's use this as the group name for his datasets (NB: the
group portion of a dataset name is limited to 8 characters).

One way to setup the dataset would be to put all of the data files in
a single directory.  Let's assume that this is the /data/projects/COMEX
directory.

IF: the GOES VIS images are in AREA files AREA2000 - AREA2019
    the GOES IR images are in AREA files AREA2020 - AREA2039
    the GOES WV images are in AREA files AREA2040 - AREA2059
    the AVN grids are in GRID files GRID2000 - GRID2009
    the ETA grids are in GRID files GRID2010 - GRID2019
    the RUC grids are in GRID files GRID2020 - GRID2029

Here is one way of setting up the COMEX dataset:

DSSERVE ADD COMEX/GOES-VIS AREA 2000 2019 TYPE=IMAGE "COMEX GOES 0.65 um images
DSSERVE ADD COMEX/GOES-IR  AREA 2020 2029 TYPE=IMAGE "COMEX GOES 10.7 um images
DSSERVE ADD COMEX/GOES-WV  AREA 2040 2059 TYPE=IMAGE "COMEX GOES  6.8 um images
DSSERVE ADD COMEX/AVN      GRID 2000 2009 TYPE=GRID  "COMEX AVN model data
DSSERVE ADD COMEX/ETA      GRID 2010 2019 TYPE=GRID  "COMEX ETA model data
DSSERVE ADD COMEX/RUC      GRID 2020 2029 TYPE=GRID  "COMEX RUC model data

These setup the RESOLV.SRV entries.  They do not, however, tell 'hsie's
McIDAS session where to look for these files.  One way of doing this
is by defining file REDIRECTions:

REDIRECT ADD AREA20* "/data/projects/COMEX
REDIRECT ADD GRID20* "/data/projects/COMEX

Of course, there are many ways that the data could be saved on disk.  Each
different way would require either a different DSSERVE invocation or a
different set of file REDIRECTions.

By default, when a user defines a dataset the DATALOC for that dataset
in his/her McIDAS environment gets set to LOCAL-DATA.  This means that
the user doesn't have to run:

DATALOC ADD COMEX LOCAL-DATA

but, it can't hurt.

OK, that was an overly long answer to a simple question, but I felt that
all aspects of setting up a dataset should, at least, be touched on.

Please let me know if the above doesn't make sense (or has any typos ;-).

Tom