20010221: ADDE dataset creation and location (cont.)

>From: "Paul L. Sirvatka" <address@hidden>
>Organization: College of DuPage
>Keywords: 200102162013.f1GKD9L01250 McIDAS-X SFCCON GRDDISP

Paul,

re: ADDE dataset definition and location
>I spent several hours last night trying to figure this out. Some I get,
>some I am still confused.

Don't feel alone on this.  Grasping the ADDE setup is not immediate for anyone.
I feel certain, however, that if we work through one or two examples then
the basic principles will then be clear enough for you to carry on.

>First...DATALOC is where the datasets are located that mcidas will use for
>the commands.

Yes, exactly.

>Now this is what dataloc does. I can say LOCAL-DATA to look
>locally for certain gropu names.

Yes.  Let's dwell on this one for a couple of minutes.  When we say that the
routines will look locally, what we really mean is that the ADDE routines that
are started up by client applications (like SFCPLOT, SFCCON, IMGDISP, etc.)
will be run from our own account.  In order to be successful, they need to
be able to know what dataset to look for (each of the POINT plot/contour 
routines
has defaults like RTPTSRC/SFCHOURLY for SFCPLOT and SFCCON; RTPTSRC/UPPERMAND
for RAOBPLOT and RAOBCON; etc.).  Just knowing what dataset to look in
means nothing UNTIL the contents of the dataset are defined.  The definition of
the contents of a dataset is done by DSSERVE.

>Does this have to do with ADDESITE.TXT?

Yes.  In the 'mcidas' account, the Unix environment variable
MCTABLE_WRITE is defined (if setup according to recommendions on the
McIDAS web pages) as follows:

MCTABLE_WRITE="$MCHOME/mcidas/data/ADDESITE.TXT"

where MCHOME is the HOME directory of the user 'mcidas'.

What this means is that the results of DATALOC commands will be written
into ADDESITE.TXT.  Here is where the user 'mcidas' differs from other
users of McIDAS-X.  Since ADDESITE.TXT is specified to be located in
~mcidas/data (by the MCTABLE_WRITE definition), and since all McIDAS-X
users whould have a MCPATH enviornment variable defined to include
~mcidas/data, then all users can see/use ADDESITE.TXT.  This allows the
McIDAS administrator, the user 'mcidas' to setup DATALOCations that can
be used by all McIDAS-X users.  This would be useful, for instance, if
there was a group of users in a lab and they all wanted to look at
NOAAPORT GINI data.  Assuming that this was setup in the first place,
and it is in your 'mcidas' environment, then all of the students can
access the data without having to run their own DATALOC commands.
Following along with this example, if for some reason the site that you
were getting GINI data from went down (or was unaccessible), then you
as the user 'mcidas' could run one DATALOC command for RTGINI (or
GINIEAST, GINIWEST, GINICOMP) and they would all immediately be trying
to get the data from the new site that you specified.

>Then...DSSERVE is where I get confued. You wrote:
>
>> DSSERVE is used to setup datasets.  It creates an association between a
>> dataset group/descriptor and actual data files. 

Right.  A dataset name is nothing more than that UNTIL you associate
the name with some actual data files.  This is DSSERVE's job.

>I need to define the descriptors for what is in each data set. For
>instance....RTPTSRC (btw why RT?)

Use of the 'RT' prefix is a convention.  RT stands for Real Time.  The
idea behind having a convention is so that you don't have to include
which dataset to look at each time you run a command.

>is point source data like SFCHOURLY and
>LIGHTNING and UPMAND, etc....

Right.  The various RTPTSRC subsets defined in DSSERVE.BAT are:

RTPTSRC/SFCHOURLY
RTPTSRC/UPPERMAND
RTPTSRC/UPPERSIG
RTPTSRC/LIGHTNING
RTPTSRC/SHIPBUOY
RTPTSRC/FOUS14
RTPTSRC/SYNOPTIC
RTPTSRC/AIRCRAFT
RTPTSRC/PROFHOURLY
RTPTSRC/PROF6MIN

Think of an ADDE dataset as just that, a set.  The set of all real time
point source files are identified by the group name, RTPTSRC.  Subsets
of point source files are grouped according to type: SFCHOURLY,
SYNOPTIC, etc.  The groupings are user selectable, but a convention was
adopted for commonly used sets like those above.

>I need to define these because there are
>default group/descripto sets for ADDE commands, right?

Yes, and because you want to group the subsets of files together so
that you can tell the McIDAS applications which subsets they should be
looking at.

>And if I do not define it...what will happen?

Applications will fail since they won't know how to access files for
the dataset specified since the assocation between the set of files and
the dataset group/descriptor was never made.  The point here is that
there is no default definitions for the contents of a dataset.

>If I do not define it and it is looking locally, it will not work.

Right.

>If I do not define it but it is at a remote site, will it work?

Yes assuming, of course, that it has been defined at the remote site.
Again, in this case you tell McIDAS to look for the dataset at the
remote site using the DATALOC command.

>At this point, what are the commands that have
>group/descriptor.something. What is the .something doing? Sometimes it is
>ALL....I am still a bit confued on that.

the 'position' portion of the dataset reference
group/descriptor.position is an index into the subset defined by
group/descriptor.  If there are 4 files that compose the
group/descriptor subset, then 'position' can range from 1 to 4.  A
'position' of ALL means look through all of the files that compose the
subset.  A 'position' of 0 means to look at the most recent one.

For negative position numbers, the meaning is a little different for
image datasets than the rest.  For image datasets, a position of -1
means look at the next most recent and most recent images; a -2 means
look at the second most recent, the first most recent, and the most
recent images , etc.

Try this out while looking at the RTGINI/GE1KVIS images:

IMGLIST RTGINI/GE1KVIS.ALL

This will list out all of the files that comprise the RTGINI/GE1KVIS set of
images.

IMGLIST RTGINI/GE1KVIS.0

This will list the most recent one.

IMGLIST RTGINI/GE1KVIS.-1

This will list the next most recent one and the most recent one.

IMGLIST RTGINI/GE1KVIS.-2

This will list out three images: the second most recent one, the first most
recent one, and the most recent one.

The GRID and POINT dataset listing applications work a little differently
since each file contains more than one object.  Try running the following to
get an idea of what the difference is:

PTLIST RTPTSRC/SFCHOURLY.ALL FORM=FILE

If you do not include FORM=FILE, you will look through all of the files in
the RTPTSRC/SFCHOURLY set and list out the first item from the most recent
file.  Compare this to:

PTLIST RTPTSRC/SFCHOURLY

This lists out the first object from the most recent file in the
RTPTSRC/SFCHOURLY set.

This will seem a little strange until you get the hang of it.

>I now figure that the USER MCIDAS, which I am setting up to do batch
>commands primarily does NOT need MYDATA right?
>That is just a group for individual users?

The MYDATA dataset is a construct that makes it easy to associate
dataset elements (files) with dataset positions.  There is a one-to-one
mapping between a McIDAS data file name and the corresponding MYDATA
element:

MYDATA/IMAGES.1    <-> AREA0001
MYDATA/IMAGES.1    <-> AREA0002
 ...
MYDATA/IMAGES.9999 <-> AREA9999

MYDATA/GIRDS.1     <-> GRID0001
MYDATA/GIRDS.2     <-> GRID0002
 ...
MYDATA/GRIDS.9999  <-> GRID9999

etc.

This makes it easy to deal with the old way of doing things in McIDAS
(i.e., having to deal with data file numbers) and the new way of doing
things in McIDAS (i.e., ADDE).

The MYDATA dataset is useful for all users who want to do specialized
things (specialized meaning things not offered in the Fkey Menu or
MCGUI).  One example of this is demonstrated by an example McBASI
script that is included in the distribution, CFILLCLP.MCB.  Try
the following:

DATALOC ADD RTGRIDS adde.ucar.edu
CFILLCLP RTGRIDS/ETA T SFC 12 #Y 0 LAMB MYDATA/IMAGES.1234 Y Y

This will display a color filled representation of the 0-hour forecast
ETA surface Temperature for the 12 Z run today (#Y).  Since McIDAS
doesn't do color filled contours, what is really happeing is an image
representing the grid is created and stored in the MYDATA/IMAGES.1234
element (which is AREA1234).  The last 'Y Y' tell the script to clip
the display at land-water boundaries and display the image in the
current frame.

The reason I mention this is because of the use of the MYDATA/IMAGES.1234
dataset group/descriptor.position.  Since I know that AREA1234 is typically
not one of the ones that a site might be using to store ingested imagery,
I can use it as a scratch image.  If AREA1234 was used for something
else, I could choose a different AREA to put the result in, say AREA4321:

CFILLCLP RTGRIDS/ETA T SFC 12 #Y 0 LAMB MYDATA/IMAGES.4321 Y Y

One note on this McBASI script:  The script will use the TOPO/GLOB
image as a mask for land-water boundaries.  Currently, you have
TOPO pointed at adde.ucar.edu.  This is fine, except that TOPO/GLOB
is an image that is 9 MB.  If you tried setting up the TOPO dataset
locally as per my message yesterday, you would find that you do
not have the TOPO/GLOB AREA file in your distribution.  The reason
for this is that it is large (9 MB) and not used by everyone.  If
you want to flesh out your TOPO dataset, do the following:

<login as 'mcidas'>
cd data
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> your_email_address
  cd put/mcidas
  binary
  get AREA9000
  quit

Another way of grabbing the file is by using ADDE itself:

REDIRECT ADD AREA9* "/home/mcidas/data
DATALOC ADD TOPO adde.ucar.edu
IMGCOPY TOPO/GLOB MYDATA/IMAGES.9000 SIZE=ALL

The REDIRECTion makes sure that the image will be written in the
/home/mcidas/data directory (this assumes that the HOME directory
for 'mcidas' on your system is /home/mcidas; if it is something
different, readjust the REDIRECT command above to match whatever
your ~mcidas/data is).  The SIZE=ALL keyword insures that the entire
image is copied.

After the transfer, you would then do:

DATALOC ADD TOPO LOCAL-DATA

If you havn't already done so, do the following:

BATCH TOPOADDE.BAT

This sets up the TOPO dataset for files on your system.  Now when
you do something like:

IMGDISP TOPO/CONF MAG=-3 EU=TOPO

you will be using the file that corresponds to TOPO/CONF that is
on your own system, and the display should be faster.

re: using the DSSERVE.BAT file that is in the distribution to setup
datasets
>OK I did that....in fact I did a bunch of .BAT files that added a lot. Is
>there a text file that can be edited to delete some of the extras or do I
>need to do DSSERVE DEL for each one.

The installation instructions recommend that McIDAS site administrators
make a local copy of DSSERVE.BAT (called LSSERVE.BAT) and edit it
to setup things to match their individual systems:

<login as 'mcidas'>
cd data
cp DSSERVE.BAT LSSERVE.BAT
<edit LSSERVE.BAT and comment out datasets you don't/won't have>

<from a McIDAS session as 'mcidas'>
BATCH LSSERVE.BAT

Since you have already run 'BATCH DSSERVE.BAT', the only way to remove
datasets that have been defined is to run 'DSSERVE DEL' commands
on the group/descriptor ones that you don't want.  For reference,
these definitions are stored in a file named RESOLV.SRV.  For
'mcidas', this file will be located in the ~mcidas/workdata directory.
RESOLV.SRV can be edited directly, but you must be careful when doing
this so as to not add characters to lines that are not supported
(see HELP for DSSERVE for a list) or delete the trailing comma at the
end of each line.

>We have MD files saved via the XCD. So we have them locally.

OK, so you are in a position to setup the RTPTSRC dataset for the
descriptors (data types) that you do have.  If you ran 'BATCH DSSERVE.BAT'
then these will be setup now.

One comment.  Along with defining a dataset with DSSERVE, you must
also tell your McIDAS session how to find the data files.  This is
done through the McIDAS REDIRECT facility and through the MCPATH
environment variable setting.  In the online installation instructions
for McIDAS, I comment about an example set of REDIRECTions that should
be everything you need to define.  The instructions tell the McIDAS
administrator to:

<login as 'mcidas'>
cd data
cp EXAMPLE.NAM LOCAL.NAM
<edit LOCAL.NAM and change the directory locations for each data file
set (set defined with file masks) to match your installation setup>

<start a McIDAS session>

REDIRECT REST LOCAL.NAM
REDIRECT ADD LWPATH.NAM ".

The last line justs makes sure that the set of redirections stored in
LWPATH.NAM get sorted the way they will be used for searching.

>BTW I got all the topo stuff which I delted in this e-mail...

OK, great.  Then the comments above about the TOPO/GLOB file should
make sense.

re: setting up the other datasets

>I did this for some....here is the new DATALOC
>
>Group Name                    Server IP Address
>--------------------         ----------------------------------------
>BLIZZARD                     144.92.109.163
>CIMSS                        <LOCAL-DATA>
>G8-GHCC                      198.116.59.198
>MYDATA                       <LOCAL-DATA>
>RTGINI                       ADDE.UCAR.EDU
>RTGRIDS                      <LOCAL-DATA>
>RTIMAGES                     <LOCAL-DATA>
>RTNEXRAD                     WEATHER.ADMIN.NIU.EDU
>RTNIDS                       ADDE.UCAR.EDU
>RTNOWRAD                     <LOCAL-DATA>
>RTPTSRC                      <LOCAL-DATA>
>RTWXTEXT                     ADDE.UCAR.EDU
>TOPO                         <LOCAL-DATA>

OK.  If you are running McIDAS-XCD then the RTWXTEXT dataset should also
be accessible to you locally.  So, I recommend that you do:

DATALOC ADD RTWXTEXT LOCAL-DATA

Also, the RTNIDS dataset is not available to you from adde.ucar.edu.
This dataset was composed of NIDS images from WSI.  By contract, they
are not to be shared until they are 48 hours old.  You don't need
access to these data anyway given that the same imagery is now freely
available by the IDD.  You should, therefore, do the following:

DATALOC DEL RTNIDS

>So if I di it right, SFCCON will look locally since RTPTSRC looks here.

Right.

>Here is the rub...RTWXTEXT is still looking at ADDE.UCAR.EDU

>I know I use DATALOC ADD RTWTEXT LOCAL-DATA to look locally but I do not
>know what to do with DSSERVE with all the descriptors. Is my confusion
>clear to you?

The RTWXTEXT setup was in DSSERVE.BAT also.  By the way, before
running 'BATCH DSSERVE.BAT' (better 'BATCH LSSERVE.BAT') the
online installation instructions advise you to define the McIDAS
string XCDDATA to be the directory where XCD will be decoding its
files.  This would look something like:

TE XCDDATA "/data/mcidas

You would replace '/data/mcidas' with the directory where your XCD
decoders are writing data files.

After defining this, you would run:

BATCH LSSERVE.BAT

Also, the online instructions direct the McIDAS administrator to
setu the LOCAL.NAM file and making the REDIRECTions in it active
before doing any DSSERVE stuff.  If you didn't do this, then
I recommend:

o setup the LOCAL.NAM file (copy of EXAMPLE.NAM and then edit the copy)
o restore those REDIRECTions
o define XCDDATA as I list above
o stop the ldm
o remove the *.R* files in the XCDDATA directory
o rerun the XCD setup routines:

BATCH XCD.BAT
BATCH XCDDEC.BAT

o restart the LDM
o (re)setup the datasets

BATCH LSSERVE.BAT

After decoding procedes for awhile, the data files for the RTWXTEXT
should be being created/updated, so you will be able to switch to
using RTWXTEXT from your local files.

re: setting up the remote ADDE server interface
>So I will serve others if desired?

So you can serve:

o other machines at COD; ADDE access to files is faster than direct
  access to files on an NFS mounted file system (this may seem hard
  to believe, but it is true.  The "pipe" for reading through ADDE
  is larger than the "pipe" for reading by NFS)

o the same machine that the remote server is hosted on.  This will
  be useful to the non-'mcidas' users as they won't have to each
  setup dataset definitions (i.e., run the various DSSERVE commands);
  file REDIRECTions, etc.  All they would have to do is run the
  appropriate DATALOC commands to point at the machine's remote
  ADDE server interface

o machines not in your domain.  This might be other Unidata users,
  and it might be you accessing your own machine from home, or while
  you are out storm chasing.

>How fast *should* the comands run. It seems like to start plotting the
>contours it is a bit slow (~2-3 secs). Is this normal?

To start plotting contours, the application has to go out and get all
of the data and then do the objective analysis.  Depending on the machine
involved, this might take anywhere from 1 to several seconds.  If the
server is on a remote machine that is heavily loaded and the network
connection is slow, the wait could be even longer.

This brings up another issue.  When you setup your account, did you
define the MCCOMPRESS environment variable?  For C shell users, this
would be done in .cshrc and would look like:

setenv MCCOMPRESS TRUE

For Bourne/Korn shell users, this would be in .profile (or, perhaps
.kshrc for Korn shell users), and it would look like:

MCCOMPRESS=TRUE
export MCCOMPRESS

Setting this environment variable causes the data to be compressed on
the server side and uncompressed on the client side.  It greatly speeds
transfers of data.  If you are not using this feature, you must have
the environment variable defined before you start your McIDAS session.
You will be able to see that loading of the GINI imagery from adde.ucar.edu
will be LOTS faster when compression is turned on.  This, like the other
things, is described in the online installation/configuration web
pages.

re: accessing RTNEXRAD data on Gilbert's machine
>Are you assuming Gilbert's machine is set up right?

Yes, I set it up myself as a favor to Gilbert.

>When I tried that out
>I got "Image directory server unable to resolve this
>dataset:  RTNEXRAD/NOR" after the third command. Same when I went to
>ADDE.UCAR.EDU

The problem here is that in the descriptor.  It is N0R (N zero R), not
NOR (N oh R).  To get a list of the descriptors available for a dataset,
all you need to do is use DSINFO:

DSINFO IMAGE RTNEXRAD

re: remember who you are pointed at or use DATALOC to see who you are pointed
at
>Right...thereofre I need only change that if I want data from somewhere
>else. Is that the glory of ADDE?

Yes.  Also, you already have a DATALOC that points to a machine that has
a dataset that you don't have:

G8-GHCC                      198.116.59.198
RTGINI                       ADDE.UCAR.EDU

You can not only access a dataset that you should have locally, but don't
due to some sort of failure, but you can also share datasets that you
don't have locally (like RTGINI and G8-GHCC).

One final (downer) note.  I have recently discovered some sort of a
bug in serving sounding data through the remote ADDE server when it
is hosted on either RedHat 6.x or 7.0 Linux (I think that I mentioned
this before).  I have been beating my head against this problem for now
over two weeks and have not pinpointed the problem (it looks like something
in the OS), nor have I developed a workaround.

Accessing the data as LOCAL-DATA works fine.  Accessing the data from
remote servers that are not running RedHat 6.x/7.x works fine (including
RedHat 5.2, FreeBSD 4.2, Solaris x86 2.[78], Sun Solaris SPARC, DEC OSF/1,
HP HP-UX 10.20/11.00, IRIX/IRIX64).  This problem is a bugger!  The
only thing I can advise sites to do now is:

o access the data as LOCAL-DATA (which means that each user will have to
  setup this dataset (DSSERVE, and REDIRECT to locate the data files)

o point at a remote serve that is not running RedHat 6.x/7.x Linux for
  the RTPTSRC data

Gilbert's box falls into the list of machines that do not work, and I
assume that yours will as well given that you are running RedHat 7.0
Linux (or so you hinted in a previous email).

Tom