20011114: McIDAS vis-a-via Solaris 7/8 (cont.)

>From: "James R. Frysinger" <address@hidden>
>Organization: College of Charleston
>Keywords: 200111061842.fA6Igt112242 McIDAS platfomrs

Jim,

re: was g77 installed
>
>I don't know if it was installed with it or not. But they have the 
>same file date, March 17 2001. That must have been a version-driven 
>date rather than the install date. On May 10 2001 we installed a new 
>hard drive, installed Solaris 7, and -- among other things -- did a 
>pkgadd gcc2.95.3. I don't see a note about our specifically adding g77. 
>That may have come along as a result of installing Solaris 7 or as a 
>result of adding gcc. Doing gcc --version gives me
>   GNU Fortran 0.5.25 20010315 (release)
>which is only two days older than the file date and that date is before 
>we built the current drive.

OK, this looks like g77 was installed with gcc.

re: /export/home/mcdata then makes good sense!
>
>That's what I did. I created a directory (not account!) called 
>/export/home/mcdata and replaced /var/data with that fully qualified 
>pathname wherever found in LOCAL.NAM. Here are some typical lines from 
>the way that reads now:
>   AREA007*     /export/home/mcdata/mcidas
>   MDXX000*     /export/home/mcdata/xcdM
>   MDXX003*    /export/home/mcdata/xcd
>Let me know if that looks bad.

There is no longer a need to separate data files created by XCD from
those created by ldm-mcidas decoders (there was such a need a long
time ago when the Unidata-Wisconsin datastream still contained
MD and GRID files), so you could put everything in /expoet/home/mcdata:

AREA007*     /export/home/mcdata
MDXX000*     /export/home/mcdata
MDXX003*    /export/home/mcdata
 ...

re: Unidata access to your remote ADDE server

>I've checked in the department and we have absolutely no objection to 
>that. We expect that you're going to get rebuffed by the college's 
>firewall, though, unless we got far enough a year ago to cause those to 
>be opened up. (I only recall being set up for data to come into our 
>site, not to be served out of our site, though.)

The ports that need to be opened through the firewall are:

Port  Use
-----+---------------------------
388   LDM
500   ADDE uncompressed transfers
503   ADDE compressed transfers

re: we get two copies of each email you send
>
>Hmmmm. Our college server has been doing that occasionally with 
>messages. Once in awhile I add my address@hidden to the cc 
>line on these messages to you, and that has a forward line in my mail 
>profile that forwards mail sent there back to my address@hidden 
>account. That shouldn't cause you to get the doubled mail though, I 
>don't think.

We get two copies nonetheless.

>OK, on to new things. I didn't get too far. I've gone into 
>~mcidas/data and have cp'd sample files into LOCAL.NAM (in which I 
>altered the pathnames as described above), LSSERVE.BAT, and 
>LOCDATA.BAT. And I'm stuck on the last two.
>
>The directions in "Configuring the mcidas Account" regarding 
>LSSERVE.BAT say that I should modify the file name cylinder to match my 
>McIDAS data files. The header comments in LSSERVE.BAT say
>LSSERVE.BAT then needs to be edited and the McIDAS
>file "cylinder" numbers need to be changed to match the 
>local setup for datasets RTIMAGES, and potentially RTGRIDS.
>Huh?

Only the most experienced McIDAS users would alter the default setup.
I suggest that you stay with the default for now.

>I see a lot of numbers after words like AREA, NEXR, NOWR, and 
>GRID. Are those the cylinder numbers?

Yes.  McIDAS has always used circular naming conventions for its
real-time data files.  So, for instance, real-time point souce files
(like decoded METARs, lightning, wind peofiler, etc.) _by convention_
are named as:

MD file names          POINT source type
----------------------+----------------------------
MDXX0001 - MDXX0010    Surface hourly
MDXX0011 - MDXX0020    Mandatory level upper air
MDXX0021 - MDXX0030    Significant level upper air
MDXX0031 - MDXX0040    Ship/Buoy
MDXX0041 - MDXX0050    FOUS14
MDXX0051 - MDXX0060    Synoptic
MDXX0061 - MDXX0070    AIREP/PIREP
MDXX0071 - MDXX0080    NLDN Lightning
MDXX0081 - MDXX0090    Hourly wind profiler
MDXX0081 - MDXX0100    6-minute wind profiler

The naming scheme is such that the last digit of the file name is
the same as the last digit of the Julian day of the data in the file
(remember, this is for real-time data only).  So, for instance, today
is Julian day 2001319, and the real time MD (Meteorological Database)
files by type are:

MDXX0009 - Surface hourly
MDXX0019 - Mandatory level upper air
MDXX0029 - Significant level upper air
MDXX0039 - Ship/Buoy
MDXX0049 - FOUS14
MDXX0059 - Synoptic
MDXX0069 - AIREP/PIREP
MDXX0079 - NLDN Lightning
MDXX0089 - Hourly wind profiler
MDXX0089 - 6-minute wind profiler

After the tenth day, the last digit of the Julian day number repeats, so
the file name in which the data can be found will also repeat.

This brings up an important point: real-time MD files _MUST_ get scoured
before they get to be ten days old.  The reason for this is twofold:

o the decoders _always_ write to the end of the appropriate MD file

o MD files have a finite size that is determined when they are first created

Given these two items, it is imperative to delete an MD file before
the decoder starts writing new data after old data, and especially
before the file fills up.  When the file fills, no new decoded data
will get filed.  'Nuf said...

GRID (that is to say model output) data filing works the same way
as MD files.  All of the above caveats must be heeded.

AREA files (containers for images) work differently.  It is the AREA
file numbers (AREAnnnn, 'nnnn' is a 4 digit number that can range from
0001 to 9999) that can be tweaked by the end user.  The default is to
store 10 of each kind of image on a user's system.  One can change this
by altering the file routing table (ROUTE.SYS) through use of the file
routing table management routine ROUTE.  I recommend, however, that you
stick with the defaults until you are comfortable with the concepts
behind the data storage.

Now (phew) we come to the entries in LSSERVE.BAT (took awhile, didn't
it?).  After you setup your system to ingest and store data files, you
have to inform McIDAS about what you have done.  In particular, you
define ADDE datasets as collections (sets) of similar things that can
be accessed as a group.  The definitions in LSSERVE (the various
DSSERVE command lines) associate an ADDE name (group/descriptor) with a
set of files.  For instance:

DSSERVE ADD RTIMAGES/EDFLOATER-II AREA   60   69 "Educational Floater II

associates the ADDE dataset name RTIMAGES/EDFLOATER-II with the set of
AREA files AREA0060 - AREA0069, inclusive.  If one setup ingestion and
decoding according to the defaults, this set of files will contain
Educational Floater II images.  If you had decided that you wanted to
keep more than 10 of this type of image on your system, you would have
had to modify the file routing table (through use of ROUTE) AND you
would have had to select a new AREA file range of numbers (cylinders)
for the images (the image numbers had been set _by convention_) so that
each type of image used 10 consecutice numbers and the next set of
consecutive numbers would differentiate one kind of image from the
next.  Keeping more of any particular kind of image would mean that the
numbers used would overlap those for a different kind of image, and
this is a NO-NO.

>What do I change them to? I am totally lost here.

You would have to modify the ADDE dataset definition in LSSERVE.BAT
_if_ you modified the default ingestion setup.  You can see that it is
possible that you might want to do this after you become knowledgable
in how McIDAS works, but it is wise to go with the defaults until you
gain that confidence.

>The header comments in LSSERVE.BAT also state that
>The user must define the McIDAS string XCDDATA to be
>the fully qualified directory where XCD-decoded data is 
>stored before running this script.

Yes.  In your case you would run the following command as 'mcidas'
to define XCDDATA:

from the Unix command prompt:

cd workdata
te.k XCDDATA \"/export/home/mcdata/xcd

- or if you accept my comment about not having to keep separate XCD and
  ldm-mcidas created data files -

te.k XCDDATA \"/export/home/mcdata

from within a McIDAS-X session (again running as 'mcidas':

TE XCDDATA "/export/home/mcdata/xcd

- or -

TE XCDDATA "/export/home/mcdata

>In the section of LSSERVE.BAT that talks about RTWXTEXT I see some 
>lines like
>   INFO=#XCDDATA/FOUS14.RAP

Exactly right.  This is, in fact why you have to define XCDDATA.  The
reference '#XCDDATA' in McIDAS means 'value of XCDDATA (like the
Unix syntax $val).

>Should I change those five lines to read like the following?:
>   INFO=#/export/home/mcdata/XCD/FOUS14.RAP

No.  You run the TE command listed above as 'mcidas'.

>Of course, I should also mkdir /export/home/mcdata/XCD if I do that, 
>right?

You will have to create the directory in which you create data files
with XCD, yes.

>       The directions in "Configuring the mcidas Account" say that I should 
>edit LOCDATA.BAT and change all instances of 
>fully_qualified_host_identifier to the fully qualified hostname or IP 
>address of my local ADDE server machine.

Right.  My recommendation is to setup the remote ADDE server and have
all data access go through it.  The reasonf for this is that it isolates
all of the ADDE dataset setup frudgery to one account, that of the user
'mcidas'.

>Isn't my local ADDE server 
>machine the very machine that I'm installing McIDAS on?

Yes.

>I seem to 
>recall putting something in there a year ago that related somehow to 
>weather.cofc.edu, but I can't remember what it was. Could be a faulty 
>memory between my ears, too. I really need a very verbose direction 
>here, Tom. According to those directions, I cannot go on till I get all 
>the above done because this gets used to create the client routing 
>table. The header comments in LOCDATA.BAT don't provide any clarity 
>that I can see. Perhaps I'm lost in the terminology.

If the machine on which you setup the ADDE remote server is
weather.cofc.edu, then change each occurrance (except TOPO) of
'fully_qualified_host_identifier' in LOCDATA.BAT to 'weather.cofc.edu'
for those datasets that you will have on your machine.

Unless you have a NOAAPORT satellite ingest system, this will likely
_not_ include GINICOMP, GINIEAST, and GINIWEST.  It will also _not_
include AMRC and ME7.  It will not by default include BLIZZARD, but
one can download the data files that comprise the BLIZZARD set
and host them locally.

Also, if you are not subscribed to WSI for NIDS data (you can get these
products for free through the IDD), then comment out the entries for
RTNIDS and RTNOWRAD.

You will point to other ADDE servers for the datasets you do not have
locally.

So, your LOCDATA entries should end up looking like:

REM
REM Note: if you do not have each kind of data on your system, you will
REM       need to specify the hostname/IP address of a site that is willing
REM       to serve you this data.  Please contact Unidata User Support
REM       <address@hidden> for more information.
REM

DATALOC ADD CIMSS     weather.cofc.edu
DATALOC ADD GINICOMP  snow.plymouth.edu
DATALOC ADD GINIEAST  cacimbo.ggy.uga.edu
DATALOC ADD GINIWEST  papagayo.unl.edu
DATALOC ADD RTGRIDS   weather.cofc.edu
DATALOC ADD RTIMAGES  weather.cofc.edu
REM DATALOC ADD RTNIDS    fully_qualified_host_identifier
DATALOC ADD RTNEXRAD  weather.cofc.edu
REM DATALOC ADD RTNOWRAD  fully_qualified_host_identifier
DATALOC ADD RTPTSRC   weather.cofc.edu
DATALOC ADD RTWXTEXT  weather.cofc.edu
DATALOC ADD TOPO      LOCAL-DATA

REM
REM Special purpose sites
REM

DATALOC ADD AMRC      uwamrc.ssec.wisc.edu
DATALOC ADD ME7       io.sca.uqam.ca

REM
REM Locate the McIDAS Learning Guide "Storm of the Century" dataset on the
REM Unidata server
REM

DATALOC ADD BLIZZARD adde.ucar.edu

REM
REM Locate the dataset MYDATA locally
REM

DATALOC ADD MYDATA LOCAL-DATA


This says that you will look to your own machine for CIMSS, RTGRIDS,
RTIMAGES, RTNEXRAD, RTPTSRC, RTWXTEXT, and TOPO.

For other datasets, you will look at the following machines:

GINICOMP  snow.plymouth.edu
GINIEAST  cacimbo.ggy.uga.edu
GINIWEST  papagayo.unl.edu
AMRC      uwamrc.ssec.wisc.edu
ME7       io.sca.uqam.ca
BLIZZARD  adde.ucar.edu

Now, assuming that you correctly defined the Unix environment variables
MCTABLE_READ and MCTABLE_WRITE (as per the online instructions), you
will create/update the file ~mcidas/data/ADDESITE.TXT when you run:

BATCH LOCDATA.BAT

And, when you setup user accounts, their MCTABLE_READ setting will
cause them to read ~mcidas/data/ADDESITE.TXT when they try to access
various datasets.  This is an example of how the user 'mcidas' can
setup things that all user accounts can take advantage of!

I know that all of this must seem pretty obscure, but it will
start making sense after you finish the setup and start displaying
data.

Keep the faith bro :-)

Tom