19990820: mcidas7.6...response (cont.)

>From: address@hidden
>Organization: SMSU
>Keywords: 199908122231.QAA22964 McIDAS-X,-XCD 7.60

Bill,

re: disk filling problems
>*****
>*Sorry.  I didn't mean to imply that mcidas had these problems.  I meant
>*when I made a mistake, things like getting your disks filled up
>*were a given with the volume of data...that was part of yesterday's 
>*problem, and as you address below, my solutions caused the additional
>*problems....My fault, not mcidas's.

One of the problems with the data and software nowadays is that they both
require a LOT of disk space.  I tell people that are going out to get
new machines to think about reserving upwards of 9 GB of disk for data.
It is easy to consume 9 GB of disk with data alone if one decides to keep
several days of data online, especially McIDAS data.

>But it does remind me of a quote
>*from the 70's...the era of jogging...George Sheehan, the "running doctor"
>*opined that man's ability to lay concrete had far outstripped the
>*ability of our feet to evolve.  I fear that mcidas's complexity has
>*outrun my ability to keep up (but not screw up).

I imagine that the problem is that you don't use McIDAS on a daily
basis, so its quirks/features never become second nature.

>*On a brighter note, we have staffing problems here, and my brand 
>*spanking new Department Head offered me all sorts of blandishments,
>*such as magnificent travel allotments, if I would teach an extra
>*section of atmospheric science.  So I guess a visit to Boulder
>*will be in the works sometime in the next year.
>*****

Sounds good.  I look forward to you visiting again.  We can drown our
sorrows over some very good local micro brewed beer.

re: REDIRECTion for STRTABLE
>*******
>*I went and set it back to current directly.  Dis is dot.
>******

re: where is SYSKEY.TAB
>*****
>* It is /home/ldm/data/mcidasd   where it oughta be
>****

Well, this is true if the REDIRECTion to it is setup correctly in
the 'mcidas' account, and if all scripts (including xcd_run and batch.k)
that do automated things have the ~mcidas/workdata as their MCDATA
directory, AND $MCDATA is the first directory in MCPATH.  The ldm-mcidas
decoders do not use McIDAS search facilities (i.e. REDIRECT and MCPATH)
to find files that they need to read/write/update (e.g. ROUTE.SYS,
SYSKEY.TAB, and SCHEMA).  McIDAS-XCD decoders, on the other hand,
_do_ use the McIDAS search facilities to find files.  This is why
it is probably OK to have SYSKEY.TAB in the ldm-mcidas decoder output
directory and not in those used by XCD (if the REDIRECTion is setup
correctly).

re: web server
>*****
>*cirrus.smsu.edu
>****

That is why I didn't remember.  It was too easy!

re: correct way to move data files to a new directory
>******
>* Acutally, I did it right (within this limited context)...stopped
>* ldm, moved files, started mcidas, redirected, exitted mcidas,
>* started ldm.

OK.  You really are on top of this.

>* I believe you are getting to my problems down below, but what 
>* has me concerned is that the batch jobs are not, in fact,
>* inheriting the mcidas environment in all ways; some ways, but
>* not all.

This is scary.  It seems to me that either they should inherit
the environment or not.  C shell users, on the other hand, have the
problem that mcenv will read .cshrc upon startup.  This has the
bad effect that the MCPATH setting so carefully done at the top of
the script gets replaced by the one in .cshrc.  You might think that
this would not cause problems, but what is going on behind the scenes
is that McIDAS appends the directory that it creates for the new session
on the end of MCPATH (the new directory is a subdirectory of .mctmp).
Apparently, this directory gets appended onto the end of MCPATH and
then .cshrc is sourced.  This causes all kinds of bad things to happen.
The situation is so bad, that I had to put special instructions into
the installation/configuration instructions for C shell users.  The
workaround consists of put a conditional around the definition of
MCDATA, MCPATH, etc. in .cshrc.  If MCPATH is already defined, then
the environment setting lines are skipped.  I wonder if you are somehow
experiencing this problem even though you use the Korn shell.

Hmm  after thinking about this more, I am suspicous of the .cshrc
file in your 'mcidas' users home directory (actually, I am concerned
with cshrc.aix which is sourced by .cshrc).  I put these files there
some time ago when I was working on your system.  I see that
cshrc.aix does NOT have the conditional around the setting of MCDATA,
MCPATH, and MCGUI.  I never expected this to be a problem, but...
In order to insure that this does not cause problems, I added
the conditional clause to cshrc.aix:

changed:

setenv MCDATA $HOME/workdata
setenv MCPATH ${MCDATA}:$HOME/data:$HOME/help
setenv MCGUI  $HOME/bin

to:

if ( ! ${?MCPATH} ) then
setenv MCDATA $HOME/workdata
setenv MCPATH ${MCDATA}:$HOME/data:$HOME/help
setenv MCGUI  $HOME/bin
endif

If this is the cause of your flakey problems, then I will be very
suprised since you are running all of your scripts in the Bourne/Korn
shell (true?).

>The problem of the moment, for example, is that the
>* shell scripts do a cd $MCDATA, which should make workdata the
>* default directory...

No, not necessarily.  The default directory for all routines using McIDAS
file location facilities will use $MCDATA IF the $MCDATA directory is
the first directory in MCPATH.  One still must cd to $MCDATA so that
file REDIRECTions defined with a directory of "." will work.

Any routine that does not use McIDAS file location facilities will get
$MCDATA as the current directory by virtue of the 'cd $MCDATA'.

>*In fact, output from the shell script 
>* (which I save to count various things) is now going to ~mcidas/data
>* rather than workdata...the code itself has been working for a year,

How is the output from the shell being generated?  If it is being done
through a DEV=T file_name construct, then the output will go into a
file in /home/mcidas/data if it already exists there.  This again is
by virtue of the McIDAS file location facilities.  Now, IF you put
in a REDIRECTion for the file that specifies that it should be in the
/home/mcidas/workdata directory, then that is where the ouput will
be written.  This unexpected behaviour is one of the hardest things
in McIDAS to grasp.  If you don't understand what I am getting at,
we need to spend more time on the concept 'cause this is a big, black
hole that one can fall into.

>* I've just got these environmental things screwed up....the code
>* is the /home/mcidas/src files eta.sh, eta_1.sh and eta_2.sh,
>* along with ETA_L.BAT, ETA_1.BAT, and ETA_2.BAT on the ~mcidas/data
>* directory.

OK.

>* eta.sh CD $MCDATA, then runs ETA_L.BAT, which does an IGG LIST on
>* what it thinks is the correct GRID file, with a DEV=T grid1.list
>* This way I can count the number of lines in the file and see if 
>* there's a high number, which would indicate it is safe to run
>* a mapping job....good likelihood that the maps made will be data...

Hopefully, this script is not still expecting the GRID files that were
coming in the Unidata-Wisconsin datastream.  You should know that those
GRID files were removed on July 1, 1999.  If you need those GRID files,
then there is a new routine in 7.6 called UWGRID.  UWGRID can be used
to recreate the files that used to come in the broadcast for  those
sites that have not updated their scripts to use the XCD produced GRID
files (the XCD produced GRID files have more data, but the ordering of
the grids in the file is not fixed.  The file gets populated however
the grib products come in.

>* mcidas is pretty stupid about being able to check to see if 
>* something is available before making a map....

Yes.

>my jobs save things
>* as gif files, and if data is not there I get blank products when, in
>* fact I'd just as soon save the previous hour's maps....so this is 
>* my work around.

Got it.

>Whew! Sorry...the point is, these files of output
>* from IGG LIST have always shown up on ~mcidas/workdata.  Today 
>* they're showing up on ~mcidas/data.  
>*****

OK, I looked in the BATCH files that you have added to ~mcidas/data
and see the construct that I was expecting.  For example, take
the IGG LIST command in ETA_L.BAT:

IGG LIST GRIDF=#?GR1 DEV=TTT grid1.list

The 'DEV=TTT grid1.list' keyword sequence says to McIDAS to write the
output to the text file 'grid1.list'.  It doesn't say where 'grid1.list'
is to be found, so it will use the McIDAS file location facilities
to locate it.  If the file was found in ~mcidas/workdata, then that is
where it would be written.  If it is found in ~mcidas/data, then that
is where it will be written.  So, the gotcha is that once 'grid1.list'
was written into ~mcidas/data, that is where it will stay.  This is
true UNLESS you give an explicit REDIRECTion that says it should always
exist in ~mcidas/workdata.  I just did a quick listing of your
REDIRECTions, and I see that you do not have any REDIRECTion for
files 'grid1*' or 'grid1.li*' (I don't think that 'grid1.list' will
work since McIDAS is still anal about the old 8.3 naming conventions).

I think that this mystery is now solved.

re: is XCD working
>****
>* All kinda data coming in through XCD onto /home/ldm/data/xcd
>****

OK, I can see that.

re: location of SYSKEY.TAB, ROUTE.SYS, and SCHEMA
>****
>* Very dim bulbs are starting to illuminate in my brain....this is the
>* first time I have seen it put this way (and below).  Think I can
>* handle it.
>****

OK.

re: leading spaces in .mcenv
>*****
>* I think not.  Although I did edit that when I was suffering
>* an attack of angst over ~mcidas/data vs. ~mcidas/workdata, the
>* spaces were always there.  Believe I cut and pasted that bit
>* of code out of an example page on your web pages.
>****

I don't think so either.  I was just being anal myself.  A space after
the '=' would have been fatal.

re: deleting bad MD file #1
>****
>* stopped ldm, started mcidas, deleted file, started ldm
>****

Very good.  A PTLIST RTPTSRC/PTSRCS.ALL FORM=FILE listing now shows
that things look good:

PTLIST RTPTSRC/PTSRCS.ALL FORM=FILE
Pos      Description                        Schema  NRows NCols  Date
------   --------------------------------   ------  ----- ----- -------
     3   SAO/METAR data for   21 AUG 1999   ISFC       72  4500 1999233
     4   SAO/METAR data for   22 AUG 1999   ISFC       72  4500 1999234
    13   Mand. Level RAOB for 21 AUG 1999   IRAB        8  1300 1999233
    14   Mand. Level RAOB for 22 AUG 1999   IRAB        8  1300 1999234
    23   Sig.  Level RAOB for 21 AUG 1999   IRSG       16  6000 1999233
    24   Sig.  Level RAOB for 22 AUG 1999   IRSG       16  6000 1999234
    33   SHIP/BUOY data for   21 AUG 1999   ISHP       24  2000 1999233
    34   SHIP/BUOY data for   22 AUG 1999   ISHP       24  2000 1999234
    43   NGM MOS for day      21 AUG 1999   FO14       38   600 1999233
    44   NGM MOS for day      22 AUG 1999   FO14       38   600 1999234
    53   SYNOPTIC data for    21 AUG 1999   SYN         8  6000 1999233
    54   SYNOPTIC data for    22 AUG 1999   SYN         8  6000 1999234
PTLIST: Done

>****
>* Disk space has always been a pain.  I note that this release of
>* mcidas fixes a bug in a scouring routine....don't know if I 
>* experienced that bug, but I had an e-mail to you last year about
>* this...mcidas seemed to want to preserve files for 2 days or 0
>* days...

The thing I corrected was not this.

>anyway, I put in ldm scouring on some files because they
>* were voluminous and didn't scour on what I considered an predictable
>* schedule....It's been working like this, but I will (eventually)
>* get back to it and change it to the correct method.
>****

What you did was correct.  The problem a lot of sites create is when
they use the LDM's scouring routine to scour McIDAS directories.  Then
files that don't change, like SCHEMA, eventually get deleted and decoding
starts breaking.

re: copies of SCHEMA and SYSKEY.TAB in the directory where XCD-produced
MD files live
>****
>* Will do.  But ya know?  I think it's more than this.
>****

I do too.

re: FSL wind profiler and NLDN lightning data
>****
>* we took the NLDN feed from Albany (?) for a while, but when I e-mailed
>* the guy running it about putting the data on web pages, he was adamant
>* that we not do that.  Proprietary data.

Absolutely right.  The agreement with the company that owns the data is
that it will _never_ show up on a web page.  If it does, all the universities
getting the data are likely to lose that privilege.

>* So I took the feed out of the ldm.
>* I don't know that the other thing is nor how to use it, so ignore it.
>* Bliss.
>****

The profiler data is very cool especially for sites in the midwest (the
profilers are basically in the midwest.  To get an idea of what the
data looks like, try running the following command from a McIDAS session:

DATALOC ADD RTPTSRC ADDE.UNIDATA.UCAR.EDU
PROFDISP BLMM7 #Y 24 -60
DATALOC ADD RTPTSRC CIRRUS.SMSU.EDU

There is both hourly summary and 6 minute profiler data in the FSL2
stream that can be captured with the LDM and turned into McIDAS MD
files with the ldm-mcidas decoder, proftomd.

re: cause of flakiness
>****
>* undoubtedly...but I think we have one more tete-a-tete, and it
>* will concern the environment of shell scripts.

I think that the comments above will help put to rest one of the
mysteries that you were confronted with.

>* OK, and here's another....just before sending this, got on the
>* web to check our maps...surface producs all fine 20Z...
>* Of course, 500/700/850 mb producs screwed from this morning, so
>* MDU LIST  showed them there, fine. Interactive use of mcidas to
>* run my 500MB.BAT (on ~mcidas/data, where it's always been), and 
>* I get: 
>* BATCH 500.BAT                                                               
>     
>* BATCH: Error reading file 
>* /home/mcidas/workdata/500.BAT                         
>*
>* I have to include a path to the file!  Never had to before.

No.  I did the following to check this out:

<login as mcidas>
cd workdata
dmap.k 500.BAT
PERM      SIZE LAST CHANGED FILENAME DIRECTORY
---- --------- ------------ -------- ---------
0 bytes in 0 files

There is not 500.BAT file to be found in any of the directories in MCPATH.
The reason the error names the /home/mcidas/workdata directory is because
this directory is the first writable directory in your MCPATH.  After
McIDAS couldn't find the file anywhere in MCPATH (note the DMAP output
above), it tells you that it couldn't find the file in the first of
the MCPATH directories.  It should probably just say that 500.BAT was
not found, but...

>* MCGUI=/home/mcidas/bin
>* MCTABLE_WRITE=/home/mcidas/data/ADDESITE.TXT
>* MCPATH=/home/mcidas/workdata:/home/mcidas/data:/home/mcidas/help
>* MCENV_NEGUC=
>* MCDATA=/home/mcidas/workdata
>* MCHOME=/home/mcidas
>*

The 500.BAT mystery is an easy one; it probably got deleted in your
initial deleting of your 7.402 installation.  This is a good reason
to put your site developed BATCH and McIBAS scripts in the
/home/mcidas/workdata directory.  They won't be deleted there when
a new version is installed.  500.BAT probably should not have been deleted
from /home/mcidas/data since there is no .BAT file in the distribution
of that name, but you could be unlucky enough to name your BATCH file
the same as something in the distribution in the future.

Let me know if the above helped.

Tom