20050511: MDXX files aren't being created right; also IMGREMAP in a script

>From: "John Hobbie" <address@hidden>
>Organization: NMSU/NSBF
>Keywords: 200505111955.j4BJtRP3013499 McIDAS-XCD MDXX

Hi John,

Sorry I couldn't get to this earlier, but I was on travel until late
Thursday when I returned to Boulder.

>Because of compatibility problems between RH9 and the latest GEMPAK
>having to do with glibc, we decided to get in step and convert to
>Fedora C3.  That resulted in having to reinstall everything,

When you say 'reinstall everything' exactly what do you mean with
respect to McIDAS?  Did you actually delete everything in your ~mcidas
directory and then unpack the source distribution and rebuild?  Knowing
exactly what was done may help me figure out what is going on with your
MDXX decoding problems.

>and even
>redo the web pages.  As a result there have been some McIDAS problems
>arise.

Hmm...  When we and at least one user site upgraded from RedHat 8/9 to
FC1 there was no need to reinstall McIDAS.  I remember remaking the
distribution on one of our systems "just to be sure", however.

>The first problem's symptom is that the sfc and upper air MDXX files
>are not being created (upper air) or populated (sfc).

Is crontab-initiated scouring of MDXX files (a cron entry in either the
'mcidas' or 'ldm' account that runs the McIDAS script 'mcscour.sh')
running?  It sounds like you might have a full set of MDXX files (i.e.,
MDXX0001 - MDXX0010 for surface; MDXX0011 - MDXX0020 for mandatory
level upper air files; MDXX0021-MDXX0030 for significant level upper
air files).  When there is a full set of files, new data will be
decoded into the end of the file until the file fills.  After that, no
new data will be decoded.

Assuming that you did not redo the needed files in the output data
directory, and since you are not decoding the data you want at all
right now, I would try the following:

<as 'ldm'>
ldmadmin stop                  <- stop the LDM

<as 'mcidas'>
cd ~mcidas/workdata
rm DCLSTIDX.PTR
cp ROUTE.SYS <<directory where you are creating the output MDXX files>
cd ~mcidas/data
cp SYSKEY.TAB <directory where you are creating the output MDXX files>
cp SCHEMA <directory where you are creating the output MDXX files>
cd <directory where you are creating the output MDXX files>
chmod 666 SCHEMA ROUTE.SYS SYSKEY.TAB

cd <directory where you are creating the output MDXX files>
rm MDXX000*
rm MDXX00[123]*

<as 'ldm'>
ldmadmin start                 <- restart the LDM

>The data are coming in and are accessible in GEMPAK, so ldm is working.

OK.

>The MDXX0001-MDXX0010 files are created and appear to have the right
>schema.

Are there 10 surface MD files?  If yes, they may be full so the decoders
can not write to the files.

Also, what does the following return:

<as 'mcidas'>
cd workdata
ptlist.k RTPTSRC/PTSRCS FORM=FILE ALL

>But the only parameter loaded appears to be the temperature
>and all stations are set to the same temperature.

Because of this comment, I would delete the MD files as in the procedure
I indicated above.

>In the case of the
>upper air, no MDXX0011-MDXX0030 files are created at all.

Now, this is very weird.

>(The profiler files area being created, MDXX0081-MDXX100.)

The wind profiler files are created by the ldm-mcidas decoder
proftomd.  The surface and upper air MD files are created by McIDAS-XCD
decoders, so it I wouldn't read anything into there being a difference
in decoding.

>A run of LSCHE ALL ALL
>shows what I think are all the required schema being registered.

Are you sure that you are reading the information out of the file
that will be used by the decoders?  To verify which copy of SCHEMA
is being used, do the following:

<as 'mcidas'>
cd ~mcidas/workdata
dmap.k SCHEMA

If this does not show SCHEMA in <directory where you are creating the output 
MDXX files>
then the LSCHE listing is not pertinent to the problem at hand.  In
this case, you would need to reestablish the set of file REDIRECTions
that is done during the McIDAS installation process.

>The
>log files have an entry for each of the three classes of MDXX files
>being created (sfc, and both profilers)

The only log indication for the surface and upper air MDXX files created
by McIDAS-XCD decoders will be comments in the log file XCD_START.LOG
that the corresponding data monitors, DMSFC for surface and DMRAOB for
upper air, have been started.  There will be no log messages showing
decoding in any LDM-specific log file (like ~ldm/logs/ldm-mcidas.log).

>and, in the case of the
>profiler files, they have entries through out the day, but no entry for
>the upper air files being created.

OK, the ldm-mcidas decoder proftomd is working correctly.

>The second problem has to do with the command IMGREMAP, and is in two
>parts. Both parts yeild the same output.  In the first case/part, the
>command was being run as part of a script file in a web page, and I
>assumed the problem was that I had not set up root and/or apache to use
>mcidas with the appropriate ~/mcidas/data directory structure so it
>could write the AREA file.   (In playing with this under the main user,
>weather, and the normal mcidas windows (not the GUI) it would run.  I
>discovered that the corresponding AREA file representing the ADDE of
>WWW/TEMP  was being created in ~weather/mcidas/data/tutorial.   Today,
>as I was getting the stuff together to write this, however, I got the
>same response--the error message-- from the mcidas window being run in
>user weather, so now I am really confused!)

I am a little confused by this.

>The error message:
>
>[weather@psnldm cgi-bin]$ imgremap.k RTIMAGES/GE-VIS WWW/TEMP.1 DSSIZE=ALL PRO
> =MERC SIZE=ALL RES=2
>***********************************************
>*                  WARNING                    *
>* The entire source image will be used to     *
>* create the destination image. If the source *
>* image is located on a remote server, the    *
>* total number of bytes transfered will be:   *
>*                  4.50 MB                    *
>***********************************************
>Beginning Image Data transfer, bytes= 4506336
>imgremap.k: transformations complete ... begin data move
>Transferring AREA data outbound, bytes= 94197728
>imgremap.k: Error writing area directory
>imgremap.k: Failed to write comment block

This seems to indicate that IMGREMAP is trying to write to a file for
which it does not have write permission.  The file that it will try to
write to will be governed by the definition of the WWW/TEMP dataset and
any file REDIRECTions you have for AREA files in the number range
included in that ADDE definition (i.e., if WWW/TEMP is defined as the
set of AREA files from 3000 to 3005, then WWW/TEMP.1 will correspond to
AREA3000; WWW/TEMP.2 will correspond to AREA3001; etc.).  Where the
file will be read/written will be through the definition of MCPATH or
through a REDIRECTion if one is in scope for those AREA files.

So, the pieces of information that are needed are:

- definition of WWW/TEMP:

dsserve.k LIST WWW

- REDIRECTions that may be in scope:

redirect.k LIST

>I did try to create a mcidas/data directory structure in the cgi-bin
>directory of apache and that did not help.

The user that the cgi-bin script runs as will need to have McIDAS
environment variables defined (e.g., MCDATA, MCPATH, MCTABLE_READ,
MCTABLE_WRITE, etc.) so that McIDAS routines run by it will know where
to read/write data files.  Just creating a mcidas/data directory will
not have any effect unless MCPATH has been defined to use that
directory.

>In fact, it was established
>AFTER I successfully ran the command in the mcidas window in as user
>weather, though I don't think that was what screwed up the correct
>running of imgremap in weather today.

Your cgi-bin scripts will be run as 'root' or some other user with
super user privilege.  It is for that user that the McIDAS environment
variables have to be defined.  The best way to do the definitions is to
define them in the script being run (as illustrated in the example
files 'mcrun.sh' and 'mcbatch.sh' included in the McIDAS
distribution).

>So.......... what things should I look for to determine what is wrong
>with the MDXX files?

See above.

>And ........ what is the diagnostic/error message telling me about
>imgremap?

It is seening a write failure.

>And, can imgremap be run in a web page script?

Yes, any McIDAS routine can be run from a script if the script defines
needed McIDAS environment variables.

>If so, do I need to
>establish an mcidas directory structure to create the AREA file in?
>Who are the owners and what permissions?

No, but the directory in which the the output from McIDAS commands
is written must be writable by the user running the cgi-bin script.

>This is long winded -- sorry for that.   But thanks in advance for all
>your help.

I'll have to admit that I am uncertain about exactly what you did
when upgrading from RH9 to FC3 (i.e., I don't know how far you dropped
back in trying to get McIDAS back up and running).  Since I am no
expert in running cgi-bin scripts (I have helped folks at U South Florida
in the past, but that was over a year ago), I am unable to pinpoint
exactly what in the upgrade would have broken what you had).

I am more than willing to logon to your system and help you get
things working again.  Please let me know if you would like me
to do this (and are able to allow me to do this by appropriately
configuring access).

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.

>From address@hidden  Mon May 16 15:44:10 2005

Hi Tom --

Thanks for your help.  All appears to be working fine, now.

To follow up on some of the questions you asked:

First the easy one about imgremap.   I hadn't checked the redirections,
and AREA8* was set to ~/mcidas/data/tutorial, which didn't exist.  A
simple fix reseting WWW/TEMP to AREA7000 - 7020 and redirecting to a
directory under apache, and all worked fine.

Second problem about the missing MDXX files is more complicated.  All
our machines have three hard drives now.  This was the first machine
that I was reconfiguring to be in a standardized file structure, and as
best I can, into the suggested unidata layout for ease of maintenance.
When we hit the problem that the new GEMPAK (now the previous version
as of today) had  with an outdated glibc library under RH9, we decided
to convert to Fedora.   Because of the future reconfiguring of other
machines, we decided to let Fedora reformat the main disk so everything
would be guaranteed the same -- the other two are data disks only.
After saving script files, pqact.conf, the web page scripts, and other
critical files, we reformated figuring a clean install of the unidata
quartet of ldm, mcidas,gempak, ldm-mcidas would ensure there were no
legacy problems.  The installations went quickly and easily.  I had
your old emails, my notes and the unidata installation manuals.

But what went amok I am not sure.  After I wrote this request, I had
another apparent problem and Griz helped on that.   It turned out what
I wrote Griz about was due to the primary disk with the OS on it being
full and that it wasn't a GEMPAK problem; I transfered or deleted some
large files and made room.  Then on Saturday, I checked and it was full
again.  A check of the system didn't yield any obvious problems; did
du's  and looked for big files, ipcs, etc.  Anyway, I did a reboot and
the disk load dropped to 17% and has stayed there.   And amazingly, the
sfc data MDXX000x for Sunday was populated but the RAOB was not
created.  After reading your email today, I looked at the XCD_START.log
and found  that the DMRAOB decoder had started after the reboot but
that there were error flags for it and for the synoptic and the misc
decoders (I didn't know to read that log till now.)  I then went back
and stopped ldm and followed your instructions, and all began working.
However, I noted that there was no DCLSTIDX.PTR found anywhere in the
mcidas directories; still isn't after following your instructions and
getting everything working again.  Do I still have a problem?

I have reflected on what I could have done to screw things up.  I am
not sure, and I haven't tried yet to dig it out of the manuals, but I
have a vague recollection of having to transfer the SCHEMA, SYSKEY.TAB
and ROUTE.SYS found in ldm-mcidas (not mcidas) to the directory where
you creating the output MDXX file  (which is, by the way,
/data/ldm/mcidas).     If that is so, could that have caused the
problem because of differences between the two programs?   Also, for
future use in setting up the other three machines, is there a proper
order to install things, eg. mcidas first then ldm-mcidas?

Again, thanks for all your help.  You guys are great.

Hobbie