20030816: McIDAS can't seem to access data in RTPTSRC

>From: Robert Mullenax <address@hidden>
>Organization: NMSU/NSBF
>Keywords: 200308010251.h712pjLd022634 McIDAS-XCD DMSFC

Robert,

I finally got some time to login to newpsn and take a look at the problems
you have been having;

>That machine is back up now. and the LDM is running.  I still can't get any
>surface data even though wxtlist shows it to be there.  The file you
>mentioned is there.

The problem was that the scouring of McIDAS data files was _not_ working.
The result of this was that you had 10 of each kind of MD file, and
all of those files were full of old data (new data gets written to
the end of MD files until the files fill up).

The problem lay in ~ldm/decoders/mcscour.sh.  The PATH specification
in that file referenced ${MCGUI} as the first PATH directory, but
MCGUI was not defined.  The solution was simple, I added:

MCGUI=$MCHOME/bin

To the list of McIDAS environment variables at the top of the file.
After doing this, 'ldm' was able to successfully run mcscour.sh
and scour the McIDAS data files produced by XCD.

The other problem I found was an incomplete set of REDIRECTions
being defined in the 'mcidas' account.  It looks like you had gone
through and updated ~mcidas/data/LOCAL.NAM correctly, but those
definitions had not been made active:

<as 'mcidas'>
cd ~mcidas/workdata
redirect.k REST LOCAL.NAM

The redirect.k invocation above should be done while the LDM is _not_
running/running XCD.

I setup the REDIRECTions and then did the following to get newpsn
going again:

- stop the LDM

- remove all core.* files in ~mcidas/workdata

- make all the REDIRECTions in LOCAL.NAM active

- delete the MD files that had been created in the ~mcidas/workdata
  directory (by lack of appropriate REDIRECTions for all MD files

- delete all of the *.XCD, MDXX*, *.IDX, *.IDT in the /var/data/mcidas
  directory

- resetup XCD decoding:

  - reverify that XCDDATA is defined: 
    cd ~mcidas/workdata
    tl.k

  - setup XCD decoding:

    rm DECOSTAT.DAT CIRCUIT.DAT DECINFO.DAT SIGCO.DAT COUNTRY.DAT GROUPS.DAT
    batch.k XCD.BAT
    batch.k XCDDEC.BAT

- restart the LDM

While I was at it, I split up the feed requests in ~ldm/etc/ldmd.conf.

Right now I am waiting for new IDS|DDPLUS data to arrive so I can
verify that MD files are being created correctly.

<later>

OK, I see current surface data, and I can do a plot of it using SFCCON:

<as 'mcidas'>
cd workdata
mcenv
sfccon.k T USA
Accessing Dataset Name = RTPTSRC/SFCHOURLY.ALL
Number of data points input to objective analysis:  1793
PTCON:  Done
SFCCON - Done

>From address@hidden Thu Aug 21 23:10:32 2003

>In addition to problems with text data decoding on newpsn..I am also
>having problems on an identical box in New Mexico..it is called
>wxmcidas, which replaced the old wxmcidas.  Username and pw are the
>same as newpsn.
>Symptoms are similar..surface obs are there in wxtlist.k but sfclist,
>sfccon says no data.  Also ETA and GFS MOS decode doesn't work but it's
>there using wxtlist.

I logged onto wxmcidas and see that the situation appears to be
different than on newpsn in that wxmcidas does _not_ have 10 of each
type of MD file.  However, different clues point to the same problem as
on newpsn:

<as 'mcidas'>
cd workdata
cat scour.log
decoders/mcscour.sh: line 99: mcenv: command not found

I logged on as 'ldm' and found that, indeed, ~ldm/decoders/mcscour.sh
was trying to use MCGUI even though it was not defined.  I defined
it:

MCGUI=$MCHOME/bin

and ran mcscour.sh by hand. The result in ~mcidas/workdata/scour.log
is what was hoped for/expected:

$ cat scour.log
qrtmdg.k: Done
DOQTL:loop 1    to 10
Delete MD file= 5
 ...

What isn't there, at least right at the moment I am writing this, is a
current MD file for today.

<as 'mcidas'>
cd workdata
tl.k
H           := 17:03:51
XCDDATA     := /var/data/mcidas
Y           := 2003234
  --END OF LIST

The day is 2003234, but there is no MDXX0004:

dmap.k MDXX000
PERM      SIZE LAST CHANGED FILENAME DIRECTORY
---- --------- ------------ -------- ---------
-rw-  45972536 Aug 21 00:00 MDXX0001 /var/data/mcidas
-rw-  46918336 Aug 21 12:35 MDXX0002 /var/data/mcidas
-rw-  25394736 Aug 21 12:36 MDXX0003 /var/data/mcidas
-rw-  46833136 Aug 18 00:00 MDXX0008 /var/data/mcidas
-rw-  46836436 Aug 18 17:28 MDXX0009 /var/data/mcidas
211955180 bytes in 5 files

This would explain why no current surface data can be plotted.

I decided to resetup XCD on wxmcidas -- just to be safe.

<as 'mcidas'>
cd /var/data/mcidas
rm *.XCD *.IDT *.IDX
cd ~mcidas/workdata
rm DECOSTAT.DAT CIRCUIT.DAT DECINFO.DAT SIGCO.DAT COUNTRY.DAT GROUPS.DAT
batch.k XCD.BAT
batch.k XCDDEC.BAT

Hmm...  I see several core files in ~mcidas/workdata:

ls -alt core*
-rw-------    1 ldm      users      581632 Aug 21 13:44 core.1760
-rw-------    1 ldm      users      581632 Aug 16 23:00 core.23251
-rw-------    1 ldm      users      647168 Aug 16 23:00 core.23260

Interestingly, all of these are from startxcd.k:

/home/ldm% file ~mcidas/workdata/core.1760
/home/mcidas/workdata/core.1760: ELF 32-bit LSB core file Intel 80386, version 
1 (SYSV), SVR4-style, from 'startxcd.k'
/home/ldm% file ~mcidas/workdata/core.23251
/home/mcidas/workdata/core.23251: ELF 32-bit LSB core file Intel 80386, version 
1 (SYSV), SVR4-style, from 'startxcd.k'
/home/ldm% file ~mcidas/workdata/core.23260
/home/mcidas/workdata/core.23260: ELF 32-bit LSB core file Intel 80386, version 
1 (SYSV), SVR4-style, from 'ingetext.k'

I don't know what this means right at the moment, but I deleted the core
files.

<later>

After stopping and restarting the LDM and _not_ seeing any new McIDAS
MD files, I decided to take a look at ~ldm/etc/ldmd.conf.  I found that
the line that starts the XCD supervisory routine was commented out

#exec   "xcd_run MONITOR"

After uncommenting the line, and stopping and restarting the LDM, things
appear to be working correctly.  In particular, I see current surface
data:

[mcidas@wxmcidas ~/workdata]$ dmap.k MDXX000
PERM      SIZE LAST CHANGED FILENAME DIRECTORY
---- --------- ------------ -------- ---------
-rw-  44183636 Aug 22 18:20 MDXX0003 /var/data/mcidas
-rw-  36674236 Aug 22 18:22 MDXX0004 /var/data/mcidas
80857872 bytes in 2 files

and can produce a contour plot using SFCCON:

[mcidas@wxmcidas ~/workdata]$ mcenv
[mcidas@wxmcidas ~/workdata]$ sfccon.k T NA
Accessing Dataset Name = RTPTSRC/SFCHOURLY.ALL
Number of data points input to objective analysis:  182
PTCON:  Done
SFCCON - Done

The question is who turrned off running of XCD and why!?

Tom

>From address@hidden Fri Aug 22 15:03:36 2003

Man I am really slipping..I used to be able to configure McIDAS
sleep..I guess at some point a family with two kids and two jobs
catches up to you.  The thing is I followed the instructions as
always..  so I am not quite sure what happened.

As far as the XCD decode..I think I copied a ldmd.conf over from
another box late last night that had the XCD decode turned off..so I am
the culprit.

Thanks for fixing this for us.

Robert