20050121: Unidata McIDAS-XCD Trouble (cont.)

>From: "James T. Brown" <address@hidden>
>Organization: Michigan State University
>Keywords: 200501201933.j0KJXOv2020815 McIDAS-XCD ADDE

Hi Jim,

>Thanks for the reply.   I checked the notes that you sent and
>for the most part, the steps of the installation seem to match
>the steps I have taken.

OK.  I thought this was the case given your excellent reporting of
your situation, but I had to list out the steps just in case...

>Some of my installation folders are
>different, but all of the proper environment variables should
>be set.

Sounds good.

>Only main difference is the "SCHEMA", "SYSKEY.TAB",
>and "ROUTE.SYS" files that reside in my /data/mcidas folder
>came from the "ldm-mcidas" decoders and not from the McIDAS
>installation - any problem with that?

Depending on which ldm-mcidas version you installed, there should
be no problem.  The McIDAS v2004a distribution has the latest
versions of these files, so if you installed v2004a it would be
best to use the copies of the files from it.

>If you would like to connect to my ADDE services for the
>RTPTSRC group, my server name is:
>
>   pileus.geo.msu.edu

OK, thanks.  I am able to contact your ADDE server so the installation
looks OK.  I note, however, that you do not have an surface MD file
for today; it would be MDXX0001.

>Here is a new listing of the MDxx files in my "/data/mcidas"
>folder:
>
>   % ls -lm MD*
>   -rw-rw-r--   1 9002     wx       49999236 Jan 21 09:32 MDXX0009
>   -rw-rw-r--   1 9002     wx        338536 Jan 20 18:48 MDXX0010
>   -rw-rw-r--   1 9002     wx         44560 Jan 19 18:49 MDXX0019
>   -rw-rw-r--   1 9002     wx        151312 Jan 20 02:53 MDXX0020
>   -rw-rw-r--   1 9002     wx         16640 Jan 19 18:49 MDXX0029
>   -rw-rw-r--   1 9002     wx        112752 Jan 20 13:38 MDXX0030
>   -rw-rw-r--   1 9002     wx       8071800 Jan 21 09:32 MDXX0059
>   -rw-rw-r--   1 9002     wx        887448 Jan 20 06:51 MDXX0060
>   -rw-rw-r--   1 9002     wx       5951016 Jan 20 01:45 MDXX0069
>   -rw-rw-r--   1 9002     wx         17360 Jan 20 13:37 MDXX0070
>
>Other than some changes in some of the modification times, most of
>the file sizes are the same as the listing that was producded yesterday.

The surface MD file for yesterday, MDXX0010, is too small to hold
much data.  A full MD file will be on the order of 50 MB per day.

>Here is the latest output from the "PTLIST" file listing:
>
>   % ./ptlist.k RTPTSRC/PTSRCS FORM=FILE ALL
>   Pos      Description                        Schema  NRows NCols Proj# 
>Created DataDate
>   ------   --------------------------------   ------  ----- ----- ----- 
>------- --------
>        9   SAO/METAR data for   19 JAN 2005   ISFC       72  7000     0 
>2005019 2005019
>       10   SAO/METAR data for   20 JAN 2005   ISFC       72  7000     0 
>2005020 2005020
>       19   Mand. Level RAOB for 19 JAN 2005   IRAB        8  1500     0 
>2005019 2005019
>       20   Mand. Level RAOB for 20 JAN 2005   IRAB        8  1500     0 
>2005020 2005020
>       29   Sig.  Level RAOB for 19 JAN 2005   IRSG       16  6000     0 
>2005019 2005019
>       30   Sig.  Level RAOB for 20 JAN 2005   IRSG       16  6000     0 
>2005020 2005020
>       59   SYNOPTIC data for    19 JAN 2005   SYN         8  6500     0 
>2005019 2005019
>       60   SYNOPTIC data for    20 JAN 2005   SYN         8  6500     0 
>2005020 2005020
>       69   PIREP/AIREP data for 19 JAN 2005   PIRP       24  1500     0 
>2005019 2005019
>       70   PIREP/AIREP data for 20 JAN 2005   PIRP       24  1500     0 
>2005020 2005020
>   ./ptlist.k: Done

I can get the same listing from here.

>I am no expect in McIDAS, but shouldn't there be MDxx files ending
>with "1" for today's date (Jan 21)?

Yes, absolutely.

>You asked for output of the "PTLIST RTPTSRC/SFCHOURLY" command:
>
>   % ./ptlist.k RTPTSRC/SFCHOURLY
>   Row :      1  Col :    885
>   TYPE        =         0             | DAY         =   2005020 
>CYD         |
>   TIME        =         0 HMS         | NREC        =         
>3             |
>   ID          =      KPIH             | LAT         =   42.9202 
>DEG         |
>   LON         =  112.5711 DEG         | ZS          =      1356 
>M           |
>   ST          =        ID             | CO          =        
>US             |
>   MOD         =         0             | HMS         =    234600 
>HMS         |
>   CIGC        = _missing_             | CC1         = 
>_missing_             |
>   CC2         = _missing_             | CIGH        = 
>_missing_             |
>   ZCL1        = _missing_             | ZCL2        = 
>_missing_             |
>   VIS         =       1.5 MI          | WX1         = 
>_missing_             |
>   WX2         = _missing_             | T           =    277.16 
>K           |
>   TD          = _missing_             | DIR         =         0 
>DEG         |
>   SPD         =       0.0 MPS         | GUS         = 
>_missing_             |
>   PSL         = _missing_             | PCP         = 
>_missing_             |
>   SNO         =         4 IN          | PRE         = 
>_missing_             |
>   P24         = _missing_             | WXC1        = 
>_missing_             |
>   WXC2        = _missing_             | WXC3        = 
>_missing_             |
>   WXC4        = _missing_             |
>   
>---------------------------------------------------------------------------
>   Number of matches found = 1
>   ./ptlist.k: Done

The reason I asked for the listing was to see if data was being decoded.
It is, but there should be lots more data than is represented by the
size of yesterday's surface MD file.

>I've used McGUI to attempt to plot Meteorograms and most stations produce
>a blank screen (station=KPIH from the record listed above only plots data
>from the entry listed above).

I tried as well and see almost no data in your MD files.

>If you would like to logon to take a look around, feel free - I would be
>very interested if you are able to determine the source of the problem.
>
>I have set a temporary password on the "mcidas" account - you should be
>able to SSH into the machine where McIDAS and McIDAS-XCD have been
>built:

OK.

>My LDM is running on another machine which is behind a firewall, but
>is temporarily accessible via SSH from the "pileus" machine (using
>the same McIDAS username and password listed above):
>
>  LDM HOST:
>
>     hostname:      accas.geo.msu.edu
>     LDM home dir:  /soft/ldm
>
>Let me know if you would like temporary access to the "ldm" user
>account.

I logged on and see one major problem right off.  'mcidas' does not
have write permission for the /data/mcidas directory:

> hostname
accas
> touch /data/mcidas/xxx
touch: /data/mcidas/xxx cannot create

I also see that files that should have been created by the XCD setup
BATCH scripts do not exist in /data/mcidas.  I feel confident that this
is one (and perhaps the only) reason your decoding is not working
correctly.

The first part of the fix is to add group write permissions to
/data/mcidas.  The easiest thing to do next is to redo the XCD
configuration from scratch (this will take only a couple of minutes):

- as 'ldm':

ldmadmin stop
cd /data
chmod 775 mcidas
cd /data/mcidas
rm *

- as 'mcidas' do the following:

cp ~mcidas/data/SCHEMA /data/mcidas
cp ~mcidas/data/SYSKEY.TAB /data/mcidas
cp ~mcidas/workdata/ROUTE.SYS /data/mcidas
cd /data/mcidas
chmod 664 SCHEMA SYSKEY.TAB ROUTE.SYS
mkdir bufr grib

cd ~mcidas/workdata

rm DECOSTAT.DAT
rm DCLSTIDX.PTR

batch.k XCD.BAT
batch.k XCDDEC.BAT

At this point, you should have a set of files in /data/mcidas that
were created by the BATCH scripts.  If you only have SCHEMA, SYSKEY.TAB,
ROUTE.SYS it means that you still have some sort of permission problem.

- if the BATCH scripts were successful in producing additional files
  in /data/mcidas, you should restart your LDM:

<as 'ldm'>
ldmadmin start

>I can certainly try to rebuild things again later in the day
>and can combine the ADDE services on the same machine as the
>LDM if that would make any difference.

That should not make any difference.  The problem is that some of the
files needed in XCD decoding do not exist in /data/mcidas.  The reason
they don't exist is that 'mcidas' does not have write permission for
the directory (and it needs to!).

>However, if you are
>able to login and take a quick look, I appreciate any suggestions
>that you may have before I do anything too drastic.

The fix should be quick and easy, and you should see proper decoding
almost immediately.

>Let me know if you have any trouble accessing the machines and
>accounts listed above.

No problems, thanks for the access.  It was very useful to be able to
see your setup.  I would not have thought about /data/mcidas not being
open for writing to 'mcidas' until all other things have been
exhausted.

One last thing.  The set of DATALOCs that are setup on in the 'mcidas'
account are not quite correct given the data you are ingesting.

> dataloc.k 

Group Name                    Server IP Address
--------------------         ----------------------------------------
AMRC                         UWAMRC.SSEC.WISC.EDU
BLIZZARD                     ADDE.UCAR.EDU
CIMSS                        PILEUS.GEO.MSU.EDU
GINICOMP                     PILEUS.GEO.MSU.EDU
GINIEAST                     PILEUS.GEO.MSU.EDU
GINIWEST                     PILEUS.GEO.MSU.EDU
GOESEAST                     PILEUS.GEO.MSU.EDU
ME7                          IO.SCA.UQAM.CA
MYDATA                       <LOCAL-DATA>
NEXRCOMP                     PILEUS.GEO.MSU.EDU
RTGRIDS                      PILEUS.GEO.MSU.EDU
RTIMAGES                     PILEUS.GEO.MSU.EDU
RTNEXRAD                     PILEUS.GEO.MSU.EDU
RTPTSRC                      PILEUS.GEO.MSU.EDU
RTWXTEXT                     PILEUS.GEO.MSU.EDU
TOPO                         <LOCAL-DATA>

<LOCAL-DATA> indicates that data will be accessed from the local data directory.
DATALOC -- done

In looking at the request lines in your ~ldm/etc/ldmd.conf file, I see
that you are not ingesting the data that would allow you to have the
data files necessary for:

GINICOMP, GINIEAST, and GINIWEST - NOAAPORT GOES imagery available in the
                                   NIMAGE IDD datastream 

GOESEAST                         - GOES satellite imagery available
                                   only from the ADDE server on
                                   unidata2.ssec.wisc.edu

NEXRCOMP                         - NEXRAD Level III composite imagery
                                   available in the FNEXRAD IDD datastream

RTGRIDS                          - GRID decoding is turned off

RTNEXRAD                         - NEXRAD Level III products available
                                   in the IDD NNEXRAD datastream

Here is what I see for data request lines in ~ldm/etc/ldmd.conf:

> grep ^request ~ldm/etc/ldmd.conf
request UNIDATA|FSL     ".*" f5.aos.wisc.edu PRIMARY
request NLDN    ".*" striker2.atmos.albany.edu PRIMARY
request FSL3    "MSR_mpe" crgateway.ncrfc.nws.gov
request CONDUIT "/MT.ruc_CY" f5.aos.wisc.edu

So, if you have no intentions of ingesting data needed for the above
datasets, I suggest that you do the following:

<as 'mcidas'>
cd ~mcidas/data

edit LOCDATA.BAT and set it up to look like the following:

DATALOC ADD CIMSS     pileus.geo.msu.edu
DATALOC ADD GINICOMP  pscwx.plymouth.edu
DATALOC ADD GINIEAST  pscwx.plymouth.edu
DATALOC ADD GINIWEST  papagayo.unl.edu
DATALOC ADD GOESEAST  unidata2.ssec.wisc.edu
DATALOC ADD NEXRCOMP  adde.ucar.edu
DATALOC ADD RTGRIDS   adde.ucar.edu
DATALOC ADD RTIMAGES  pileus.geo.msu.edu
DATALOC ADD RTNEXRAD  atm.geo.nsf.gov
DATALOC ADD RTPTSRC   pileus.geo.msu.edu
DATALOC ADD RTWXTEXT  pileus.geo.msu.edu
DATALOC ADD TOPO      LOCAL-DATA

Leave the other DATALOC settings in LOCDATA.BAT the same and then do:

cd ~mcidas/workdata
batch.k LOCDATA.BAT

After doing this, the 'mcidas' account will be setup to access data in
the datasets you are not ingesting data for from remote ADDE servers.

If you do have intentions for ingesting data necessary for the datasets,
there is still some work to be done to define the datasets.  We can
talk about this in detail if/when you decide to change what you are
ingesting.

>Thanks.

No worries.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.