20051128: some AREA file not being updated (cont.)

>From: Myron Kowalski <address@hidden>
>Organization: Moravian
>Keywords: 200511281523.jASFNF7s008982 ldm-mcidas McIDAS

Hi Myron,

This is a long email, please read everything before making any changes...

re: ROUTE.SYS and SYSKEY.TAB
>The files ROUTE.SYS and SYSKEY.TAB are in the right place.

OK.  The reason I mentioned them is that ROUTE.SYS governs whether
or not the pqact.conf action you sent in your original email:

MCIDAS  ^pnga2area Q. (..) (.*) (.*) (.*) (.*) (........) (....)
        PIPE    -close
        pnga2area -v
        -a /users/ldm/etc/SATANNOT
        -b /users/ldm/etc/SATBAND
        -d /mcidasdata/mcidas -r \1,\2

actually decodes products.  This may sound a bit strange given
that the LDM is configured to decode images, but it something that
one has to get used to especially with older McIDAS releases.

>The particular files I'm looking for are the ones that give
>the air contour plots.

The files you need are the MDXX files for mandatory level upper
air data:

MDXX0011 ... MDXX0020

>Pressure gradients for PA. The Stuve diagrams.

For thermodynamic diagrams, you will need both the mandatory level
upper air MD files and the significant level upper air MD files:

MDXX0021 ... MDXX0030

Given that your McIDAS installation is now old, you may not be able
to just FTP valid files from some other machine and use them.  McIDAS,
however, has another way depending on how old your distribution is
(what version are you running (cat ~mcidas/data/VERSION.TXT)?).
McIDAS has the ability to get the data from remote ADDE servers.
All your system needs is:

- any/all firewalls allow bidirectional traffic on ports 112, 500
  and 503

- the user 'mcidas' or each individual user configure "pointing" at
  the remote server(s) using the DATALOC command.

  Examples:

  <user 'mcidas'>
  cd ~mcidas/workdata
  dataloc.k ADD RTPTSRC adde.ucar.edu


  <user that is not 'mcidas'>
  cd ~/mcidas/data
  dataloc.k ADD RTPTSRC adde.ucar.edu

  After doing this, the user(s) should be able to produce upper air
  contour plots and thermodynamic diagrams as long as its machine
  is allowed to communicate on ports 500 and/or 503 (McIDAS v2004
  and previous: 500 for uncompressed transfers; 503 for compressed
  transfers; v2004 and v2005: port 112 for any transfer)
  
>Without having the class syllabus in front of me, I can't remember.
>If you look in /mcidasdata/mcidas, you can see which ones haven't been
>updated.

I just logged onto batboat as 'mcidas'.  I see that there are what
appears to be decoded surface and upper air MD files.  What I quickly
learned, however, is that the mandatory level upper air file does
not have data for 0 or 12Z for today.  This could be due to LWPATH.NAM
being incorrectly configured as you mentioned earlier.  It may be
the case that the data will be there tomorrow when you need it.  To
be sure, however, I would stop and restart the LDM:

<as 'ldm'>
ldmadmin stop
<wait until LDM has exited completely>
ldmadmin start

The reason for the restart of the LDM is so that McIDAS decoders
get restarted and use the modified version of ~mcidas/workdata/LWPATH.NAM.

Back to the AREA files that are not being updated:  I just listed the
contents of ROUTE.SYS from the 'mcidas' account and see that the file
is damaged:

batboat% dmap.k ROUTE.SYS
PERM      SIZE LAST CHANGED FILENAME  DIRECTORY
---- --------- ------------ --------- ---------
-rwx     16384 Nov 28 18:30 ROUTE.SYS /mcidasdata/mcidas
16384 bytes in 1 files
batboat% route.k LIST
S Pd         Description         Range       Last      Received  Post Process C
- -- ------------------------- --------- ------------ ---------- ------------ -
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
 ...
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
   
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213  62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
  CA Cloud Top Pressure        1100-1109 AREA1109     05332 2145 CTP.BAT      3
  CB Precipitable Water        1110-1119 AREA1110     05332 1631 PW.BAT       3
  CC Sea Sfc. Temperature      1120-1129 AREA1120     05332 1147 SST.BAT      3
 ...
  UY Mollweide Composite H2O    110-119  AREA0118     05332 1434     none     3
  a< 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213 AREA0118     62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4
  \ufffft 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff
 2144--213 AREA0118     62144 0621 
\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff\uffff 4

The version of the ldm-mcidas decoder 'pnga2area' that you are using
had a bug in it that corrupted ROUTE.SYS.  The corruption is as
listed above.

So, I logged onto catwoman and did the following:

<as 'ldm'>
cd ~ldm
mkdir ldm-mcidas
mv ldm-mcidas.tar.Z ldm-mcidas-7.6.4 ldm-mcidas-7.4.0 ldm-mcidas
cd ldm-mcidas
compress ldm-mcidas.tar
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> address@hidden
  cd pub/binary/sunos_5.6-sparc
  binary
  get ldm-mcidas-2004.tar.Z
  quit

zcat ldm-mcidas-2004.tar.Z | tar xvf -
ln -s ldm-mcidas-2004 runtime
ln -s runtime/* .
cd bin
cp area2png nldn2md pnga2area proftomd ~/decoders
cd ~/decoders
rm gunrv2 lwtmd2 lwtoa2 lwfile nids2area

This completed the upgrade of your ldm-mcidas decoders to a current
release.  It should stop the corruption of ROUTE.SYS _after_ a
new copy has been installed:

<as 'mcidas' on catwoman>
cd ~mcidas/workdata
pushd /mcidasdata/mcidas
mv ROUTE.SYS ROUTE.SYS.bad
cp ~mcidas/workdata/ROUTE.SYS .
pushd
route.k REL C
route.k REL N
route.k SUS R
route.k SUS UR US

Now, all images should be decoded as they come in.

NOTE: you will _not_ be seeing updates to the following AREA files:

AREA0060 - AREA0069    <- Educational Floater II
AREA0160 - AREA0169    <- Educational Floater I

The floater images were removed from the IDD UNIWISC (aka MCIDAS)
datastream a couple of months ago.

re:
>batplane: /users/mcidas/bin/dmap.k \*.R\*
>PERM      SIZE LAST CHANGED FILENAME     DIRECTORY
>---- --------- ------------ ------------ ---------
>-r--    142596 Jun 19  2004 FOUS14.RAP   /mcidasdata/mcidas
>-r--  33554260 Jun 19  2004 FOUS14.RAT   /mcidasdata/mcidas
>...

The FOUS14 stuff has changed in recent version of McIDAS.  You would
need to upgrade to the current version of McIDAS to fix this.

>It look like the files ARE being updated now. I just checked and I'm down
>to 116 to go from 137 this morning.

With your ROUTE.SYS file messed up, all bets were off re: decoding
of images in the IDD UNIWISC (aka MCIDAS) datastream.  The decoding
should proceed correctly now that I replaced the copy of ROUTE.SYS
in the output directory.  We will see as new images are ingested by
the LDM.

Back to the upper air MD files:  while logged onto catwoman as 'ldm', I
checked the 'request' lines in the ~ldm/etc/ldmd.conf file.  When
I saw that the request for NLDN lightning data was incorrect, I
decided to edit the 'request' to correct it.  It is now:

catwoman% grep ^request etc/ldmd.conf
request UNIDATA|FSL2    ".*" vortex.esc.brockport.edu
request NLDN    ".*" striker2.atmos.albany.edu

The change was stiker -> striker2.  This machine name change was done
quite some time ago.

The other thing I notice is that you are still running a _very_ old
version of the LDM:

<as 'ldm'>
cd ~ldm
ls -alt
lrwxrwxrwx   1 ldm      unidata       15 Dec  7  2004 data -> /mcidasdata/ldm
lrwxrwxrwx   1 ldm      unidata       25 Dec  7  2004 logs -> 
/export/app/data/ldm/logs
lrwxrwxrwx   1 ldm      unidata       11 Jul 30  2004 bin -> runtime/bin
lrwxrwxrwx   1 ldm      unidata       11 Jul 30  2004 doc -> runtime/doc
lrwxrwxrwx   1 ldm      unidata       15 Jul 30  2004 include -> runtime/include
lrwxrwxrwx   1 ldm      unidata       11 Jul 30  2004 lib -> runtime/lib
lrwxrwxrwx   1 ldm      unidata       11 Jul 30  2004 man -> runtime/man
lrwxrwxrwx   1 ldm      unidata        9 Jul 30  2004 runtime -> ldm-5.1.2

The last link shows that you are using LDM-5.1.2 even though I see
that you downloaded LDM-6.4.2:

-rw-rw-r--   1 ldm      unidata   968737 Oct 25 09:13 ldm-6.4.2.tar.Z

Looks like you never built the new distribution!  Given this, I decided
to go ahead and download and build the latest LDM release:

<still as 'ldm'>
cd ~ldm
ftp ftp.unidata.ucar.edu
  <user> anonymous
  <pass> address@hidden
  cd pub/ldm
  binary
  get ldm-6.4.3.tar.Z
  quit

zcat ldm-6.4.3.tar.Z | tar xvf -
cd ldm-6.4.3/src
./configure && make && make install

-- the next step needs to be done as 'root':

su
<Password: >
make install_setuids
exit

After running 'make install_setuids' as 'root' (in ~ldm/ldm-6.4.3/src)
you will need to do the following:

<as 'ldm'>
cd ~ldm
ldmadmin stop
rm runtime
ln -s ldm-6.4.3 runtime

cd etc
-- edit ldmadmin-pl.conf and set $hostname:

$hostname = "catwoman.cs.moravian.edu";

-- next edit ~ldm/etc/ldmd.conf:

change:

exec    "pqexpire"
exec    "xcd_run MONITOR"
exec    "pqbinstats -d /users/ldm/logs"
exec    "pqact -d /users/ldm /users/ldm/etc/pqact.conf"

to:

# exec    "pqexpire"
exec    "xcd_run MONITOR"
# exec    "pqbinstats -d /users/ldm/logs"
exec    "pqact -d /users/ldm /users/ldm/etc/pqact.conf"
exec    "rtstats -h rtstats.unidata.ucar.edu"

-- delete the LDM 5 queue; make a new one; and restart the LDM:

cd ~ldm
ldmadmin delqueue
ldmadmin mkqueue
ldmadmin start


If you see that you are having problems doing upper air contour
plots or plotting thermodynamic diagrams, do the following on
catwoman:

<as 'mcidas'>
cd ~mcidas/workdata
dataloc.k ADD RTPTSRC adde.ucar.edu

This will update the client routing table on catwoman
(~mcidas/data/ADDESITE.TXT).  To test to see if your users use
this configuration, do the following as one of them:

<as the end user>
cd ~/mcidas/data
dataloc.k LIST

If the listing shows that RTPTSRC is set for adde.ucar.edu, you
need do nothing more.  If it shows that RTPTSRC is still to be
grabbed from catwoman, have the user(s) do the following:

dataloc.k ADD RTPTSRC adde.ucar.edu

I think things are as setup now as they can be (until you upgrade
the LDM and McIDAS).

Please let me know if you see anything amiss.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.