19990708: help with mcidas-xcd (cont.)

>From: Zuo Dong Zheng <address@hidden>
>Organization: CCNY
>Keywords: 199907011430.IAA14251 McIDAS make IRIX compilers

Zuodong,

re: help sort out LDM, McIDAS, and McIDAS-XCD on halo
>sure.  We be very delighted if you can help us.

OK.

re: logins
>We have ...

Super, thanks.  I am on halo right now cleaning up the LDM account.
Here is what I have done:

o correct the XCD entry in ~ldm/etc/pqact.conf.  The problem was that
  the entry had spaces where tabs are mandatory.  This error will
  cause the LDM program pqact to fail (and hence, you would not get
  any XCD things to work)

o I noticed that you had not set umask in the ldm's .cshrc file; I
  set it to

  umask 002

  I did the same in mcidas' .chsrc file

o I commented out the McIDAS environment variable definitions in
  ~ldm/.login.  The ldm account should know nothing about McIDAS.
  All knowledge of how to run McIDAS stuff is kept in shell script
  files (like xcd_run, batch.k, and mcscour.sh).  This way one
  doesn't have to stop/restart the LDM if a change is needed to
  McIDAS stuff.

o cd ~ldm/logs

  I see hundreds of 'stats' files there dating back to June of 1998.
  I deleted all of the stats files to start anew.  Later, I will try
  to get the statistics reporting back to Unidata working.  This will
  not only keep us up to date about how your setup is working, but it
  will also scour out the stats files so ~ldm/logs will not get overrun
  with files.  I setup the stats stuff in ldm's crontab file along
  with rotation of the ldmd.log and ldm-mcidas.log files:

# Scour McIDAS data files
#
00 21 * * * /usr/local/ldm/decoders/mcscour.sh
#
# Shift LDM-McIDAS log files
#
0 17 * * * bin/newlog logs/ldm-mcidas.log
#
# Shift LDM log files, 1700 MST, 1800 MDT is 0 zulu.
#
0 17 * * * bin/ldmadmin newlog
#
# LDM statistics
#
35 * * * * bin/ldmadmin dostats 

o I checked ~ldm/decoders/xcd_run, ~ldm/decoders/batch.k, and
  ~ldm/decoders/mcscour.sh  to make sure that the environment variables
  needed for McIDAS processing were set correctly.  

  In ~ldm/decoders/batch.k, I removed the /opt/apps/SUNWspro/SC3.0/lib
  path from LD_LIBRARY_PATH since it doesn't exist;
  /opt/apps/SUNWspro/lib was correct, so I left it.

  In ~ldm/decoders/xcd_run, I had to add /opt/apps/SUNWspro/lib to
  LD_LIBRARY_PATH so that the McIDAS applications could run correctly.
  The reason that your XCD applications were not starting was that
  they couldn't find the shared Fortran library.

  I changed /opt/SUNWspro/SC3.0.1/lib in ~ldm/decoders/mcscour.sh to
  /opt/apps/SUNWspro/lib.

o I started the LDM and waited for products to start coming in.  I
  saw McIDAS images but no textual data, so I checked your ~ldm/etc/ldmd.conf
  file to make sure that you were requesting appropriate data from
  your upstream feed site:

request ANY
        ".*"
        snow.cit.cornell.edu.

  You are.  I then checked the feeder to see if they what products they
  have gotten in the last hour:

notifyme -vxl- -h snow.cit.cornell.edu -o 3600
Jul 08 21:36:51 notifyme[13128]: Starting Up: snow.cit.cornell.edu: 
19990708203651.288 TS_ENDT {{ANY,  ".*"}}
        NOTIFYME(snow.cit.cornell.edu) returns RECLASS
Jul 08 21:36:51 notifyme[13128]: NOTIFYME(snow.cit.cornell.edu): reclass: 
19990708203651.288 TS_ENDT {{MCIDAS,  ".*"}}
        NOTIFYME(snow.cit.cornell.edu) returns OK
Jul 08 21:36:51 notifyme[13128]: NOTIFYME(snow.cit.cornell.edu): OK
Jul 08 21:36:53 notifyme[13128]: cc943bb0234bd3e004e04e4f3b1d8e34    46041 
19990708210526.067  MCIDAS 000  LWTOA3 202 DIALPROD=U3 99189 210525
Jul 08 21:36:53 notifyme[13128]: 0cc4f4a2598788a34cc86f361ceae27b   194333 
19990708211025.646  MCIDAS 000  LWTOA3 192 DIALPROD=U1 99189 211023
Jul 08 21:36:53 notifyme[13128]: 09262b74103fed8fdf31e0cb040daedb   223834 
19990708211619.272  MCIDAS 000  LWTOA3 173 DIALPROD=UB 99189 211615
...

  I originally thought that the fact that this list only has MCIDAS products
  was telling us that Cornell is not getting the other products from its
  upstream feed site, but upon closer inspection it is telling us that
  it was only allowing you to request MCIDAS data (the RECLASS message
  tells us that your ANY request was being turned into a MCIDAS request).
  I contacted Bill Noon of Cornell, and he changed his ldmd.conf file to
  allow you to request all data.

To get the XCD stuff running, I did the following as the user mcidas:

o cd ~mcidas/data
  cp EXAMPLE.NAM LOCAL.NAM
  <edit LOCAL.NAM>

o FTPed the McIDAS 7.5 mcinet shell script (mcinet7.5.sh) to ~mcidas

o created ~mcidas/.mcenv file for mcadde use as per my web page
  instructions

o cd ~mcidas/data
  cp DSSERVE.BAT LSSERVE.BAT      (made no changes)
  cp DATALOC.BAT LOCDATA.BAT
  <edit LOCDATA.BAT and replace fully_qualified_host_identifier with
   halo.sci.ccny.cuny.edu>

o cd ~mcidas/workdata
  tl.k OUT
  td.k XCDDATA           (was /MCIDAS/WORKDATA/XCD)
  td.k XCDDATE           (was /MCIDAS/WORDDATA)
  te.k XCDDATA \"/scratch/mcidasd
  batch.k LSSERVE.BAT
  batch.k XCD.BAT
  batch.k XCDDEC.BAT

o several times I had problems doing a 'pwd':

halo{ldm}% cd /scratch
halo{ldm}% pwd
pwd: cannot determine current directory!

  You get the same result when doing:

halo{ldm}% cd ~ldm/logs
halo{ldm}% pwd
pwd: cannot determine current directory!

  This is not suprising given that ~ldm/logs is /scratch/logs.

To get the ADDE remote server working, I did the following as root:

o create user mcadde by adding an entry in /etc/passwd.  You will note
  that for this entry, the shell is specified as /bin/false so that
  the 'mcadde' account is not a login account.

o ran the mcinet7.5.sh script as:

  cd ~mcidas
  sh mcinet7.5.sh install mcadde

  This sets entries in /etc/services and /etc/inetd.conf and then sends
  a HUP to inted.  The ADDE remote server is now working; I tested
  it by pointing at your machine and loading an image off of it.

o I noticed that your LDM was not logging to the ~ldm/logs/ldmd.log file
  as it should be.  I got LDM logging working by killing and restarting
  syslogd.

o In looking through /var/adm/messages, I saw indications that there
  might have been disk and network problems a few days ago:

Jul  4 12:39:14 halo unix:      SCSI transport failed: reason 'reset': retrying 
command
Jul  4 20:24:44 halo su: 'su root' failed for mcidas on /dev/pts/3
Jul  6 10:22:17 halo unix: le0: No carrier - cable disconnected or hub link test
 disabled?

I am now saving all McIDAS and McIDAS-XCD data to /scratch/mcidasd.  I note,
however, that the /scratch file system is very short of space:

halo{mcidas}% df -k
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t3d0s0     143927   14359  115178    12%    /
/dev/dsk/c0t3d0s6     551518  302237  194131    61%    /usr
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
/dev/dsk/c0t3d0s3     143927   45793   83744    36%    /var
/dev/dsk/c0t0d0s0     480815  418282   14453    97%    /export/home
/dev/dsk/c0t0d0s1     481823  103932  329711    24%    /opt
/dev/dsk/c0t1d0s0     963662  386894  480408    45%    /terascan
/dev/dsk/c0t5d0s0     962226  649531  216475    76%    /opt/apps
/dev/dsk/c0t5d0s5     402709      50  362389     1%    /var/mail
/dev/dsk/c0t5d0s1     962226  483992  382014    56%    /mcidas
/dev/dsk/c0t5d0s4     673182   40553  565319     7%    /usr/local/ldm
/dev/dsk/c0t5d0s3     962226  365928  500078    43%    /scratch
swap                  112832     284  112548     1%    /tmp
/export/home/students/wxnut
                      480815  418282   14453    97%    /home/wxnut

The 500 MB that is there might seem like a lot, but XCD decodes a lot
of data.  If it is at all possible, you should find somewhere with a
couple of GB of data so that all of the data now coming into CCNY
can be decoded and saved.  I currently have GRID decoding disabled in
XCD since you simply do not have enough disk to process any of it.

As a final thought, I see that your LDM product queue is probably too
small.  If you start seeing messages in the ~ldm/logs/ldmd.log file
about deleting to make space, then the queue size will need to be
increased.  Instead of doing so right now, and given that the amount
of disk space that you have in /scratch is marginal, I think that
we should wait.

We just checked and see that you seem to have a 1.6 GB disk in the system
that is not being used:

AVAILABLE DISK SELECTIONS:
       0. c0t0d0 <SUN1.05 cyl 2036 alt 2 hd 14 sec 72>
          /iommu@f,e0000000/sbus@f,e0001000/espdma@f,400000/esp@f,800000/sd@0,0
       1. c0t1d0 <SUN1.05 cyl 2036 alt 2 hd 14 sec 72>
          /iommu@f,e0000000/sbus@f,e0001000/espdma@f,400000/esp@f,800000/sd@1,0
       2. c0t2d0 <MICROP-2217-15MQ1001901-HQ30 cyl 2052 alt 2 hd 15 sec 112>
          /iommu@f,e0000000/sbus@f,e0001000/espdma@f,400000/esp@f,800000/sd@2,0
       3. c0t3d0 <SUN1.05 cyl 2036 alt 2 hd 14 sec 72>
          /iommu@f,e0000000/sbus@f,e0001000/espdma@f,400000/esp@f,800000/sd@3,0
       4. c0t5d0 <QUANTUM-FIREBALLST4.3S-0F0C cyl 7066 alt 2 hd 6 sec 199>
          /iommu@f,e0000000/sbus@f,e0001000/espdma@f,400000/esp@f,800000/sd@5,0

#2: c0t2d0 does not show up in the list of filesystems from a df -k:

Filesystem            kbytes    used   avail capacity  Mounted on
/dev/dsk/c0t3d0s0     143927   14359  115178    12%    /
/dev/dsk/c0t3d0s6     551518  302237  194131    61%    /usr
/proc                      0       0       0     0%    /proc
fd                         0       0       0     0%    /dev/fd
/dev/dsk/c0t3d0s3     143927   45793   83744    36%    /var
/dev/dsk/c0t0d0s0     480815  418282   14453    97%    /export/home
/dev/dsk/c0t0d0s1     481823  103932  329711    24%    /opt
/dev/dsk/c0t1d0s0     963662  386894  480408    45%    /terascan
/dev/dsk/c0t5d0s0     962226  649531  216475    76%    /opt/apps
/dev/dsk/c0t5d0s5     402709      50  362389     1%    /var/mail
/dev/dsk/c0t5d0s1     962226  483992  382014    56%    /mcidas
/dev/dsk/c0t5d0s4     673182   40553  565319     7%    /usr/local/ldm
/dev/dsk/c0t5d0s3     962226  372315  493691    43%    /scratch
swap                  113544     284  113260     1%    /tmp
/export/home/students/wxnut
                      480815  418282   14453    97%    /home/wxnut

I strongly recommend that if this disk is usable that you put it into
service and use it as the McIDAS/XCD data disk.  If 1.6 GB is not enough
space, then you can use it in combination with /scratch to get 2.1 GB
which will be enough for what I think Ward, et. al. want to do.

Please let me know if anything above does not make sense, or if you have
any other questions.

As I log off, I see that XCD is running nicely and logs of data is flowing
into the system.  Someone needs to keep on top of this given the lack
of space in /scratch.

Tom

>From address@hidden  Thu Jul  8 20:05:25 1999

i don't how i can ever thank you for that.  You did too much for us.  All i can 
say is that city college owe you one.

zuodong

>From address@hidden  Fri Jul  9 08:19:12 1999

Tom:

Thank you for your terrific work yesterday that got our McIDAS-XCD 
installation completed.  I now have the wonderful Global Mollweide to 
show the teachers Monday.

I will work systematically with Zuo Dong to implement the suggestions you 
made in your "tutorial" message of yesterday.

I'm looking forward to further education at your McIDAS workshop next month.

Sincerely,

Prof. Ward Hindman