[LDM #FAJ-199818]: LDM installation on yobui.cicese.mx

Hi Luis,

I have created a Unidata User Support inquiry to record all of the work we
we did on yogui yesterday (October 31) at the end of the LDM training workshop.
I am hopeful that the information in this inquiry will be helpful to other
Unidata users who are faced with a problem similar to what you were seeing.

Recap:

- your system is running RedHat Enterprise 5, and it is a 64-bit system:

  <as 'ldm'>
  uname -a
   Linux yogui.cicese.mx 2.6.18-8.el5 #1 SMP Fri Jan 26 14:15:14 EST 2007 
x86_64 x86_64 x86_64 GNU/Linux

  cat /etc/redhat-release
  Red Hat Enterprise Linux Client release 5 (Tikanga)

- the version of 'gcc' in use on your system is 4.1.1:

  [ldm@yogui ~]$ gcc --version
  gcc (GCC) 4.1.1 20070105 (Red Hat 4.1.1-52)
  Copyright (C) 2006 Free Software Foundation, Inc.
  This is free software; see the source for copying conditions.  There is NO
  warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

  'c89' was used to build the LDM.

- both LDM-6.6.5 and 6.7.0 exhibited the strange behavior of:

  - 'ldmadmin watch' would show the arrival of HDS products into the
    LDM queue while 'notifyme -vl-' would not (!?)

  - even though 'ldmadmin watch' would show that there should be products
    in the LDM queue, 'pqcat -vl- -f HDS -o 3600 > /dev/null' would only
    show IDS|DDPLUS products in the queue.

    The lack of HDS products in the 'pqcat' output matched with our observation
    that there were no invocations of the GEMPAK 'dcgrib2' decoder running,
    and all of the 'dcgrib2' decoder log files in ~ldm/data/gempak/logs were
    empty

- we tried changing the LDM queue size from 600M to 2G to see if this would
  affect anything.  The result of this was no change.

- we tried creating the LDM queue on the same (root) filesystem where the LDM
  was installed.  The result of this was no change.

- when originally trying to run 'notifyme -vl-', we would experience a _long_ 
delay
  before it would start listing out any products.  This appeared to be a 
DNS-related
  issue.

  After verifying that the /etc/nsswitch.conf settings for DNS was 'files dns', 
this
  problem appeared to get fixed when I added an entry in /etc/hosts defining 
yogui's
  IP address:

  158.97.97.15    yogui.cicese.mx yogui

  What we didn't notice at the time was that the definition for localhost also
  included a definition for yogui:

  127.0.0.1       localhost.localdomain   localhost   yogui

  We removed this definition late in the troubleshooting effort

- in desperation, I decided to turn off use of IPv6 on yogui.  I did this by 
following
  the procedure listed in:

  How To Disable IPv6 on Fedora / Linux & Why
  
http://blog.taragana.com/index.php/archive/how-to-disable-ipv6-on-fedora-linux-why/

  This page was found by a Google search using the search keys 'disable ipv6 
+fedora'

  In order to have the configuration for eth0 (yogui's Gigabit Ethernet 
interface)
  clear of any/all IPv6 settings, we had to reboot yogui

Results:

- after turning off IPv6 support and removing 'yogui' from the /etc/hosts line 
for
  localhost and commenting out the IPv6 setting in /etc/hosts, the LDM was 
turned back
  on and all requested data products began to be received AND processed by the
  GEMPAK 5.11.1 decoders (installed from a Unidata binary distribution)

Other things that were done that are significant, but not directly related to 
the
strange problem of 'ldmadmin watch' showing the receipt of HDS products, but
'pqcat -vl- -o 3600 > /dev/null' not showing any HDS products in the LDM queue 
were:

- in order to get LDM logging to work, we needed to edit (as 'root') 
/etc/selinux/config
  and set SELINUX to 'disabled':

  SELINUX=disabled

  To institute this change, we had to reboot yogui.

- the /home/gempak directory was not readable by 'ldm'; this was changed

- two processing entries in ~ldm/etc/pqact.gempak_decoders had been modified
  to use a different directory structure for decoded file output; this was
  changed back to the default pqact.conf entries created by Chiz's script that
  creates LDM pattern-action files and ldmd.conf entries which is presented in:

  Site Configuration for Products
  http://www.unidata.ucar.edu/software/gempak/GEMPAK/configuration.html

- we cut down the NIMAGE request from all images to only those from GOES-11.  We
  agreed at the time that if limited bandwidth remains as a problem, we would
  go back and refine the NIMAGE request in ~ldm/etc/ldmd.conf to include just 
the
  GOES-West images that you want/use.

  We did _not_ change the ldmd.conf request line for HDS data as you indicated 
that
  you would like to get the RUC, NAM, etc. models even though their main focus
  was to the north of the CICESE installation in La Paz, Baja California.

- we copied an LDM start at boot script from one of the Unidata 
idd.unidata.ucar.edu
  cluster nodes to the file /etc/init.d/ldmd.  The file was configured to be run
  at bootup:

  - the mode of the newly created file on yogui was changed to be executable
    (chmod +x /etc/init.d/ldmd) so that it could be run at boot

  - 'chkconfig' was used to setup the OS to run 'ldmd' at run levels 3,4, and 5:

    <as 'root'>
    chkconfig --level 345 ldmd on

- the final configuration of the LDM on yogui is:

  ~ldm/data -> /mnt/disk2/data/ldm
  ~ldm/logs -> /mnt/disk2/data/ldm/logs

  ldmadmin config:
  
  hostname:      yogui.cicese.mx
  os:            Linux
  release:       2.6.18-8.el5
  ldmhome:       /usr/local/ldm
  bin path:      /usr/local/ldm/bin
  conf file:     /usr/local/ldm/etc/ldmd.conf
  log file:      /usr/local/ldm/logs/ldmd.log
  numlogs:       7
  log_rotate:    1
  data path:     /usr/local/ldm/data
  product queue: /usr/local/ldm/data/ldm.pq
  queue size:    1G bytes
  queue slots:   default
  IP address:    all
  port:          388
  PID file:      /usr/local/ldm/ldmd.pid
  LDMHOSTNAME:   yogui.cicese.mx
  PATH:          
/usr/local/ldm/bin:/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/usr/X11R6/bin:/usr/local
 /ldm/bin:/usr/local/ldm/decoders:/usr/local/ldm/util

- the LDM currently in use was built from the Makefiles created by LDM-6.7.0's
  './configure --disable-max-flags' run in the ~ldm/ldm-6.7.0/src directory

Final comments:

- As I write this summary of what we did some 13 hours after the effort, the 
LDM seems
  to be running nicely on yogui:

  - model data is being decoded
  - imagery is being processed
  - most text data is being processed

  The exception to the last item is the fact that ~ldm/logs/ldmd.log is 
routinely
  reporting segmentation violations for 'dcmsfc':

Nov 01 08:53:15 yogui pqact[5182] NOTE: child 7047 terminated by signal 11 
(decoders/dcmsfc-b9-ddata/gempak/logs/dcmsfc.log-eGEMTBL=/home/gempak/NAWIPS/gempak/tablesdata/gempak/ship/YYYYMMDDHH_sb.gem)
 

  We tried using a GEMPAK 5.10.4 version of 'dcmsfc' to see if the segmentation 
violations
  would stop, but this did not have any effect (aside: no core files are being 
produced because
  coredumpsize is set to 0 (zero) using 'limit').  This morning I reverted back 
to using the
  5.11.1 version of 'dcmsfc'.

- 'rtstats' latency plots for the various feeds that yogui is receiving show 
that
  there is limited bandwith into CICESE's installation in La Paz:

  HDS
  http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?HDS+yogui.cicese.mx
  
  IDS|DDPLUS
  
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+yogui.cicese.mx

  NIMAGE
  http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NIMAGE+yogui.cicese.mx

  UNIWISC
  
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?UNIWISC+yogui.cicese.mx


  While the latencies are not what we would like, the data products are making 
it
  to yogui in a short enough time to be usable.  Hopefully, the routing for LDM
  traffic to yogui can be altered to use Internet2 (as you mentioned).

That's all for now...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************