[IDD #DLW-777823]: Problems with LDM NLDN configuration

Hi Brian,

I did some reorganization on weather.rsmas.miami.edu yesterday
afternoon and evening in an attempt to regularize the LDM installation:

- move the HOME directory of 'ldm' to /usr/local/ldm

- download, build and install the latest version of the LDM,
  v6.11.5

- changed the mount point of the machine's second disk from
  /ldm to /data

  This required that I modify /etc/fstab.

- played around with the system logging daemon and its configuration
  file to get LDM logging to work

  It was working after some twiddling of things like manually
  editing /etc/syslog.conf and stopping and restarting the
  system logging daemon (as 'root') using:

  service sysklogd stop
  service sysklogd start

- install and setup the network time protocol daemon (ntpd)

  I did this to fix the clock drift problem that we talked
  about on the phone yesterday.  The time was, indeed, off
  by over 5 minutes; it appears to be correct now.

- updated the GEMPAK decoders in ~ldm/decoders with the ones
  built in GEMPAK 6.6.0 (in /home/gempak)

  The GEMPAK decoders in ~ldm/decoders had timestamps from
  2009, so I figured that they needed updating no matter what

- downloaded, built and installed the latest ldm-mcidas decoders,
  v2012

- setup rotation of GEMPAK log files

  This is accomplished by running ~ldm/util/dcrotatelog.csh from
  'ldm's crontab instance

All of the above was basically routine maintenance that looked
to be needed.

I started the newly installed LDM using the configuration files
that were in place for the previous installation mainly to check
out the functioning of the system.

While the LDM was running (up until this morning), I started
investigating the contents of the data directories being
populated by the LDM and the scouring of those directories
by the LDM 'scour' utility (which is also run from 'ldm's
crontab file).  I found that there are over 32 million NEXRAD
Level III files in subdirectories of /data/ldm/gempak/nexrad/NIDS.
This is a LOT of files!  I wrote a script to try and organize
the NEXRAD Level III files creating daily directories at the
lowest level of the directory hierarchy.  I left the script
running overnight since it seemed to be running very slowly
(to say the least).  Since only a small fraction of the directories
to be processed had been processed by the time that I looked
in on things this morning, I became even more concerned at
how sluggish performance was on weather.  I finally decided
to stop my script (~ldm/util/refile.csh), stop the LDM
(as 'ldm' run 'ldmadmin stop'), and reboot the machine.
I was hoping that a system reboot would clear-up several
zombies that had been running since March 7 (all were from
a process named 'lightdm'.

The machine rebooted OK (albeit somewhat slowly), and I logged
back on from metofis.  I restarted my script with the hope
that it would now run at the speed that it should, but it still
seems very sluggish.  I don't know if this is an indication that
there is some problem with the disk on which data is being
decoded/written, or with the very large number of files that
the disk is currently holding.  In order to try to get the
script running as fast as it could, I stopped the LDM (which
had been automatically started on reboot).  Even with virtually
nothing running (tomcat7 is running; I think it provides the
RAMADDA service), the script is creeping along; most of the time
appears to be spent in the script's running of the 'ls' command!?

I will be getting together with our system administrator today
to discuss the findings on weather.  I have a suspicion that
there is something wrong with the disk where files are being
written, but I am not sure.

More later...





Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: DLW-777823
Department: Support IDD
Priority: Normal
Status: Closed