[McIDAS #QMD-212170]: XCD MySQL Errors and File System Cleanup

Hi Martha,

re:
> We are having an issue with the system where we are running the XCD
> software to decode the NOAAPORT data collected via LDM.

Since your problem seems to revolve around McIDAS-XCD, I moved this
inquiry into our McIDAS area.

re:
> The file system where we are writing NOAAPORT and CONDUIT data filled up,
> despite purge and cleanup jobs running regularly.  When I stopped  the
> ldm process, the file system cleaned itself up.

Your comment about things clearing up when you stopped the LDM strongly suggest
that one or more of your scouring ("purge and cleanup jobs") processes was
deleting a file(s) that were open for writing by a process being run by
the LDM.  In this case, the file appears to disappear (at least a 'ls' listing
doesn't show it), but it is not removed.  In fact, the process (or processes)
that was writing to the file(s) continues to write, and the file will grow
and grow.  When the LDM is stopped, the writing process(es) are stopped, and
the file is closed and then actually deleted.  So, if your "purge and cleanup
jobs" are deleting files that are actively being written to, you need to 
close the file before deleting it.  Another approach that one site implemented
a long time ago was to 'cat /dev/null > <file>'.

re:
> I looked a bit at the logs in the ldm area, and found that there are MySQL
> error messages, saying that "the table mcrtgrib.file_list does not exist"
> in the XCD log.  I had to move certain MySQL-related files from /var,
> where they had been residing, to another location due to the size of our
> /var file system.   I set up links that I thought would cause MySQL to
> continue to operate correctly, and the linked files in their new locations
> are shown to be open by the MySQL daemon, but I may have missed something
> or perhaps something got clobbered when our /var file system filled up.

I would guess that something got clobbered when your /var file system filled.

re:
> I don't know if one issue has to do with the other, and LDM is holding
> onto files that aren't being handled properly by MySQL, but I suppose
> it is a good idea for the MySQL is sure to be fixed in any event.

Again, it sounds like one or more of your "purge and delete" processes
is trying to delete a file that is actively open for writing.  The
behavior of *nix systems to apparently delete the file but keep writing
to it is the reason that the LDM sends its log messages to the syslog
daemon.  When it is time to "rotate" LDM log files, the files are renamed
and then a HUP signal is sent to (r)syslogd.  When (r)syslogd gets a
HUP signal, it closes all of its open file descriptors, rereads its
configuration file (/etc/(r)syslog.conf), and then reopens all of its
file descriptors.  The LDM file rotation action of 'ldmadmin' renames
the log files (e.g., ldmd.log.3 -> ldmd.log.4, ldmd.log.2 -> ldmd.log.1,
ldmd.log -> ldmd.log.1) and creates a new log file (ldmd.log).  A HUP
signal is then sent to (r)syslogd, and it will open the newly created
AND empty ldmd.log file for continued logging.

re:
> Any advise you can offer is much appreciated.

I feel confident that some "purge and delete" (we call this scouring,
by the way), process you are running is trying to delete a file that
is open for writing.  Find this/these processes and implement a new
strategy for scouring, and your problem should go away.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: QMD-212170
Department: Support McIDAS
Priority: Normal
Status: Closed