20040221: Help! LDM on weather3.admin.niu.edu going crazy...

>From: Gilbert Sebenste <address@hidden>
>Organization: NIU
>Keywords: 200402212235.i1LMZlrV009065 LDM scouring

Gilbert,

>OK, I am stumped. I cannot figure out what is happening.
>
>My LDM on weather3 is filling up my hard drive. Over the last two days, 
>files in my /home/data/domestic/state_zone_forecasts and 
>/home/data/domestic/hourly_weather_roundup directories have been huge. 

I logged onto weather3 and did not see a directory named
/home/data/domestic/hourly_weather_roundup, nor did I see any actions
in ~ldm/etc/pqact.conf that explicitly put data products into
this directory.

When I got on, the products in /home/data/domestic/state_zone_forecasts
were all small.

>I store zone forecasts and HWR's in each directory, respectively. I have 
>essentially the same pqact.conf on weather and weather2, and they are the 
>same for the entries on all 3 machines.
>
>If there's any chance one of you could look at this, that would be great.
>I know:
>
>The problem is only occurring on weather3, which is the ingest/relay 
>machine for others and to my in-house weather and weather2 machines
>
>I am not getting any duplicate feeds
>
>I have not recompiled the LDM in the last month
>
>I have the latest Fedora version on there (1), fully patched
>
>Any ideas?

We will need to catch the disk filling while it is progress.  At
the moment, everything looks normal, and your disk is no where
near full:

weather3-niu ldm-18> df -k
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/hda2            238172636  70186320 155887768  32% /
/dev/hda1               202220     12696    179084   7% /boot
none                    774692         0    774692   0% /dev/shm

I recommend that the next time you find the disk full in the
same way, capture the output from 'top' for LDM processes to
see if some process is in a tight loop.  Also, try to get a
long listing of each of the directories which are consuming your
disk space so we can see which files are becoming huge.

Again, it is virtually impossible to see what may be wrong when
everything is working smoothly.

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.