[AWIPS #NTZ-805469]: Data Disk Completely full!

Hi Bob,

I'm curious if this is because of LDM scour or EDEX scour not doing its job.  
Is most of your disk space taken by /data_store/ or by /awips2/edex/data/hdf5/, 
and if the latter, whic

Any old grids handing around? 

find /awips2/edex/data/hdf5/grid/  -type f -printf '%T+ %p\n' | sort | head -10

or satellite files?

find /awips2/edex/data/hdf5/satellite/  -type f -printf '%T+ %p\n' | sort | 
head -10


If you see very old files in these directories, it may be that the EDEX system 
has flipped the "failedcount" value in the purgejobs database.



Attached is the latest "edex" management script which can check for and fix 
this purging problem.  Copy to /awips2/tools/bin and chmod 755 and run 

edex purge

and if you see plugins reported, you can reset them with 

edex purge reset


You can also do this manually with the commands:

su awips -c 'psql metadata -c "select plugin from purgejobs where failedcount = 
3;"'

and if this returns plugin names, reset with 

su awips -c 'psql metadata -c "select plugin from purgejobs where failedcount = 
3;"'







> I was curious if there was any progress to report on this
> issue?
> 
> There were a list of recent updates which would take EDEX from
> 14.4.1-6n3 to 14.4.1-7n8 . Is that a fix for the problem?
> 
> I haven't installed them as there's only 13Mb of free space on
> my /awips2 disk and I fear there's insufficient space to install.
> If 14.4.1-7n8 fix will resolve the problem, can you suggest
> the files I should delete to create the space for the updates?
> 
> Thanks...
> 
> -- Bob
> 
> On Fri, Oct 9, 2015 at 3:33 PM, Unidata AWIPS Support
> <address@hidden> wrote:
> > Hi Bob,
> >
> > I have experienced something similar on a cloud instance, and discovered 
> > this week that when the disk was full, each EDEX plugin's purge status was 
> > flagged in postgres which *prevented* scouring, which is exactly what we do 
> > not want to happen.  I'm working on a fix and have added a reset option to 
> > the edex sservice manager (something like "edex purge reset" run 
> > automatically every so often).
> >
> > I would check the size of /awips2/edex/data/hdf5/ and manually remove some 
> > of the larger directories and files which are old.
> >
> > You can run this to check for the oldest files in hdf5/
> >
> > find /awips2/edex/data/hdf5/ -type f -printf '%T+ %p\n' | sort | head -10
> >
> > See anything extremely old?
> >
> >
> >
> >
> >
> >
> >> Hi Michael,
> >>
> >> As you know, I recently installed the AWIPS2 software, it
> >> seemed to be running just fine, but last night the data disk
> >> filled up:
> >>
> >> # df -h /awips2/
> >>
> >> Filesystem            Size  Used Avail Use% Mounted on
> >>
> >> /dev/mapper/vg_flurry-LogVol06
> >>
> >> 566G  566G   36K 100% /awips2
> >>
> >>
> >> There was actually 20K free but then /etc/cron.daily/pypiesLogCleanup.sh
> >>
> >> ran, which freed up an additional 16K.
> >>
> >>
> >> As you know this time I installed it on a ZFS file system. I'm not seeing
> >> the
> >>
> >> error messages about the directory index being full, but nearly all
> >> the inodes
> >>
> >> are consumed:
> >>
> >>
> >> # df -hi /awips2/
> >>
> >> Filesystem           Inodes IUsed IFree IUse% Mounted on
> >>
> >> /dev/mapper/vg_flurry-LogVol06
> >>
> >> 12M   12M   188  100% /awips2
> >>
> >>
> >> I can't find it now, but I seem to recall the /awips2 disk was recommend to
> >>
> >> be 500Gb -- as you can see ours is 566Gb, am I remembering incorrectly?
> >>
> >>
> >> Is there some data cleanup/garbage collection which is not running
> >>
> >> which will keep the data at a manageable level?
> >>
> >>
> >> I notice there are new updates for several components of the AWIPS2
> >>
> >> repository, but I'm sure I don't have space on the disk to install.
> >>
> >>
> >> What can I do to clean up the disk? Any other suggestions?
> >>
> >>
> >> Thanks.
> >>
> >>
> >> -- Bob
> >>
> >>
> >
> > Ticket Details
> > ===================
> > Ticket ID: NTZ-805469
> > Department: Support AWIPS
> > Priority: Normal
> > Status: Open
> >
> 
> 

Ticket Details
===================
Ticket ID: NTZ-805469
Department: Support AWIPS
Priority: High
Status: Open