20040626: scouring on bigbird

>From: Unidata User Support <address@hidden>
>Organization:  Texas A&M University -- AATLT
>Keywords:  200406220437.i5M4beWb016431 LDM RAID storage IDD CRAFT

Hi Gerry,

I was checking on bigbird this morning and I noticed that the
disk space usage was much less than I expected.  When I looked further,
I found that all but 3 of the subdirectories under 
/data/ldm/gempak/nexrad/craft_all and /data/ldm/gempak/nexrad/NIDS_all
were empty.  I checked the crontab-initiated scouring for thse
directories and they are correctly setup to keep 30 days of data:

# Prune radar (NEXRAD/CRAFT) and images (GINI/NEXRCOMP/UNIWISC) trees
#
00  0-23/3 * * * util/scourBYnumber.tcl /data/ldm/gempak/images 256
15  0 * * * util/scourBYday.tcl /data/ldm/gempak/nexrad/NIDS_all  30
00  2 * * * util/scourBYday.tcl /data/ldm/gempak/nexrad/craft_all 30

Since scourBYday.tcl and scourBYnumber.tcl have been working with no
problems on bigbird for quite some time, I was mystified by what could
have been causing the many days of NEXRAD data to disappear.  Working
on a guess, I did a long list of /data/ldm/gempak/nexrad and saw a
.scour* file there.  This told me that 'scour' ('ldmadmin scour')
scouring had been turned on for this directory tree, something I had
purposefully turned off right after loading the LDM.  I am guessing
that you uncommented the line in ~ldm/etc/scour.conf:

#~ldm/data/gempak/nexrad        3

when you first started thinking about keeping 30 days of NEXRAD data.
The result of uncommenting the line was the loss of almost two weeks of
both Level II and Level III data.

To get things back to the test of how much data can be kept online, I
commented out the entry in ~ldm/etc/scour.conf.  Since 'scour' doesn't
remove directories, I manually removed all of the subdirectories of
/data/ldm/gempak/nexrad that had no data files in them (rm -rf ...).

So, in 27 days we should see how well bigbird is doing on disk
space.  With today's radar volumes, it should be OK:

Level II volume/day ~= 25 GB
30 days             ~= 750 GB
Level III volume/day is lot less than the Level II, so you should
end up using about 1.1 TB.

Could uncommenting of the scour.conf line have been the reason
that the KAMA data was disappearing on you?  I don't see how,
but it is an interesting idea.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.