20040624: data storage on bigbird

>From:  Gerry Creager N5JXS <address@hidden>
>Organization:  Texas A&M University -- AATLT
>Keywords:  200406220437.i5M4beWb016431 LDM RAID storage IDD CRAFT

Hi Gerry,

>Sorry to have to run... the boss was calling, and that after telling me 
>earlier (he was going to a Dallas meeting, me to an Austin meeting) that 
>he'd call only in an emergency...

Absolutely no problem.  I have plenty to do :-)

>Again, thanks for finding the glitch and fixing it.  Real big help!

My bad, so I should have been the one to find and fix the problem.

>A question re: file storage... we've got a full terabyte available.  Do 
>you think we can archive 30 days' data safely?

With the current data volumes, and with filing the compressed data,
yes.  I was meaning to ask you about this.

>Management wants to look 
>at this as a test and I've got a nay-sayer who claims we can't get more 
>than 10-15 days' worth out there.

Given the disk usage at present and current data volume size in
CRAFT, then 30 days will be easy.  In fact, I believe that you could
do 30 days for both the Level II and III data.

>I can change the scour file if you think it'd be worth a look.

All that needs to be changed is (are if you decide to do the same
for the Level III data) the crontab entry(ies):

# Prune radar (NEXRAD/CRAFT) and images (GINI/NEXRCOMP/UNIWISC) trees
#
00  0-23/3 * * * util/scourBYnumber.tcl /data/ldm/gempak/images 256
15  0 * * * util/scourBYday.tcl /data/ldm/gempak/nexrad/NIDS_all  15
00  2 * * * util/scourBYday.tcl /data/ldm/gempak/nexrad/craft_all 15

The last item in each entry is the number to keep:

scourBYnumber.tcl -> number of files to keep
scourBYday.tcl    -> number of days of data to keep

As you can see, the number of days of both Level II and Level III
data are now set to be 15, and changing to 30 would be a piece
of cake.

scourBYday.tcl _assumes_ a directory structure where the subdirectories
under the specified node have names that fit the pattern CCYYMMDD.
For example:

[ldm@bigbird ~]$ ls /data/ldm/gempak/nexrad/craft_all
20040610  20040612  20040614  20040616  20040618  20040620  20040622  20040624
20040611  20040613  20040615  20040617  20040619  20040621  20040623

So, if you decided to keep multiple days of satellite imagery on disk
(currently stored in subdirectories of /data/ldm/gempak/images), you
would need to change the images' filing to follow what is being used
for the Level II and Level III data.

>Also: I've started feeding a couple of boxes from bigbird, but if you've 
>sites looking for primary or failover, let me know how/who I need to all 
>allows for and I'll make 'em available.

Thanks!

I notice that bigfoot.tamu.edu is feeding from emo.unidata.ucar.edu:

 -- on emo --

/local/ldm% netstat | grep ESTAB | grep ldm
 ...
tcp4       0   9216  emo.ldm                bigfoot.tamu.edu.49659 ESTABLISHED
 ...

Can you move bigfoot's emo feed to bigbird?

At some point in the not too distant future, I will be wanting to
change bigbird to feed CRAFT data off of Purdue (it is currently
feeding off of OU).  After that and after bigbird becomes phoenix -- if
that is still a possibility --, I would love to see it start feeding
some downstream sites CRAFT data.  The first site to feed will be the
University of Arizona.

>Again, thanks for all the help!

No worries.

Talk to you later...

Tom

>From address@hidden  Thu Jun 24 22:21:25 2004

Howdy!

re: My bad

>Still think I should have looked for it and taken a shot.

re: With the current data volumes, and with filing the compressed data,
yes.

>I'll accomplish this in either a few minutes... or the morning.  OK. 
>Done.  I've modified the crontab scourByDay values for NIDS and CRAFT to 
>30.

>LOADTEST!

re: Can you move bigfoot's emo feed to bigbird?

>I'll accomplish this in a minute.  I thought I'd done that, but I may 
>have fumble-fingered it when I changed over from LSU's feed recently.

Done; restarting LDM now.

re: name change to phoenix

>Perhaps Fawkes... phoenix.tamu.edu is taken.  Let me see if I can con 
>the boss out of a subdomain mesonet.tamu.edu.  If I can, then we can do 
>phoenix.mesonet.tamu.edu; I'll ask in the morning.  Otherwise, I'm ready 
>to declare it operational and go on.  We will suffer one 2 hr outage 
>when we move to a new lab facility in mid-July; there WILL be warning.

>Later!
>gerry
>-- 
>Gerry Creager -- address@hidden
>Texas Mesonet -- AATLT, Texas A&M University   
>Cell: 979.229.5301 Office: 979.458.4020 FAX: 979.847.8578
>Page: 979.228.0173
>Office: 903A Eller Bldg, TAMU, College Station, TX 77843