20040612: setup of bigbird and Level II data (cont.)

>From: Unidata Support <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200405311647.i4VGlNtK006820 LDM IDD THREDDS

Hi Gerry,

I see that you rebooted bigbird back into the smp kernel yesterday
and the system now "sees" 4-2.4 Ghz Xeon processors (two processors
with hyperthreading).  The load averages looked much lower after
the reboot, so my curiosity about perforance using the SMP kernel
has been satisfied :-)

In looking through the performance monitoring log file:

~ldm/logs/bigbird.uptime

I noted that the age of the oldest product in bigbird's LDM queue
was consistently less than 1 hour:

 ...
20040612.1546   2.32  2.36  1.81   12   0  12   2806  47M  11M  1 
scourBY(number|day)
20040612.1547   2.68  2.41  1.86   12   0  12   2776  45M  11M  1 
scourBY(number|day)
20040612.1548   2.64  2.49  1.92   12   0  12   2791  44M  11M  1 
scourBY(number|day)
20040612.1549   2.68  2.47  1.94   12   0  12   2773  44M  11M  1 
scourBY(number|day)
20040612.1550   1.86  2.29  1.91   12   0  12   2777  47M  11M  0 
scourBY(number|day)
20040612.1551   1.10  2.00  1.84   12   0  12   2780  48M  11M  0 
scourBY(number|day)
20040612.1552   1.00  1.80  1.78   12   0  12   2758  49M  11M  0 
scourBY(number|day)
20040612.1553   1.06  1.67  1.73   12   0  12   2741  49M  11M  0 
scourBY(number|day)
20040612.1554   1.35  1.65  1.72   12   0  12   2777  48M  11M  1 
scourBY(number|day)
 ...

Because of this, I decided to rebuild the LDM 6.0.14 distribution with
large file support:

- added a define of CPPFLAGS in ~ldm/.cshrc:

# Environment variable(s) used in LDM build
#
# 20040612 - support for queue > 2 GB
setenv CPPFLAGS "-D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE"

source .cshrc

- rebuilt the distribution

cd ldm-6.0.14/src
make distclean
./configure && make
ldmadmin stop
pushd ../bin
rm -f *
popd
make install
sudo make install_setuids
cd ~ldm
ldmadmin delqueue

- edit ~ldm/bin/ldmadmin and set the queue size to 4 GB

I had been under the impression that all I needed to do next was to
edit ~ldm/bin/ldmadmin and set the $pq_size parameter to 4GB:

$pq_size = 4000000000;

and then run 'ldmadmin mkqueue -f'.  This did NOT result in a 4 GB
queue, however.  The queue size remained at 2 GB.

Through experimentation, I found that I could make a 4 GB queue
manually using pqcreate:

pqcreate -q $HOME/data/ldm.pq -s 4000M

or change the format of $pq_size in ldmadmin:

$pq_size = "4000M";

Along the way, I found that the following did not work in ldmadin:

$pq_size = 4000M;

We will be addressing the need to tell users to quote the size value
when it is non numeric in an upcoming release.

I just wanted to let you know that bigbird is now running with a 4 GB
queue and the age of its oldest products has climbed past 1 hour (and
is still climbing):

 ...                                             +-- age of oldest queue product
                                                 |
                                                 v

20040612.1737   0.69  0.82  1.19   12   0  12   4174  49M   1M  0 
scourBY(number|day)
20040612.1738   0.63  0.80  1.16   12   0  12   4229  48M   1M  0 
scourBY(number|day)
20040612.1739   0.57  0.75  1.11   12   0  12   4281  48M   1M  0 
scourBY(number|day)
20040612.1740   0.77  0.77  1.09   12   0  12   4330  49M   1M  0 
scourBY(number|day)
20040612.1741   0.65  0.73  1.06   12   0  12   4380  49M   1M  0 
scourBY(number|day)

 ...

The other comment I would like to make for the tracking system is that
the process of scouring full day's of CRAFT data is now reasonably
quick on bigbird.  Cleaning up the /etc/raidtab setup and remaking the RAID
really helped improve performance!

I think we can now officially declare bigbird as a "smoking unit" :-)

Is it now time for a name change to phoenix?

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.

>From address@hidden  Sun Jun 13 12:19:58 2004

re: so my curiosity about perforance using the SMP kernel has been satisfied :-)

>And mine!  Although it was really performing well on uniproc after 
>fixing RAID!

re: rebuild of LDM queue > 2 GB

>An earlier release of LDM *HAD* allowed large queue support.  It also 
>had allowed queue redirection, which died in 6.0.14; we talked briefly 
>about this but I may not have communicated it clearly...

re: set $pq_size = "4000M";

>Yep!

re: why '$pq_size = 4000M;' doesn't work

>I suspect it's a perl thing and I didn't think it throuh, on retrospect!

re: now running with a 4 GB queue that has more than 1 hr of data

>Cool.

re: Is it now time for a name change to phoenix?

>I'll set the process in motion, but I've got to find a subnet to place 
>it officially in.  'phoenix.tamu.edu' has now been grabbed...  I'll see 
>if the owner will consider a name change!

>Later!
>Gerry
>-- 
>Gerry Creager -- address@hidden
>Texas Mesonet -- AATLT, Texas A&M University   
>Cell: 979.229.5301 Office: 979.458.4020
>FAX:  979.847.8578 Pager:  979.228.0173
>Office: 903A Eller Bldg, TAMU, College Station, TX 77843