20050401: Long write delays on bigbird

>From:  Gerry Creager n5jxs <address@hidden>
>Organization:  AATLT, Texas A&M University
>Keywords:  200504012058.j31KwRv2003675 LDM queue large file Fedora Core 2

Hi Gerry,

Sorry for the silence!

>I'd remade the queue to 4G, per the last recommendation ("if 2G's good, 
>4G's better!?") but pretty much as soon as I got the queue filled up, 
>products started seeing a long time to get written to disk.

This could be caused by more products being processed (not overwritten
in a too small of a queue (based on what you are ingesting)); by
the pqact's having too many actions to process for each header 
pattern matched in a pqact.conf file; or by slow access to the
products in the queue.  If the cause is the latter, then I suggest
rebuilding the LDM _without_ large queue support:

<as 'ldm'>
ldmadmin stop
cd ldm-6.2.1/src            <- or whatever version you are using
make distclean
./configure ----disable-max-size
make
make install
sudo make install_setuids
cd ~
ldmadmin delqueue
-- edit ~ldm/etc/ldmadmin-pl.conf and make sure that the queue size is
   less than or equal to 2 GB
ldmadmin mkqueue -f
ldmadmin start

I am suspicious of Fedora Core 2, however, given that you were able
to process all of the data coming into a 4 GB queue under Fedora
Core 1.  It may well be time to upgrade to FC3...

>One effect 
>of this is my CONUS L3 mosaic was complaining the data was too old to 
>use... an accurate assessment.  Another is the Level II data are old 
>when we can make the images.
>
>I've dropped the queue size to 2G (counterintuitive but things had been 
>writing/working with the smaller pq before) to see what's happening.

OK.

>I'm at a loss, and going too many directions.  If you've got 
>suggestions, perhaps we can figure out what I've done that's causing the 
>machine grief.

I am suspicious of:

- FC2
- the overhead of having a large queue support built into the LDM
  modules under FC2

>I'm cc:' ing James Esslinger, who's doing (good!) sysadmin work for me. 
>He may better catch something I've missed.

Try the above and let us know if it helps.  I will say it again,
however, that we have found FC3 to be a _much_ better performer than
FC2, and we dumped our FC2 installations in favor of FC3 because of
that observation.

>Thanks, Gerry

Again, sorry for the protracted silence!

Cheers,

Tom
--
+-----------------------------------------------------------------------------+
* Tom Yoksas                                             UCAR Unidata Program *
* (303) 497-8642 (last resort)                                  P.O. Box 3000 *
* address@hidden                                   Boulder, CO 80307 *
* Unidata WWW Service                             http://www.unidata.ucar.edu/*
+-----------------------------------------------------------------------------+