20020815: LDM product queue on Linux

Interesting LDM queue experiment run by David Wojtowicz on Linux.


------- Forwarded Message

>To: address@hidden
>From: David Wojtowicz <address@hidden>
>Subject: observation about LDM product queue
>Organization: UCAR/Unidata
>Keywords: 200208142221.g7EMLOK12910


Here's an observation I've made that may or may not be of interest to
you all....  our main LDM machine is now flood.atmos.uiuc.edu.... it
is doing *tons* of traffic (UNIDATA,NNEXRAD,NMC2, plus several other
3rd party model feeds)

It is a dual processor AMD machine with 2GB of RAM and a 1GB product
queue running Linux.  This is handy because the whole product queue
can fit into RAM cache, speeding things up.  But still I found, it
spends alot of time flushing changes to the cache back to disk...

I tried an experiment and put ldm.pq on /dev/shm... which is defined
by default on recent RedHat linux releases as a tmpfs
filesystem...which is similar to a ramdisk, but only has a fixed
limit, not a fixed size.  This is similar to using a regular file
that can stay in cache, except that no writes to the disk need be
done.  Although I am having difficulty quantifying it, there seems to
be an increase in performance.   The machine would previously bog
down during heavy traffic periods, but doesn't anymore.

This wouldn't work for everyone... for one thing, you need about 2x
as much RAM as your product queue and be willing to devote half of it
for this purpose.  This works fine in our case.    And, while the
ldm.pq file persists between ldm invocations, it does not persist
between system reboots.   However, in our case, our system is quite
stable and when an administrative reboot is necessary, the file can
be copied to a standard disk file and then copied back after reboot,
if the downtime is going to be short where it's worth preserving
what's in the queue.

As I said, not for everyone, but an interesting experiment you might
want to know about.


-- 
| David Wojtowicz, Sr. Research Programmer
| Department of Atmospheric Sciences Computer Services
| University of Illinois at Urbana-Champaign
| email: address@hidden  phone: (217)333-8390


------- End of Forwarded Message