[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20040712: FILEing NNEXRAD/NIDS on Linux reiserfs

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: 20040712: FILEing NNEXRAD/NIDS on Linux reiserfs
  • Date: Mon, 12 Jul 2004 16:30:21 -0600
>From: Harry Edmon <address@hidden>
>Organization: University of Washington
>Keywords: 200407122128.i6CLScaW020180 LDM FILE NNEXRAD reiserfs

Hi Harry,

>I am in the process of setting up a new weather data ingestor.  I am using the
>suggested pqact files for gempak.  I notice that the process dealing with the
>NIDS data falls behind in its processing.  Everything else is okay.  I am
>wondering if the problem is the large number of files this process has to
>create.

It is typically a bottleneck in the file I/O when one is ingesting
all Level III products and FILEing them to disk.

>It is writing to a reiserfs.  Have you seen this behavior?

Yes, we have, but not on reiserfs since we havn't used this file
system.  We found that it is hard to keep up with the FILEing of all
NEXRAD Level III products to a disk that is not a striped RAID.

If you have one ldmd.conf pqact invocation processing just the NNEXRAD
data (it is not looking at any other feed type), and it is falling
behind, you might try splitting the processing load among several
pqacts like is being done for the CRAFT data.  This may or may not
help.  Writing to a fast, striped RAID really does help -- that is how
we process the Level III data here.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publically available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.