[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Support #BLB-756234]: LDM data lagging when writing to disk

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [Support #BLB-756234]: LDM data lagging when writing to disk
  • Date: Thu, 30 Sep 2010 13:25:25 -0600
Stacey,

> We are having and LDM issue that I'm hoping you might be able to help us
> with.  We are placing data on the LDM queue for a machine and that
> machine writes the data out to files.  What we are seeing is a lag of
> 8-10 minutes or more from when the data is received on the LDM queue to
> when it actually gets written out to disk.  Would you happen to know
> what might be causing this lag?  The machine does receive a lot of data
> on the LDM queue, but it seems to work fine some times and then take
> 8-10 minutes to write the files out at other times.  We are stumped on
> this one.  Have you seen this behavior before?  Do you know what might
> be causing it?  Any help you can provide would be greatly appreciated.

Assuming that the data-products are being written to files using the FILE 
action of pqact(1), then the reason for the delay could very well be caused by 
the throughput through the LDM product-queue. If you're receiving a lot of 
data-products (e.g., all of CONDUIT) and doing a lot of processing of that 
data, then when the data arrives at a high rate (which CONDUIT does 
periodically) the system will be so busy that it might take some minutes to 
FILE a just-added data-product.

For this reason, we recommend that LDMs that request a lot of data don't 
process that data locally but, instead, pass the data on to other, requesting 
LDMs that process subsets of the data.

Hope this helps.

> Thank you,
> Mr. Stacey Simmons
> WeahterTAP.com

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: BLB-756234
Department: Support LDM
Priority: Normal
Status: Closed