[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Support #SYM-524686]: ldm limitations/capabilities

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [Support #SYM-524686]: ldm limitations/capabilities
  • Date: Mon, 12 Mar 2007 14:02:04 -0600
Joey,

> Thanks so much for the input and answering these questions.  NCEP is
> tinkering with the idea of using the ldm to feed NCDC a supply of NOMADS
> data.  Data quantities could approach 300 GB/day to begin with about a
> 10% increase per year.  Most of the 300 GB/day would occur over a 3 hour
> time span 4 times per/day during GFS and GENS time (25-30 GB/hour).  At
> the current time, we are just attempting to evaluate what methodologies
> are possible with budgeting/hardware/network requirements to come later.
> 
> It sounds like the ldm is a good candidate given the adequate
> hardware/memory solution.

300 GB/d is a greater data-rate than any LDM is currently receiving.
If the network bandwidth is available, however, I see no theoretical
obstacles.

What is the available bandwidth between NCDC and the source of the
data?

With that much data, you'll probably want the data-products to be on
the large side (i.e., on the order of a megabyte or greater).

I would also recommend using the recently-added "-i" option of the
"pqinsert" utility when creating the data-products because it greatly
reduces the load on the CPU.

The LDM package has the capability of building a test version that
can determine the maximum throughput possible.  The upstream LDM
uses a faux product-queue which auto-generates data-products of a 
specifiable size at a specifiable rate.  Couple that with the
existing "rtstats" reporting system and you should be able to
quickly determine the feasibility of using the LDM.

Please contact us if you have any questions.

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: SYM-524686
Department: Support LDM
Priority: Normal
Status: On Hold