[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[CONDUIT #ZDP-578360]: Will UIllinois be capable of relaying 0.25 degree GFS data in CONDUIT?

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [CONDUIT #ZDP-578360]: Will UIllinois be capable of relaying 0.25 degree GFS data in CONDUIT?
  • Date: Tue, 17 Feb 2015 16:26:13 -0700
Hi David,

re: will UIllinois be able to relay the 0.25 degree GFS data?

> We'd certainly like to be able to be part of relaying this data.

Excellent!

re:
> I'm unsure if the currently in-use systems would be sufficient
> to handle the larger data volumes adequately, but we should be
> able to provision hardware that will.  We should also have plenty
> of network bandwidth available to us.

Very good.

re:
> Can I turn it around and ask the following? What would be the
> recommended specs for a system to comfortably handle this task?

I need to answer this in two ways:

- if one relays all of the data available in the IDD, then the important
  things are:

  - Gbps Ethernet interface(s)

  - LOTS of RAM

    Our experience is that the amount of RAM needed is about 1.3 times
    or more of the size of the LDM queue being used.  So, if one were to
    use a 50 GB queue, it would be best to have at least 64 GB of RAM.

    NB: our rule of thumb is based on our top level IDD relay cluster
    which has 6 backend real servers that are each relaying to 150+
    downstream connections.  If there are a lot less downstreams, the
    amount of RAM needed goes down.

- if CONDUIT is relayed by a single purpose machine, then the specs
  can be significantly less

  Peaks in our test CONDUIT feed indicates up to around 26 GB/hr of
  data is received.  This would imply that one should have at least
  32 GB of RAM to handle the queue memory mapping and LDM and OS use.
  Gpbs Ethernet is also highly recommended.

- since the LDM does not use a lot of CPU, it is not hyper critical
  to have very fast CPUs or a lot of cores

  A modern, single quad core process that supports hyper-threading 
  should be more than enough for the job.

Please let us know if the above was comprehensive enough to proceed.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ZDP-578360
Department: Support CONDUIT
Priority: Normal
Status: Closed