[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[CONDUIT #VCP-345838]: Will TAMU be capable of relaying 0.25 degree GFS data in CONDUIT?

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [CONDUIT #VCP-345838]: Will TAMU be capable of relaying 0.25 degree GFS data in CONDUIT?
  • Date: Tue, 17 Feb 2015 15:50:15 -0700
Hi Donna,

This note is a follow-up to the email to Pete Pokrandt (UW/AOS) that you
were CCed on last week...

Can you let us know if the TAMU IDD relay cluster would be capable of
relaying the global 0.25 degree GFS data if those are added to CONDUIT?

The key issues to consider:

- the size of the LDM queues on all machines that make up the idd.tamu.edu
  IDD relay cluster

  The issue with the queue size is what the minimum latency time would
  likely be after the addition of the high volume 0.25 degree GFS data.
  Our testing indicates that the CONDUIT volumes will:

  increase from:

                         Ave/hr                     Max/hr      #Products
  CONDUIT                3522.477    [ 23.987%]     8154.082    69116.104

  to:

                         Ave/hr                     Max/hr      #Products
  CONDUIT                7730.457    [ 32.181%]    21264.600    89421.500

  As you can see, at peak times, there can be over 21 GB/hr of CONDUIT
  data by itself.

  The total volumes will:

  increase from approx:

  Maximum hourly volume  26471.998 M bytes/hour
  Average hourly volume  14697.989 M bytes/hour

  to something like:

  Maximum hourly volume  46971.541 M bytes/hour
  Average hourly volume  24021.893 M bytes/hour

  This suggests that LDM queue sizes will likely need to grow from
  about 30 GB now to almost 50 GB once the 0.25 degree GFS data is
  added to CONDUIT!

- the increase in bandwidth use for bring the 0.25 degree GFS data in
  and relaying it to downstream users

  This is thought to be much less of an impact than the need to
  substantially increase LDM queue size.

We are concerned that the top relayers of CONDUIT data may not be
able to update their systems to be able to handle the 0.25 degree GFS
data, so we want to proceed cautiously before turning on this new
fire hose of data.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************