[CONDUIT #VCP-345838]: Will TAMU be capable of relaying 0.25 degree GFS data in CONDUIT?

Hi Donna and Gerry,

re: will TAMU be able to handle the increased volumes in CONDUIT if the
0.25 degree GFS were added

> I am investigating. The TAMU relay cluster (ldm.tamu.edu) had 16Gb queues
> with age-of-oldest-product, during peak times, going down to 28 minutes.

OK.

re:
> The queues have been bumped up to 20Gb but I haven't checked on it yet (I'm
> at an out-of-town conference today and part of tomorrow.

We project that the peak volumes for all feeds relayed in the IDD will
increase to on the order of 38 GB per hour if/when the 0.25 degree GFS data
is added to CONDUIT.  The numbers I provided earlier (46 GB/hr) incorrectly
included the HRRR data we are receiving in a point-to-point feed from NOAA/GSD,
so they were an overestimate.  BUT, the snapshot volumes provided were also
made at a time when the volume of NEXRAD2/NEXRAD3 data was relatively low.  
Given
this under-representation of radar data in the estimate, it is not unreasonable
to project that data volumes could be has high as 46+ GB/hr in the coming year
for some periods.  If this estimate holds, then it is not unreasonable to
suggest that LDM queue sizes of 50 GB need to be used in order for there to
be an hour of data available in one's queue.

re:
> I think Gerry would have a better idea of how to answer your question. I
> can also ask about adding memory to the cluster nodes (currently 32Gb).

The real server backends of our top level IDD relay cluster, 
idd.unidata.ucar.edu,
all have 64 GB of RAM.  We will be increasing the LDM queue sizes on each of
these machines in the coming weeks (we will be doing this in conjunction with
OS upgrades) so that we can handle the increased volume while maintaining
at least a hour's worth of data that can be relayed.

re:
> I'm currently trying to find a way to measure REQUESTs from downstream
> users and give my boss a good idea of how "invaluable" the TAMU ldm is (he
> wants to get to know our downstream users too, for instance, are they
> mostly academic institutions?

The extended regular expression a downstream uses in their REQUEST is
logged in the upstream's LDM log file.  You can farm you log files to
get the set of REQUESTs being made and then estimate how much data is
flowing off of your top level IDD relay cluster, idd.tamu.edu.

re:
> and how much do our downstream users depend
> on our relay host(s)? or how few would "miss" us if we didn't (Heaven
> forbid!) relay LDM products?

There are 4 top level relay nodes in the IDD, and TAMU is one of them:

idd.unidata.ucar.edu
idd.meteo.psu.edu
idd.tamu.edu
idd.aos.wisc.edu

UIllinois also relays most of what is available in the IDD, but they
are not included in this list because their installation is not as
robust as the 4 listed.

To answer the question at hand:  TAMU ceasing to act as a top level IDD
relay would have serious negative impacts on the operation of the IDD!!!

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: VCP-345838
Department: Support CONDUIT
Priority: Normal
Status: Closed