[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[CONDUIT #EKQ-308585]: GFS data problem

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [CONDUIT #EKQ-308585]: GFS data problem
  • Date: Wed, 25 Aug 2021 15:42:51 -0600
Hi,

re: how much RAM does your machine have
> It has 256 GB of RAM. Do you think it would make sense to increase queue size 
> even more?

If the machine in question is ldm.dri.edu, then the following is a snapshot of 
the
volume of data that it is receiving:

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_summary_volume?ldm.dri.edu

Data Volume Summary for ldm.dri.edu

Maximum hourly volume  66216.456 M bytes/hour
Average hourly volume  25798.251 M bytes/hour

Average products per hour     242594 prods/hour

Feed                           Average             Maximum     Products
                     (M byte/hour)            (M byte/hour)   number/hour
CONDUIT               14858.372    [ 57.594%]    51172.197    91321.298
NGRID                  7107.668    [ 27.551%]    12533.545    66386.064
HDS                    3748.626    [ 14.531%]    11502.015    29879.234
IDS|DDPLUS               83.545    [  0.324%]       97.793    54488.809
EXP                       0.040    [  0.000%]        0.054      518.723

Assuming that this volume will stay the same, it would be "good" to 
set the queue size to be 32 GB as it would hold, on average, a bit more
than 1 hour of data.

Why is it important/useful to hold more data in one's LDM queue?

- The longer a product is in the local LDM queue, the more time the
  system has to process data out of the queue.

- The LDM can detect and reject duplicate products

  It rejects a newly-received product when a product with the same MD5 signature
  exists in the LDM queue. So, the longer a product resides in the queue, the
  better the duplicate product detection and rejection system can work.

I hope that this helps...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: EKQ-308585
Department: Support CONDUIT
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.