[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #GLG-935498]: CONDUIT GFS data

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [LDM #GLG-935498]: CONDUIT GFS data
  • Date: Fri, 03 Jun 2016 15:45:08 -0600
Hi Hsie

re: what is the size of your LDM queue

> rainbow:[42]% ldmadmin config
> 
> hostname:              rainbow.al.noaa.gov
> os:                    Linux
> release:               2.6.18-410.el5
> ldmhome:               /home/ldm
> LDM version:           6.13.0
> PATH:
> /home/ldm/ldm-6.13.0/bin:/usr/kerberos/bin:/bin:/usr/bin:/usr/local/bin:/usr/X11R6/bin:/opt/pgi/linux86-64/8.0-3/bin:/opt/intel/fce/10.1.008/bin:/opt/sun/sunstudio12/bin:/usr/local/ncarg/bin:/home/ldm/bin:/home/ldm/decoders/bin:/home/ldm/util:/home/mcidas/linux/bin:/home/mcidas/util:.
> LDM conf file:         /home/ldm/etc/ldmd.conf
> pqact(1) conf file:    /home/ldm/etc/pqact.conf
> scour(1) conf file:    /home/ldm/etc/scour.conf
> product queue:         /home/ldm/queues/ldm.pq
> queue size:            4G bytes
> queue slots:           default
> reconciliation mode:   increase queue
> pqsurf(1) path:        /home/ldm/queues/pqsurf.pq
> pqsurf(1) size:        2M
> IP address:            140.172.240.190
> port:                  388
> PID file:              /home/ldm/ldmd.pid
> Lock file:             /home/ldm/.ldmadmin.lck
> maximum clients:       256
> maximum latency:       3600
> time offset:           3600
> log file:              /home/ldm/logs/ldmd.log
> numlogs:               7
> log_rotate:            1
> netstat:               netstat -A inet -t -n
> top:                   /usr/bin/top -b -n 1
> metrics file:          /home/ldm/logs/metrics.txt
> metrics files:         /home/ldm/logs/metrics.txt*
> num_metrics:           4
> check time:            1
> delete info files:     0
> ntpdate(1):            /usr/sbin/ntpdate
> ntpdate(1) timeout:    20
> time servers:          cirrus.al.noaa.gov tick.boulder.noaa.gov
> tock.boulder.noaa.gov ntp.ucsd.edu ntp1.cs.wisc.edu ntppub.tamu.edu
> otc1.psu.edu timeserver.unidata.ucar.edu
> time-offset limit:     10

OK, your LDM queue is 4 GB in size.  At peak times, rainbow is receiving
over 38 GB of data from all feeds:

Data Volume Summary for rainbow.al.noaa.gov

Maximum hourly volume  38478.037 M bytes/hour
Average hourly volume  17131.927 M bytes/hour

Average products per hour     307307 prods/hour

Feed                           Average             Maximum     Products
                     (M byte/hour)            (M byte/hour)   number/hour
CONDUIT                7909.353    [ 46.167%]    23417.657    91054.277
NGRID                  6002.365    [ 35.036%]    11090.312    37421.340
NEXRAD3                2475.231    [ 14.448%]     3269.320   113673.830
HDS                     285.235    [  1.665%]      589.473    19145.447
NIMAGE                  217.073    [  1.267%]      363.443      208.872
FNEXRAD                 114.977    [  0.671%]      139.929      105.915
IDS|DDPLUS               65.866    [  0.384%]       80.986    45648.191
UNIWISC                  61.828    [  0.361%]      118.010       49.383

The peak time volume is heavily dominated by CONDUIT data.  So, at peak
reception times, the products being received have to be processed out
of the LDM queue within about 6 minutes.  Given this, I recommend that
you increase your LDM queue size to at last 8 GB or more if you have
enough RAM installed on rainbow.  After you increase your queue size,
we need to see if you are correctly processing all of the CONDUIT data.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: GLG-935498
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.