[LDM #AFT-567406]: GOES-17 data filing question

Hi Mike,

re:
> Thanks for continuing to ponder this.

No worries.  These weird things really nag me!

re:
> Some feedback below, but first I
> want to boil this down to what I think are the most relevant
> characteristics of my issue.  On our machine Charney I stopped all the
> feeds Friday, then on Sunday morning I started feeding just Conus and Full
> disk imagery.  And on Vulcan, I have a bunch of stuff coming in still,
> including only Conus and FullDisk for the ABI/L1 data.

OK.

re:
> Charney is exhibiting the exact same behavior and latency pattern:
> 
> 1) All Conus is coming in since yesterday AM.  Only a few FullDisk images
>    are making it through, even when latency in near zero.
> 2) Latency shows same rough shape.
>   (we'll see what happens over the next few hours with campus traffic ramping
>   up)
> 
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?SATELLITE+vulcan.science.sjsu.edu
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?SATELLITE+charney.met.sjsu.edu
> 
> 
> To me this strongly suggests something general on the network, somewhere
> between Unidata and my servers and most likely at SJSU.
> 
> Agree?

Ordinarily, I would agree, but something you include below makes me want to
reserve judgment.

re:
> Along one of the (many) other lines of investigation:

> Both Vulcan and Charney configs included here:
> 
> [ldm@vulcan ~]$ ldmadmin config
> 
> hostname:              vulcan.science.sjsu.edu
> os:                    Linux
> release:               3.10.0-1062.1.2.el7.x86_64
> ldmhome:               /usr/local/ldm
> LDM version:           6.13.11
> PATH:
> /usr/local/ldm/ldm-6.13.11/bin:/usr/local/ldm/decoders:/usr/local/ldm/util:/usr/local/ldm/bin:/usr/lib64/mpich/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/home/gempak/GEMPAK7/os/linux64/bin:/home/gempak/GEMPAK7/bin
> LDM conf file:         /usr/local/ldm/etc/ldmd.conf
> pqact(1) conf file:    /usr/local/ldm/etc/pqact.conf
> scour(1) conf file:    /usr/local/ldm/etc/scour.conf
> product queue:         /usr/local/ldm/var/queues/ldm.pq
> queue size:            500M bytes
> queue slots:           default
> reconciliation mode:   do nothing
> pqsurf(1) path:        /usr/local/ldm/var/queues/pqsurf.pq
> pqsurf(1) size:        2M
> IP address:            0.0.0.0
> port:                  388
> PID file:              /usr/local/ldm/ldmd.pid
> Lock file:             /usr/local/ldm/.ldmadmin.lck
> maximum clients:       256
> maximum latency:       3600
> time offset:           3600
> log file:              /usr/local/ldm/var/logs/ldmd.log
> numlogs:               7
> log_rotate:            1
> netstat:               /bin/netstat -A inet -t -n
> top:                   /usr/bin/top -b -n 1
> metrics file:          /usr/local/ldm/var/logs/metrics.txt
> metrics files:         /usr/local/ldm/var/logs/metrics.txt*
> num_metrics:           4
> check time:            1
> delete info files:     0
> ntpdate(1):            /usr/sbin/ntpdate
> ntpdate(1) timeout:    5
> time servers:          ntp.ucsd.edu ntp1.cs.wisc.edu ntppub.tamu.edu
> otc1.psu.edu timeserver.unidata.ucar.edu
> time-offset limit:     10

Quick comment:

The LDM queue on vulcan is MASSIVELY undersized.  The current queue size is 
500MB
and the FullDisk Channel 02 images can be as large as 445MB!

The very first thing that needs to happen is to make the LDM queue on
vulcan MUCH, MUCH larger.  Given the current amount of data being
REQUESTed:

Data Volume Summary for vulcan.science.sjsu.edu

Maximum hourly volume  23575.039 M bytes/hour
Average hourly volume   9862.106 M bytes/hour

Average products per hour     138814 prods/hour

Feed                           Average             Maximum     Products
                     (M byte/hour)            (M byte/hour)   number/hour
CONDUIT                5813.895    [ 58.952%]    19895.421    52265.535
SATELLITE              2446.310    [ 24.805%]     3717.159      369.884
HDS                    1239.508    [ 12.568%]     1785.246    38765.767
FNEXRAD                 108.688    [  1.102%]      137.162      105.233
UNIWISC                  94.174    [  0.955%]      145.057       50.628
NIMAGE                   84.159    [  0.853%]      130.801      120.488
IDS|DDPLUS               75.348    [  0.764%]       86.674    47076.907
LIGHTNING                 0.023    [  0.000%]        0.064       59.814

The LDM queue should be at least 32 GB in size.

Question:

- does vulcan have enough RAM to make a large LDM queue without impacting
  other things running on the machine?

  Another way of asking this is:  how much RAM does vulcan have?

re:
> [ldm@charney current]$ ldmadmin config
> 
> hostname:              charney.met.sjsu.edu
> os:                    Linux
> release:               3.10.0-1062.1.2.el7.x86_64
> ldmhome:               /usr/local/ldm
> LDM version:           6.13.11
> PATH:
> /usr/local/ldm/ldm-6.13.11/bin:/usr/local/ldm/decoders:/usr/local/ldm/util:/usr/local/ldm/bin:/usr/lib64/mpich/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/home/gempak/GEMPAK7/os/linux64/bin:/home/gempak/GEMPAK7/bin
> LDM conf file:         /usr/local/ldm/etc/ldmd.conf
> pqact(1) conf file:    /usr/local/ldm/etc/pqact.conf
> scour(1) conf file:    /usr/local/ldm/etc/scour.conf
> product queue:         /usr/local/ldm/var/queues/ldm.pq
> queue size:            500M bytes
> queue slots:           default
> reconciliation mode:   do nothing
> pqsurf(1) path:        /usr/local/ldm/var/queues/pqsurf.pq
> pqsurf(1) size:        2M
> IP address:            0.0.0.0
> port:                  388
> PID file:              /usr/local/ldm/ldmd.pid
> Lock file:             /usr/local/ldm/.ldmadmin.lck
> maximum clients:       256
> maximum latency:       3600
> time offset:           3600
> log file:              /usr/local/ldm/var/logs/ldmd.log
> numlogs:               7
> log_rotate:            1
> netstat:               /bin/netstat -A inet -t -n
> top:                   /usr/bin/top -b -n 1
> metrics file:          /usr/local/ldm/var/logs/metrics.txt
> metrics files:         /usr/local/ldm/var/logs/metrics.txt*
> num_metrics:           4
> check time:            1
> delete info files:     0
> ntpdate(1):            /usr/sbin/ntpdate
> ntpdate(1) timeout:    5
> time servers:          ntp.ucsd.edu ntp1.cs.wisc.edu ntppub.tamu.edu
> otc1.psu.edu timeserver.unidata.ucar.edu
> time-offset limit:     10

The exact same comment goes for charney: its LDM queue is severely
undersized given the data being REQUESTed.

Question:

- the same question goes for charney as the one I posed above for
  vulcan:

  How much RAM is installed on charney?

re: Are you gathering system metrics using 'ldmadmain addmetrics'

> No, but just started.  I'll look into gnuplot in a bit here.
> Unfortunately I have bunch of meetings this morning, so I'll be watching at
> a distance.  My hunch is the latency will spike again around 0900 local
> because of campus Internet traffic, but we'll see.

We can say without hesitation that until the LDM queues on both vulcan
and charney are made much larger, all bets are off wrt to proper processing
of received products.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: AFT-567406
Department: Support LDM
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.