[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[LDM #AFT-567406]: GOES-17 data filing question
- Subject: [LDM #AFT-567406]: GOES-17 data filing question
- Date: Mon, 04 Nov 2019 10:57:43 -0700
Hi Mike,
re:
> Thanks for continuing to ponder this.
No worries. These weird things really nag me!
re:
> Some feedback below, but first I
> want to boil this down to what I think are the most relevant
> characteristics of my issue. On our machine Charney I stopped all the
> feeds Friday, then on Sunday morning I started feeding just Conus and Full
> disk imagery. And on Vulcan, I have a bunch of stuff coming in still,
> including only Conus and FullDisk for the ABI/L1 data.
OK.
re:
> Charney is exhibiting the exact same behavior and latency pattern:
>
> 1) All Conus is coming in since yesterday AM. Only a few FullDisk images
> are making it through, even when latency in near zero.
> 2) Latency shows same rough shape.
> (we'll see what happens over the next few hours with campus traffic ramping
> up)
>
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?SATELLITE+vulcan.science.sjsu.edu
> http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?SATELLITE+charney.met.sjsu.edu
>
>
> To me this strongly suggests something general on the network, somewhere
> between Unidata and my servers and most likely at SJSU.
>
> Agree?
Ordinarily, I would agree, but something you include below makes me want to
reserve judgment.
re:
> Along one of the (many) other lines of investigation:
> Both Vulcan and Charney configs included here:
>
> [ldm@vulcan ~]$ ldmadmin config
>
> hostname: vulcan.science.sjsu.edu
> os: Linux
> release: 3.10.0-1062.1.2.el7.x86_64
> ldmhome: /usr/local/ldm
> LDM version: 6.13.11
> PATH:
> /usr/local/ldm/ldm-6.13.11/bin:/usr/local/ldm/decoders:/usr/local/ldm/util:/usr/local/ldm/bin:/usr/lib64/mpich/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/home/gempak/GEMPAK7/os/linux64/bin:/home/gempak/GEMPAK7/bin
> LDM conf file: /usr/local/ldm/etc/ldmd.conf
> pqact(1) conf file: /usr/local/ldm/etc/pqact.conf
> scour(1) conf file: /usr/local/ldm/etc/scour.conf
> product queue: /usr/local/ldm/var/queues/ldm.pq
> queue size: 500M bytes
> queue slots: default
> reconciliation mode: do nothing
> pqsurf(1) path: /usr/local/ldm/var/queues/pqsurf.pq
> pqsurf(1) size: 2M
> IP address: 0.0.0.0
> port: 388
> PID file: /usr/local/ldm/ldmd.pid
> Lock file: /usr/local/ldm/.ldmadmin.lck
> maximum clients: 256
> maximum latency: 3600
> time offset: 3600
> log file: /usr/local/ldm/var/logs/ldmd.log
> numlogs: 7
> log_rotate: 1
> netstat: /bin/netstat -A inet -t -n
> top: /usr/bin/top -b -n 1
> metrics file: /usr/local/ldm/var/logs/metrics.txt
> metrics files: /usr/local/ldm/var/logs/metrics.txt*
> num_metrics: 4
> check time: 1
> delete info files: 0
> ntpdate(1): /usr/sbin/ntpdate
> ntpdate(1) timeout: 5
> time servers: ntp.ucsd.edu ntp1.cs.wisc.edu ntppub.tamu.edu
> otc1.psu.edu timeserver.unidata.ucar.edu
> time-offset limit: 10
Quick comment:
The LDM queue on vulcan is MASSIVELY undersized. The current queue size is
500MB
and the FullDisk Channel 02 images can be as large as 445MB!
The very first thing that needs to happen is to make the LDM queue on
vulcan MUCH, MUCH larger. Given the current amount of data being
REQUESTed:
Data Volume Summary for vulcan.science.sjsu.edu
Maximum hourly volume 23575.039 M bytes/hour
Average hourly volume 9862.106 M bytes/hour
Average products per hour 138814 prods/hour
Feed Average Maximum Products
(M byte/hour) (M byte/hour) number/hour
CONDUIT 5813.895 [ 58.952%] 19895.421 52265.535
SATELLITE 2446.310 [ 24.805%] 3717.159 369.884
HDS 1239.508 [ 12.568%] 1785.246 38765.767
FNEXRAD 108.688 [ 1.102%] 137.162 105.233
UNIWISC 94.174 [ 0.955%] 145.057 50.628
NIMAGE 84.159 [ 0.853%] 130.801 120.488
IDS|DDPLUS 75.348 [ 0.764%] 86.674 47076.907
LIGHTNING 0.023 [ 0.000%] 0.064 59.814
The LDM queue should be at least 32 GB in size.
Question:
- does vulcan have enough RAM to make a large LDM queue without impacting
other things running on the machine?
Another way of asking this is: how much RAM does vulcan have?
re:
> [ldm@charney current]$ ldmadmin config
>
> hostname: charney.met.sjsu.edu
> os: Linux
> release: 3.10.0-1062.1.2.el7.x86_64
> ldmhome: /usr/local/ldm
> LDM version: 6.13.11
> PATH:
> /usr/local/ldm/ldm-6.13.11/bin:/usr/local/ldm/decoders:/usr/local/ldm/util:/usr/local/ldm/bin:/usr/lib64/mpich/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin:/home/gempak/GEMPAK7/os/linux64/bin:/home/gempak/GEMPAK7/bin
> LDM conf file: /usr/local/ldm/etc/ldmd.conf
> pqact(1) conf file: /usr/local/ldm/etc/pqact.conf
> scour(1) conf file: /usr/local/ldm/etc/scour.conf
> product queue: /usr/local/ldm/var/queues/ldm.pq
> queue size: 500M bytes
> queue slots: default
> reconciliation mode: do nothing
> pqsurf(1) path: /usr/local/ldm/var/queues/pqsurf.pq
> pqsurf(1) size: 2M
> IP address: 0.0.0.0
> port: 388
> PID file: /usr/local/ldm/ldmd.pid
> Lock file: /usr/local/ldm/.ldmadmin.lck
> maximum clients: 256
> maximum latency: 3600
> time offset: 3600
> log file: /usr/local/ldm/var/logs/ldmd.log
> numlogs: 7
> log_rotate: 1
> netstat: /bin/netstat -A inet -t -n
> top: /usr/bin/top -b -n 1
> metrics file: /usr/local/ldm/var/logs/metrics.txt
> metrics files: /usr/local/ldm/var/logs/metrics.txt*
> num_metrics: 4
> check time: 1
> delete info files: 0
> ntpdate(1): /usr/sbin/ntpdate
> ntpdate(1) timeout: 5
> time servers: ntp.ucsd.edu ntp1.cs.wisc.edu ntppub.tamu.edu
> otc1.psu.edu timeserver.unidata.ucar.edu
> time-offset limit: 10
The exact same comment goes for charney: its LDM queue is severely
undersized given the data being REQUESTed.
Question:
- the same question goes for charney as the one I posed above for
vulcan:
How much RAM is installed on charney?
re: Are you gathering system metrics using 'ldmadmain addmetrics'
> No, but just started. I'll look into gnuplot in a bit here.
> Unfortunately I have bunch of meetings this morning, so I'll be watching at
> a distance. My hunch is the latency will spike again around 0900 local
> because of campus Internet traffic, but we'll see.
We can say without hesitation that until the LDM queues on both vulcan
and charney are made much larger, all bets are off wrt to proper processing
of received products.
Cheers,
Tom
--
****************************************************************************
Unidata User Support UCAR Unidata Program
(303) 497-8642 P.O. Box 3000
address@hidden Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage http://www.unidata.ucar.edu
****************************************************************************
Ticket Details
===================
Ticket ID: AFT-567406
Department: Support LDM
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata
inquiry tracking system and then made publicly available through the web. If
you do not want to have your interactions made available in this way, you must
let us know in each email you send to us.