[LDM #MIY-170840]: One of our two LDM servers is having problems

Hi Jennifer,

Tom Yoksas here jumping in to point out something I just noticed:

- the latencies being reported from the LDM running on colaweb.gmu.edu
  are high enough to completely explain the unusually small surface METAR
  data files you are reporting:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+colaweb.gmu.edu

As you can see from the plot for the IDS|DDPLUS data feed, the latencies
exceed 3600 seconds (1 hour), and this indicates that you are not actually
receiving all of the data from your upstream feed site(s).

The question now is what is causing the unusually high latencies to
colaweb.gmu.edu?

I would compare/contrast the latencies being experienced on cola.gmu.edu with
those being reported by colaweb.gmu.edu, but it appears that cola.gmu.edu is
not reporting real-time statistics back to us:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex

or, perhaps, it is reporting real-time statistics under a different name.

Cheers,

Tom

Jennifer Adams commented:
> I made another mistake. I was looking at the wrong pqact.conf file (the one in
> the src code instead of $LDMHOME/etc/). Sorry!
> 
> Here is what is in $LDMHOME/etc/pqact.conf regarding the .sa files:
> 
> IDS|DDPLUS
> ^SA.... .... (..)(..).*
> FILE data/dds/sa/(\1:yyyy)(\1:mm)\1\2.sa
> 
> Even with Tom’s recent change to the ulimit settings, the problem with the .sa
> files being too small persists:
> 
> cola.gmu.edu<http://gmu.edu>[ldm]:~/data/dds/sa # ll 20170516*
> -rw-r--r-- 1 ldm   cola 1121394 May 16 12:07 2017051610.sa
> -rw-r--r-- 1 ldm   cola 1099976 May 16 12:07 2017051611.sa
> -rw-r--r-- 1 ldm   cola 1166954 May 16 12:07 2017051612.sa
> -rw-r--r-- 1 ldm   cola 1098117 May 16 16:33 2017051613.sa
> -rw-r--r-- 1 ldm   cola 1095308 May 16 16:33 2017051614.sa
> -rw-r--r-- 1 ldm   cola 1128580 May 16 16:33 2017051615.sa
> -rw-r--r-- 1 ldm   cola 1089389 May 16 16:33 2017051616.sa
> -rw-r--r-- 1 ldm   cola 1105080 May 16 16:33 2017051617.sa
> -rw-r--r-- 1 ldm   cola 1119545 May 16 16:33 2017051618.sa
> -rw-r--r-- 1 ldm   cola 1081663 May 16 16:40 2017051619.sa
> -rw-r--r-- 1 ldm   cola  975945 May 16 16:45 2017051620.sa
> -rw-r--r-- 1 ldm   cola    1302 May 16 16:42 2017051621.sa
> 
> 
> colaweb.gmu.edu<http://gmu.edu>[ldm]:~/data/dds/sa # ll 20170516*
> -rw-r--r-- 1 ldm nobody 1121690 May 16 13:07 2017051610.sa
> -rw-r--r-- 1 ldm nobody 1100366 May 16 13:07 2017051611.sa
> -rw-r--r-- 1 ldm nobody   66045 May 16 13:07 2017051612.sa
> -rw-r--r-- 1 ldm nobody  618290 May 16 16:28 2017051613.sa
> -rw-r--r-- 1 ldm nobody     346 May 16 16:34 2017051614.sa
> -rw-r--r-- 1 ldm nobody  845978 May 16 16:34 2017051615.sa
> -rw-r--r-- 1 ldm nobody  876423 May 16 16:34 2017051616.sa
> -rw-r--r-- 1 ldm nobody  761826 May 16 16:34 2017051617.sa
> -rw-r--r-- 1 ldm nobody  908188 May 16 16:36 2017051618.sa
> -rw-r--r-- 1 ldm nobody  710480 May 16 16:46 2017051619.sa
> -rw-r--r-- 1 ldm nobody    1736 May 16 16:46 2017051620.sa
> 
> —Jennifer
> 
> On May 16, 2017, at 3:25 PM, Unidata LDM Support 
> <address@hidden<mailto:address@hidden>> wrote:
> 
> Thomas,
> 
> The limit on the number of open files is smaller on the upgraded system than
> on the other system -- so my unstated concern about not using a "-close" or
> "-flush" option is specious.
> 
> Let us know if you have further issues.
> 
> 
> Support--
> 
> ulimts:
> 
> working server:
> ulimit a-a
> -bash: ulimit: a-a: invalid number
> [root@gradsusr ldm]# ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 128321
> max locked memory       (kbytes, -l) unlimited
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 16384
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 300000
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 128321
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> 
> 
> new broken server:
> 
> ulimit -a
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> scheduling priority             (-e) 0
> file size               (blocks, -f) unlimited
> pending signals                 (-i) 46271
> max locked memory       (kbytes, -l) 64
> max memory size         (kbytes, -m) unlimited
> open files                      (-n) 1024
> pipe size            (512 bytes, -p) 8
> POSIX message queues     (bytes, -q) 819200
> real-time priority              (-r) 0
> stack size              (kbytes, -s) 8192
> cpu time               (seconds, -t) unlimited
> max user processes              (-u) 46271
> virtual memory          (kbytes, -v) unlimited
> file locks                      (-x) unlimited
> 
> Regards,
> Steve Emmerson
> 
> Ticket Details
> ===================
> Ticket ID: MIY-170840
> Department: Support LDM
> Priority: Normal
> Status: Closed
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 
> --
> Jennifer Miletta Adams
> Center for Ocean-Land-Atmosphere Studies (COLA)
> George Mason University
> 
> 
> 
> 
> 

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: MIY-170840
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.