[LDM #MIY-170840]: One of our two LDM servers is having problems

Hi Brian, Tom and Jennifer,

Brian wrote:
> the -flush made it worse.

The only thing I can imagine that would cause this is if the file system
being written to is _very_ slow.

re:
> I have turned off the LDM on colaweb, while we try the test
> on cola6.

OK.

Questions:

- is cola6.gmu.edu a VM also?

  If yes, is the virtualization environment the same as for colaweb?

- where is the LDM queue located?

  What I mean, is if your LDM queue is located on an NFS-mounted file
  system.  If it is, this may be the cause of your problems.

Comments:

- we define the product latency as the time difference between the receipt
  of a product by the local LDM (when the product is inserted into the
  local LDM queue) and the time that the product was first inserted into
  an LDM queue for relay.

- the slowness of writes by LDM FILE actions _should_ have nothing
  to do with product latencies

  The only exception to the notion that file system writes should have
  nothing to do with receipt latencies is when the LDM queue is located
  on a file system where writes are _very_ slow.  It is for this reason
  that we advise LDM users to _never_ put their LDM queue on an NFS-mounted
  file system.

- I run a CentOS 6.9 x86_64 VM instance inside of VMware Player that is
  running on my Windows 7 laptop, and I have run an LDM that was FILEing
  all NEXRAD3 products to a file system inside of the VM, and I experienced
  no write-related problems.

  I must quickly add, however, that I did not run the ingest and FILEing for
  NEXRAD3 products for a long period of time, so it is possible that "things"
  would have become bound up if I had run for long "enough".

- other users are running LDMs in VMs in a couple different types of
  virtualization environments (e.g., VMware and virtualbox come to mind),
  and none have contacted us with problems of high latencies or impaired
  ability to write products to disk

  In fact, the toplevel IDD nodes that relay CONDUIT are VMware-hosted VMs,
  and CONDUIT has a lot more products and much, much more volume than
  IDS|DDPLUS.  To get an idea of the number of products and volumes of
  the significant IDD data feeds, consider the following summary statistics
  from a motherlode-class machine that we run here in Unidata:

Data Volume Summary for lead.unidata.ucar.edu

Maximum hourly volume  70976.622 M bytes/hour
Average hourly volume  44292.075 M bytes/hour

Average products per hour     412439 prods/hour

Feed                           Average             Maximum     Products
                     (M byte/hour)            (M byte/hour)   number/hour
FSL2                  14616.139    [ 32.999%]    17570.506    22496.596
CONDUIT                8488.236    [ 19.164%]    23905.443    96092.234
NGRID                  6828.765    [ 15.418%]    12763.837    41711.787
NEXRAD2                6439.101    [ 14.538%]     9712.475    70899.021
NOTHER                 3629.480    [  8.194%]     7077.190     6937.000
NEXRAD3                2092.125    [  4.723%]     2924.046    96889.149
FNMOC                  1223.423    [  2.762%]     4410.106     3395.340
HDS                     300.532    [  0.679%]      537.559    26194.447
NIMAGE                  203.051    [  0.458%]      350.919      202.511
GEM                     180.209    [  0.407%]      870.590     1084.681
FNEXRAD                 125.649    [  0.284%]      154.638      104.936
IDS|DDPLUS               70.652    [  0.160%]       80.433    45867.702
UNIWISC                  66.133    [  0.149%]      116.881       47.894
EXP                      26.623    [  0.060%]       47.300      219.255
LIGHTNING                 1.958    [  0.004%]        5.673      296.383

Please keep us apprised of your attempts of running the LDM on your
cola6 machine.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: MIY-170840
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.