[LDM #MIW-275261]: some questions on hardware configs for our LDM server

Hi Greg (with CC to Sen),

Two of us (Steve Emmerson and I) have been working with Sen to get
a better understanding of the LDM data feeds and processing on
titan.met.sjsu.edu, and to do some tuning to help titan function better.
A lot has been learned over the past week, and Sen asked that we update
you on what has been found and changed, and the effect it has had on
titan's performance.

First, what has been done

- increase the size of the LDM queue from 500 MB (which is the default)
  to 12 GB

  The first thing that was found on titan was the LDM queue was
  not sized properly to handle the volume of data that titan
  was/is REQUESTing.

  Our "rule of thumb" is to try and size one's LDM queue so that it
  will hold about 1 hour of received data.  The principle reason for
  this is that the LDM can reject newly received products if they
  have the same MD5 signature as ones already in the LDM queue.
  Rejection of duplicate products decreases the number/volume of
  products that get processed by LDM pattern-action file actions.

  A quick look at the real time stats that titan has been sending back
  to us (links are included below) showed that the volume of
  data being written into titan's LDM queue was peaking at over 100 GB
  per hour.  This anomalous volume was a BIG red flag for us since the
  feeds being REQUESTed do not contain this much data.  The cause of
  the anomalous volume of data being inserted into titan's LDM queue
  was that duplicate products were not being rejected as the product
  residency time in the LDM queue was on the order of a few seconds,
  not the recommended one hour.  The extremely low residency time resulted
  in some products that had been received not being processed at all,
  and in a LOT of products that had been received and processed being
  received and processed again.

- we find it most useful for sites running the LDM to monitor system
  performance by running the LDM utility 'ldmadmin addmetrics' once per
  minute from a cron job

  Since this was not being done on titan, it was added.  The log file
  used for performance monitoring is ~ldm/logs/metrics.log.  This
  file will be "rotated" (renamed to metrics.log.1, etc.) periodically
  by another action in 'ldm's crontab.

  One of the things that the metrics.txt log file contains is the age of
  the oldest product in the LDM queue. It was by turning on generation
  of the metrics.log file that the extremely short residency time in
  the LDM queue was found.  Again, when metrics monitoring was first
  turned on, the residency time (which is indicated by the age of the
  oldest product in the queue) was found to be only a few seconds, not
  an hour like is recommended.

  Steve and my hunch was that the anomalously high volume of data flowing
  into titan's LDM queue was a result of products being received and
  inserted into the LDM queue multiple times. This can easily happen when
  there are redundant REQUESTs for feeds.

  In order to increase product residency times, titan's LDM queue
  was increased from 500 MB to 12 GB.  It was felt that making the queue
  any larger would interfere with the processing that is being done on
  titan since titan only has 24 GB of RAM.

  After increasing the LDM queue size, we saw a significant decrease
  in the volume of data received being reported in the real time stats
  that titan is sending to us.  We interpret this as an indication that
  rejection of products received more than once is now working as designed.
  The decrease in data being received has also resulted in a decrease in
  the amount of processing (decoding) that titan is doing since it no longer
  is running the same actions on duplicate data.

- eliminate duplicate LDM feed REQUESTs while making sure that all
  of the data that was being REQUESTed is still being REQUESTed

  It turns out that there was quite a bit of duplication of REQUESTs
  for a number of the data feeds that titan was asking for. The duplicated
  feed REQUESTs were eliminated, and this resulted in a further decrease
  in not only the data being received by titan, but in the amount of
  LDM-related processing that titan is doing (e.g., decoding data
  into GEMPAK-compatible formats, etc.).

- consolidated feed REQUESTs

  After letting the LDM run for awhile after increasing the size of
  the LDM queue and eliminating duplicate feed REQUESTs and while
  monitoring the reported reception latencies for the various feeds,
  efforts turned to consolidating feed REQUESTs where consolidation made
  sense.  This was done to decrease the number of LDM processes that
  run continuously.

  Keeping down the number of feed REQUESTs that a system makes will
  help to decrease the LDM related processing load as there will be
  fewer REQUESTing processes. At the same time, splitting feed
  REQUESTs for high volume feeds like what is being done for CONDUIT
  has the beneficial effect of minimizing reception latencies.
  Exactly how one should consolidate some feed REQUESTs while splitting
  others is totally dependent on the number of products and volumes
  in the feeds being REQUESTed. Smart splitting of feed REQUESTs
  is also dependent on the LDM/IDD Product IDs for the products in
  a feed. Each CONDUIT product, for instance, has a sequence number
  as the last value in the Product ID. This sequence number makes it
  easy to split the feed in a way that all data is REQUESTed. The
  HRRR products in the FSL2 feed that originates at NOAA/GSD, on the
  other hand, are not easily split into multiple, disjoint REQUESTs.
  The other feeds that suffer from the same problem as FSL2 are NGRID,
  FNMOC and HDS, and this is a problem for NGRID and FNMOC as they are
  both high volume feeds that have lots of products. NEXRAD2, which
  contains NEXRAD Level 2 volume scan chunks, is a bit easier to split
  as is NEXRAD3, which contains NEXRAD Level 3 products. NEXRAD2 is
  one of the higher volume feeds in the IDD, and NEXRAD3 is the
  feed that has the most number of products per hour.

- installed the latest LDM release, ldm-6.13.6

  The latest release of the LDM was installed to make sure that none
  of what is being seen is a result of an older and possibly less
  efficient LDM.

  It was not expected that this would result in significant
  improvements in terms of getting and processing data, and it did not.
  Nonetheless, it is always the best idea to be running the latest
  version of any software since newer releases typically are more
  efficient and many times have new features that can be useful.

Where titan stands now:

- titan is now processing a LOT more of the data that it is
  REQUESTing than it was before the changes outlined above

  Also, the processing impact on the file system as measured
  by I/O wait has decreased substantially. This is a very good thing
  since the multiple processing of data that had already had been
  received was likely causing performance problems indicated by
  very high I/O waits that you reported in a previous email.

- splitting high volume feeds to minimize their product latencies

  An attempt was made to split the single FSL2 HRRR feed REQUEST into
  thirds with the hope that this would help reduce the receipt latency
  for those  HRRR products. The latencies for the FSL2 HRRR products have
  decreased a bit, but not as nicely/substantially/much as the latencies
  for the CONDUIT feed. The reason for this is that the FSL2 HRRR
  products Product IDs do not lend themselves to an easy split of
  feed REQUEST(s).  In practice this means that one needs to have
  sufficient network bandwidth to receive the high volume FSL2
  feed in a single feed REQUEST.

  The latencies for NGRID products remain high enough that not
  all of the NGRID products are actually being received, since their
  residency time in the upstream LDM queue(s) is not larger than
  the time it takes for the products to be received on titan.
  The problem with the NGRID feed is exactly analogous to that
  with the FSL2 feed.

  FNMOC latencies also remain high, but it appears that all of the
  products are received and processed for most hours.

  NEXRAD2 latencies also remain very high. In fact, a lot of the
  time NEXRAD2 products are not being received since the latencies
  exceed their residency time in the upstream LDM's queue. This
  is a problem that may be mitigated (but possibly not solved)
  by smart splitting of the NEXRAD2 feed REQUEST into multiple,
  disjoint REQUESTs.

Observations:

- After the experimentation with consolidating and splitting feed
  REQUESTs, it is our opinion that the principle cause of the inability
  to receive all of the data being REQUESTed is the network bandwidth
  available to titan

  We say this after reviewing a couple plots of the metrics being gathered
  by 'ldmadmin plotmetrics'.  We encourage you to run 'ldmadmin plotmetrics'
  as user 'ldm' to get an idea of the variety of metrics that are now being
  gathered once per minute.

  In particular, we encourage you to look at:

  - the plot that shows the time series of CPU modes and I/O wait

    This plot shows that titan is idle much of the time, so the
    machine is not the cause of data not being processed.

  - the plot that shows the time series of CPU load

    NB: titan has 24 CPUs, so load averages of up to 10 should not be
    interpreted as being overly high.  Our interpretation of this plot
    is that titan does get busy, but not criplingly so.

  Our view is also based on a careful review of the real time stats
  latencies being reported by titan back to us. You can generate current
  plots of latencies for all feeds being requested online at:

  Unidata HomePage
  http://www.unidata.ucar.edu

    Data -> IDD Operational Status
    http://rtstats.unidata.ucar.edu/rtstats/

      Statistics by Host
      http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex

        titan.met.sjsu.edu [6.13.6]
        
http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?titan.met.sjsu.edu

- further increasing the size of the LDM queue could result in even
  fewer products being redundantly received (i.e., more duplicates
  being rejected), and this would, in turn, reduce the processing
  load even more

  NB: The only way that the LDM queue could/should be increased to
  something like the maximum amount of data being received in an
  hour is by increasing the amount of RAM installed in titan.

  The following cumulative volume summary listing from titan shows what
  is currently being received by titan:

http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_summary_volume?titan.met.sjsu.edu

Data Volume Summary for titan.met.sjsu.edu

Maximum hourly volume  60026.065 M bytes/hour
Average hourly volume  34232.870 M bytes/hour

Average products per hour     387841 prods/hour

Feed                           Average             Maximum     Products
                     (M byte/hour)            (M byte/hour)   number/hour
FSL2                  11177.067    [ 32.650%]    17054.068    15580.956
CONDUIT                7957.826    [ 23.246%]    21930.253    89129.044
NGRID                  4936.321    [ 14.420%]    12914.648    32253.000
NEXRAD2                3070.162    [  8.968%]    11036.379    33935.333
NEXRAD3                2916.793    [  8.520%]     3480.750   122637.156
FNMOC                  1968.504    [  5.750%]    10018.393     3435.956
HDS                    1190.873    [  3.479%]     1848.008    43404.244
FSL3                    608.333    [  1.777%]      644.139       52.911
FNEXRAD                 130.943    [  0.383%]      155.958      104.156
NIMAGE                   98.102    [  0.287%]      146.058      124.044
UNIWISC                  96.527    [  0.282%]      143.653       49.422
IDS|DDPLUS               76.936    [  0.225%]       95.393    47044.200
EXP                       4.067    [  0.012%]       17.561       31.400
LIGHTNING                 0.416    [  0.001%]        0.917       59.111

  If there was enough memory (e.g., 96 GB or more), we would recommend that
  the LDM queue size be increased the to about 60 GB.  If it is only
  possible to increase titan's RAM to 64 GB, the LDM queue size could be
  increased to 35 GB.  Either of these would likely have beneficial
  effects. Our motto is "the more RAM the better" :-)

- it was verified that the Ethernet interface (em1) on titan is running at
  1 Gbps (1000 Mbps) (ethtool em1 | grep Speed)

  This was checked since we have seen some institutions running their
  Ethernet interface at 100 Mbps, and this is not fast enough to get
  all of the data desired.  The fact that the Ethernet interface is
  being run at 1 Gbps tells us that the inability to get all of the FSL2
  HRRR data in a more timely manner in a single REQUEST would not be
  solved by installing a higher speed (e.g., 10 Gbps) Ethernet interface,
  and that the limiting factor is likely the network bandwidth available
  to titan.

Our recommendations:

- install more RAM in titan and then

- further increase the LDM queue size

  Exactly how much will depend on how much more RAM can be added to
  titan.

- meet with SJSU network folks to see if there is any way more network
  resources could be made available to titan

  If the available network bandwidth is limited, then we recommend:

  - reducing the set of data being REQUESTed on titan

    Things to consider are:

    Are all of the products in the FSL2 HRRR feed being used?

    Are all of the products in the CONDUIT feed being used?

    Are all of the products in the NEXRAD2 and NEXRAD3 feeds being used?

    Are all of the products in the NGRID, FNMOC and HDS feeds being
    used?

  The volume of data in the other feeds being REQUESTed is relatively
  small, so there is not much to be gained by restricting what is
  REQUESTed in those feeds.

Final comments:

- we realize that the information contained above is pretty dense and
  so may be hard to wrap one's mind around all at once

  Please send any/all questions that occur to you, and we will try to
  be more clear.

- titan is a decent machine

  titan can be used effectively by itself or in combination with new
  equipment to be purchased under the Unidata equipment grant that SJSU
  was awarded.

- no "heroic" effort is needed to configure any new machine to be able
  to handle the data that is desired

  By "heroic", we mean that switching to use of SSDs is probably not needed.

- one of the best expenditures of money when purchasing new or upgrading
  old equipment is to buy more RAM

  More RAM in titan would allow for the LDM queue size to be increased,
  and this should have the beneficial effect of decreasing the volume
  of data to be processed (by LDM rejection of duplicate products), and
  this, in turn, will lower the impact (e.g., I/O wait) on the file system.

  As a comparison, the machines we are running here in Unidata that receive
  and process ALL of the data available in the IDD are configured with a
  minumum of 192 GB of RAM.  Those machines are also running the ZFS
  file system, but we believe that current implementations of XFS should
  also work very well.


Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: MIW-275261
Department: Support LDM
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.