[LDM #MIY-170840]: One of our two LDM servers is having problems

Hi Tom,

re: LDM configuration files requested
> Here are the files you requested.

Thanks for sending the ldmd.conf, pqact.conf and registry.xml files
from colaweb.gmu.edu.  I'm reviewing them now to see if anything
jumps out at me.

re:
> I saw there is a netcheck command in etc, so I ran it with out data
> providers. All I can tell is that we can reach both hosts. And the
> ping times seem reasonable.

Thanks.  Ping times to upstream LDM feed hosts being low do not
necessarily indicate that data will flow with low latencies (the
condition is necessary, but not sufficient).

Brian wrote:
> I have a question regarding the way the LDM interacts with the OS;
> For the DDS data feed, the report headers are parsed and then the
> reports are put into various files based on the header and the
> date/time stamp.
> 
> These files -- are they kept open ...or are they opened, written
> to, and then closed?

The answer would depend on whether or not you include the '-close'
option on the LDM FILE action(s) that are filing the products.
A quick review of the LDM pattern-action file, ~ldm/etc/pqact.conf,
that Tom sent along yesterday evening shows that the '-close' option
is not included on any of the FILE/STDIOFILE actions.  In this case,
the files will be kept open by the OS and only closed when 'pqact'
runs out of file descriptors and needs to close old ones.   When
the OS will flush pending data writes to disk, is totally dependent
on the OS, not the LDM.

Here is the current METAR processing action from the pattern-action
file that Tom sent:

#  All metars in files by day and hour.
IDS|DDPLUS
        ^SA.... .... (..)(..).*
        FILE    data/dds/sa/(\1:yyyy)(\1:mm)\1\2.sa

One could force the products to be flushed to the file by including
the '-flush' option on the FILE action.  For example:

#  All metars in files by day and hour.
IDS|DDPLUS
        ^SA.... .... (..)(..).*
        FILE  -flush  data/dds/sa/(\1:yyyy)(\1:mm)\1\2.sa

In a similar way, one can force the file to be closed after
each write (and this would also cause the product to be
written to the file):

#  All metars in files by day and hour.
IDS|DDPLUS
        ^SA.... .... (..)(..).*
        FILE  -close  data/dds/sa/(\1:yyyy)(\1:mm)\1\2.sa

NB: the white space between FILE and its option (e.g., '-close')
must be a tab, not spaces.

If you decide to alter your pattern-action file actions, you
should always do a gross check to make sure that there are no
obvious syntax errors by running:

ldmadmin pqactcheck

You can cause the newly modified action(s) to become active
(after running the check!) by running:

ldmadmin pqactHUP

You can see all of the things that 'ldmadmin' supports by 
simply running:

ldmadmin

Comments on files sent by Tom yesterday evening:

- I see nothing wrong with the REQUEST settings in ldmd.conf:

REQUEST IDS|DDPLUS ".*" idd.meteo.psu.edu
REQUEST IDS|DDPLUS ".*" idd.unidata.ucar.edu
REQUEST FNEXRAD "^radar_mosaic" idd.meteo.psu.edu
REQUEST FNEXRAD "^radar_mosaic" idd.unidata.ucar.edu
REQUEST NIMAGE " TIG[NE]" idd.meteo.psu.edu
REQUEST NIMAGE " TIG[NE]" idd.unidata.ucar.edu

  I would expect that on any machine that has any sort of reasonable
  network connection (and by reasonable I mean really low bandwidth)
  would receive all of the products REQUESTed with no problems.  As
  a comparison, I routinely REQUEST LOTS (100x) more data in the LDM
  I have setup on my Odroid U3 single board computer that is running
  Lbuntu 14.04 LTS (it is like a Raspberry Pi) and is connected to
  the Internet by a slow (~5 Mbps) wireless connection.

- I see nothing obviously wrong with colaweb's LDM pattern-action file,
  pqact.conf

- I see nothing wrong with the default LDM queue configuration in colaweb's
  LDM registry:

  <queue>
    <path>/home/ldm/var/queues/ldm.pq</path>
    <size>500M</size>
    <slots>default</slots>
  </queue>

Given that nothing seems untoward in colaweb's LDM configuration files, my
suspicion returns back to colaweb's network connection.

Questions:

- are there any quality of service (QoS) setups on the firewall either/both
  on colaweb or on the department/college/university firewalls?

- it is possible (edge case) that there is something wrong with the
  Ethernet adapter that LDM data is flowing into on colaweb

  I don't think that this is likely because you (Jennifer) indicated that
  colaweb's OS was upgraded from CentOS 5 to CentOS 7, and the LDM was
  upgraded from 6.7.0 to 6.13.6.  Although not explicitly stated, the
  inference was that the old LDM running in the old OS on the same hardware
  was working correctly, but now things are not working.  If the inference
  is true, it should be the case that the network connection to colaweb
  is OK (meaning sufficient to transfer the very low volume of data that
  is being REQUESTed), and that the Ethernet interface is OK.  This leaves
  something in the OS as being the culprit, but, for the life of me, I
  can't imagine what this would/could be!

We will get together to discuss your situation today and get back to you
with any ideas/requests for tests that we can come up with.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: MIY-170840
Department: Support LDM
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.