[NOAAPORT #GGP-872890]: New Noaaport server with ldm

Hi Heather,

re:
> Thanks for looking into this!

No worries.  Since we will be moving some of our services/computing to CentOS 7 
(or 8 or 9)
coming up, I figured it would be a good thing for us to try and figure out a 
perplexing
problem... and help you out, of course.

re:
> I can't believe that the problem is this crazy!

I came to the conclusion yesterday morning that there seems to be something 
strangely
wrong with your OS installation.  Why did I come to this conclusion?  Aside 
from all
of the things that we tried that _should_ have worked but didn't, the following 
made
me come to my conclusion:

- we researched the 'bad length' messages being shown by 'tcpdump -i eno2' when
  run on your system, and verified that it should have been the case that
  your version of 'tcpdump' should have _NOT_ shown this error/warning/message

  If you Google the problem, you will find a lot of discussion of how this
  message is normal IF the capture size of 'tcpdump' is less than the packet
  size of the packets it is showing.  When we ran 'tcpdump -c 10 -i eth2'
  on one of our NOAAPort ingesters where one of our Novra S300Ns is plugged
  into eth2, we saw that its capture size, 65535, is more than large enough
  to handle the size of the UDP packets being send by the Novra.  Figuring
  that the capture size for the 'tcpdump' on your machine was, for some
  inexplicable reason' smaller than 4K, I VPNed in and ran some tests on
  Saturday morning.  Much to my surprise, the indicated capture size from
  your 'tcpdump' is even larger than for ours, so you should NOT see the
  'bad length' messages.  I even tried setting the capture size using the
  '-s' flag to set the capture size to match ours, but this had no effect
  on the 'tcpdump' output.

- while on your machine on Saturday morning, I made sure (yes, yet again)
  that firewalld AND iptables were not active, and I still couldn't get
  the LDM noaaportIngester instances (run from EXEC lines in the LDM
  configuration file, ~ldm/etc/ldmd.conf) to read from the 'eno2' interface

  This is just bizarre given that the LDM is built correctly and installed
  correctly on your machine.

- third, I am bothered by there first being an 'eth1' interface on your
  machine and then it changing to 'eno2'

  Yes, I know that the interface was known as 'eth1' probably because you
  brought over an Ethernet configuration file

  /etc/sysconfig/network-scripts/ifcfg-eth1

  from your previous machine, but it still did not make any sense to me.
  Not making sense is likely me not understanding everything, but...

I sent a note off to our system administrator and LDM developer reviewing
my latest tests, and made the comment that I think that there is something
amiss with your OS installation, and I asked if they agree or not.  Eventually
after running additional tests, our LDM developer said that the strange things
we are running into could be a problem with the OS installation.  I have not
heard back from our system administrator yet, so I can't say what his opinion
is.

re:
> I am in office on Monday.  Is there anything that I should look at with the
> connection from my novra to the server?  Should I try and disconnect and 
> reconnect?

I do not think that the 'bad length' messages are being caused by either the
Novra or its physical connection to your machine.

re:
> Let me know.
> I am also open to doing a rebuild to a lower Centos-7 or back to Centos-6 if 
> need be.

I have one more thing that I want to try in the 'eno2' configuration file,
/etc/sysconfig/network-scripts/ifcfg-eno2, before I finally cry "uncle".
I will send you a follow-up email after this test has been run.  If the test
fails to produce anything useful, I will be voting for a re-installation of
the OS.  Whether or not you choose to use a slightly older version of CentOS
7, or drop back to CentOS 6 will be up to you.

One last comment IF we get to the point of an OS re-installation: I would _not_
try to apply OS changes from your previous ingest machine.  Rather, I would
try to work with a clean/pristine installation making the few configuration
changes (e.g., routing, firewall, kernel tuning) in a step by step process
with copious amounts of testing in between each step.

I'm sending this off now before running the last test because as soon as
I VPN into your machine, I will lose the ability to send emails from my
UCAR email account.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: GGP-872890
Department: Support NOAAPORT
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.