20030618: LSU throttling HDS feed to ULM? (cont.)

>From: address@hidden
>Organization: ULM
>Keywords: 200306161954.h5GJs2Ld016710 LDM-6 IDD

Adam and Bob,

re:

>The not being able to traceroute to tornado is our firewall.  Or security guy 
>has all but only a hand full of computer even able to be pinged from the 
>outside and he has disabled the traceroute function all together.  Tornado is 
>suppose to be one of them but he might of accidently reset it not to ping.  I 
>will correct this Thursday morring.  Also, finally tornado has been up for a 
>couple of days.  Had a overheating issue from a failed fan that failed to set 
>off a warning(basically didn't know about it untill i cracked open the case). 
>Anyways, that seems to have been fixed.  Apparently LDM-6 (from what unidata 
>said) processes faster so in turn generated more heat.  Anyways, tornado seems
>to be doing good for now so let's see what happens in the next couple of days.

A couple of things here:

- LDM-6 is much more efficient at delivering data even at long distances
  (hence our ability to deliver real time data to Brazil and beyond).
  The consequence of this is that the decoding processes on the
  receiving machine get a lot more data in a shorter period of time to
  work on.  The net effect is that the machine works harder and
  produces more heat.  Our theory had been that CPU fans on tornado
  were either not turning fast enough or were not working at all.
  Adam's opening of the case revealed that his Dell does not have CPU
  fans (a little strange for a dual 900 Mhz PIII machine), but that the
  front case fan was not working.  Monitoring of the CPU temperature
  after the case was opened has shown that the CPUs have been operating
  within tolerances so far.  The jury is still out on whether or not
  overheating was the sole cause for the system lock-ups that tornado
  was experiencing while running LDM-6, but we feel confident that
  the problem is not in LDM-6 (since there are a number of sites
  running orders of magnitude more data through their RH 7.x Linux
  machines that are running LDM-6).

- even though data ingestion is now working well on tornado, it is
  currently feeding HDS data (NOAAPORT model output) from CU/CIRES
  (rainbow.al.noaa.gov).  The reason we switched the feed to rainbow
  was the latencies from seistan were consistently climbing to 5000
  seconds after a model run was made available for transfer.  Evidence
  shows that seistan (LSU in general?) appears to be able to
  effectively deliver data in streams that have modest volumes (like
  IDS|DDPLUS, and UNIWISC) but it is unable to deliver streams with
  higher volumes (e.g., HDS) at least to ULM.  We will be setting up a
  feed of HDS data to a machine here at Unidata from seistan to see if
  the problem is really at LSU (firewall, packet shaping, etc.).  If we
  can reliably receive the HDS feed with little to no latencies (as
  should be the case given the very short traceroute/mtr/ldmping times
  from UCAR to LSU), then the volume limiting would most likely be
  something at ULM.  If our feed experience from LSU matches the ULM
  HDS case, the problem is most likely at LSU.

Adam,

After watching the real time stats from tornado, I noticed that its
clock was drifting unacceptably.  I took the liberty of adjusting the
'root' crontab entry for ntpdate to run once per hour instead of every
twelve hours.  I also switched the time server being used from thelma
to timeserver.unidata.ucar.edu.  The clock now appears to be tracking
nicely.

Bob,

Hopefully, you can adjust the IP chains/tcpd setup on seistan to allow
port 388 access for the list of test machines we would like to try
feeding HDS data from:

*.unidata.ucar.edu         <- any machine in the unidata.ucar.edu domain
motherlode.ucar.edu        <- Unidata operated
thelma.ucar.edu            <- Unidata operated
atm.geo.nsf.gov            <- Unidata operated
metlab.cas.usf.edu         <- I have full access to this machine
aeolus.ucsd.edu            <- a new addition since the list I sent last night
atmos.uprm.edu             <- a new addition since last night; I have full
                              access
brisa.meteoro.ufrj.br      <- a new addition since last night; I have full
                              access

I personally would make port 388 open to all machines, at least until
testing is done.

Thanks in advance!

Tom

From address@hidden Thu Jun 19 07:36:06 2003
Received: from seistan.srcc.lsu.edu (seistan.srcc.lsu.edu [130.39.188.204])
        by unidata.ucar.edu (UCAR/Unidata) with ESMTP id h5JDa6Ld023387
        for <address@hidden>; Thu, 19 Jun 2003 07:36:06 -0600 (MDT)
Organization: UCAR/Unidata
Keywords: 200306191336.h5JDa6Ld023387
Received: from srcc.lsu.edu (sirocco.srcc.lsu.edu [130.39.188.220])
        (authenticated)
        by seistan.srcc.lsu.edu (8.11.6/8.11.6) with ESMTP id h5JDa3905709;
        Thu, 19 Jun 2003 08:36:03 -0500
Message-ID: <address@hidden>
Date: Thu, 19 Jun 2003 08:36:03 -0500
From: Robert Leche <address@hidden>
Reply-To: address@hidden
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.0.1) Gecko/20021003
X-Accept-Language: en-us, en
MIME-Version: 1.0
To: address@hidden, Unidata Support <address@hidden>
Subject: Re: 20030613: LSU throttling HDS feed to ULM?
References: <address@hidden> <address@hidden> <address@hidden>
Content-Type: text/html; charset=us-ascii
Content-Transfer-Encoding: 7bit

Hello Adam,

Glad to hear Tornado is up and running.

Security is always a compromise to functionality.  One point I will
offer on firewalls and networking..... While talking the issue over
with your firewall administrator, suggest  port 388 service to the
following, ping service to the following , and traceroute service to
the following:

Seistan.srcc.lsu.edu
Sirocco.srcc.lsu.edu
Datoo.srcc.lsu.edu

This will allow you quick changes to your LDM feed and yet allow us
connectivity tests.

Bob


Bob,

The not being able to traceroute to tornado is our firewall.  Or
security guy has all but only a hand full of computer even able to be
pinged from the outside and he has disabled the traceroute function all
together.  Tornado is suppose to be one of them but he might of
accidently reset it not to ping.  I will correct this Thursday
morring.  Also, finally tornado has been up for a couple of days.  Had
a overheating issue from a failed fan that failed to set off a
warning(basically didn't know about it untill i cracked open the
case).  Anyways, that seems to have been fixed.  Apparently LDM-6 (from
what unidata said) processes faster so in turn generated more heat.
Anyways, tornado seems to be doing good for now so let's see what
happens in the next couple of days.

Adam Taylor
University of Louisiana at Monroe

ps.  I am forwarding this to unidata also.



Quoting Robert Leche <a class="moz-txt-link-rfc2396E" 
href="mailto:address@hidden";>&lt;address@hidden&gt;</a>:

  </pre>
  <blockquote type="cite">
    <pre wrap="">

  


Steve

I just got off the phone with LSU's network people. And found out  LSU
is not placing any special traffic shaping controls on LDM port 388 nor
any of the hosts we communicate with. In fact, looking at the flow of
traffic places our flow in to and out of the same networking. Meaning,
The data that feeds Jackson State U, also feeds ULM. Same with getting
data from Unidata.  Looks like we are all I2 users. We are not and have
not had  network problems to JSUMS, only ULM. No problems receiving
data from Thelma either.  As a matter of fact, ULM is unreachable
today. The problem seems to be on the ULM campus as this trace route
indicates:



[ldm@seistan ~]$ /usr/sbin/traceroute tornado.geos.ulm.edu

traceroute to tornado.geos.ulm.edu (198.202.242.22), 30 hops max, 38 byte
packets

 1  130.39.188.1 (130.39.188.1)  10.500 ms  13.884 ms  3.006 ms
 2  lsubr1-118-6509-dsw-1.g1.lsu.edu (130.39.1.20)  1.719 ms  2.896 ms 1.346 ms
 3  laNoc-lsubr.LEARN.la.net (162.75.0.9)  7.790 ms  3.387 ms  2.874 ms
 4  ulm-laNoc.LEARN.la.net (162.75.0.38)  16.607 ms  15.101 ms  15.425 ms
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  *


I am  not able to ping the ldm host.   I will contact Adam Taylor at
ULM and let him know. Also, when communications are restored, our
network people indicated a willingness to monitor the connections to
ULM (for a short period),
looking for problems.


Bob

Emmerson wrote:


Bob,

Date: Wed, 18 Jun 2003 11:23:58 -0500
From: Robert Leche <address@hidden>
To: Steve Emmerson <address@hidden>
Subject: Re: 20030613: LSU throttling HDS feed to ULM?


The above message contained the following:

I want to  let you know that I have not  ignored  your request for 
information on LSU's network traffic shaping or throttling (If this, 
indeed, exists on campus).

I am still waiting to hear from the right people on this issue.  

Thanks.  I appreciate it.

Regards,
Steve Emmerson

  





-- 
----------------------------------------------------------------
Robert Leche
System Administrator
Louisiana State University - Southern Regional Climate Center
260 Howe-Russell Building
Baton Rouge, La. 70803
<a class="moz-txt-link-abbreviated" 
href="mailto:address@hidden";>address@hidden</a>
225 578 5023
----------------------------------------------------------------







    </pre>
  </blockquote>
  <pre wrap=""><!---->

  </pre>
</blockquote>


<pre class="moz-signature" cols="$mailwrapcol">-- 
----------------------------------------------------------------
Robert Leche
System Administrator
Louisiana State University - Southern Regional Climate Center
260 Howe-Russell Building
Baton Rouge, La. 70803
<a class="moz-txt-link-abbreviated" 
href="mailto:address@hidden";>address@hidden</a>
225 578 5023
----------------------------------------------------------------
</pre>

</body>
</html>