[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[THREDDS #IXX-362335]: Urgent: UMASS Production Tomcat/THREDDS server shut down due to flood of DNS requests
- Subject: [THREDDS #IXX-362335]: Urgent: UMASS Production Tomcat/THREDDS server shut down due to flood of DNS requests
- Date: Tue, 22 Apr 2014 09:46:56 -0600
Rich,
Is there a competent sys admin for this TDS instance? I'm not really a
security expert, but yet, upgrading OpenSSL to eliminate the
vulnerability would seem to be sufficient. I can't speak to the
necessity of re-compiling the APR or Tomcat Native library. Checking
against a service that identifies vulnerabilities would seem to be a
good idea.
Turning off OpenSSL and remote access (i.e., "locking down" the server)
is probably still the safest bet. This is done in the
{$tomcat_home}/conf/server.xml file.
-Lansing
On 4/22/2014 9:40 AM, Signell, Richard wrote:
> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut down
> due to flood of DNS requests
>
> I saw this comment:
>
> "Confirmed my app was vulnerable according to filippo.io/Heartbleed
> After upgrading OpenSSL, simply restarting Tomcat eliminated the
> vulnerability. Re-compiling APR or Tomcat Native was not necessary"
>
> Do you agree that this simple approach is sufficient?
>
> -Rich
>
> On Tue, Apr 22, 2014 at 11:35 AM, Unidata THREDDS Support
> <address@hidden> wrote:
>> My experience is with Windows, since that is what I have. I'll try to
>> find out what is included in downloads for other o/s. Information
>> directly from apache is here:
>>
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=56363
>>
>> and a discussion on security.stackexchange is here:
>>
>> http://security.stackexchange.com/questions/55139/does-the-heartbleed-vulnerability-affect-apache-tomcat-servers-using-tomcat-nati
>>
>> -Lansing
>>
>> On 4/22/2014 9:31 AM, Signell, Richard wrote:
>>> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut down
>>> due to flood of DNS requests
>>>
>>> Does this pertain to Tomcat/THREDDS on Windows machines only, or all
>>> machines?
>>>
>>> On Tue, Apr 22, 2014 at 11:28 AM, Unidata THREDDS Support
>>> <address@hidden> wrote:
>>>> Hi Rich,
>>>>
>>>> We noticed that with the most recent jdk and tomcat downloads (1.7u55
>>>> and 7u53, respectively), our Windows machines had an old SSL, as
>>>> evidenced here:
>>>>
>>>> Apr 21, 2014 2:34:22 PM org.apache.catalina.core.AprLifecycleListener
>>>> initializeSSL
>>>> INFO: OpenSSL successfully initialized (OpenSSL 1.0.1e 11 Feb 2013)
>>>>
>>>> Clearly, the OpenSSL library predates the Heartbleed fix. We asked Jen
>>>> about this, and her advice was to keep the APRLifecycleListener commented
>>>> out (as we have on our production machines). This avoids the
>>>> vulnerability until Apache Tomcat updates.
>>>>
>>>> -Lansing
>>>>
>>>> On 4/22/2014 9:24 AM, Signell, Richard wrote:
>>>>> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut
>>>>> down due to flood of DNS requests
>>>>>
>>>>> Are there things we should do to respond to Heartbleed on public
>>>>> THREDDS servers?
>>>>>
>>>>> For folks running public IPython notebook servers, we received this
>>>>> advice:"If you run a public #IPython notebook server over HTTPS, shut
>>>>> it down, upgrade OpenSSL, regenerate your TLS certificate & restart."
>>>>>
>>>>> Does the same apply to tomcat/thredds?
>>>>>
>>>>> -Rich
>>>>>
>>>>> On Mon, Apr 21, 2014 at 5:12 PM, Unidata THREDDS Support
>>>>> <address@hidden> wrote:
>>>>>> Hi Rich, all,
>>>>>>
>>>>>> Nothing is jumping out at me. Do you have the access log and
>>>>>> threddsServlet.log files for the time period when the DNS requests were
>>>>>> being generated?
>>>>>>
>>>>>> The only /tmp file listed below that looks legit (i.e., likely produced
>>>>>> by the TDS) is the hsperfdata_tomcat/ directory (more below [1]). The
>>>>>> only other one that seems like it might be OK is the httpdlog file. But
>>>>>> even that seems a bit odd unless you are running an Apache server as the
>>>>>> tomcat user. Is the tomcat user only used to run Tomcat instances
>>>>>> running TDS?
>>>>>>
>>>>>>
>>>>>> [1] The hsperfdata_tomcat/ directory is the only one from the file
>>>>>> listing below that we have on our main server. It, according to [2], is
>>>>>> created by Java and used to allow/improve monitoring capabilities with
>>>>>> jconsole and the like.
>>>>>>
>>>>>> [2]
>>>>>> http://stackoverflow.com/questions/76327/how-can-i-prevent-java-from-creating-hsperfdata-files
>>>>>>
>>>>>>> Thredds guys,
>>>>>>>
>>>>>>> UMASS shutdown their production tomcat/thredds and disabled the tomcat
>>>>>>> user on Saturday, which of course is causing an interuption in ocean
>>>>>>> forecast products in New England used by the US Coast Guard, US IOOS
>>>>>>> and the local Weather Service Offices.
>>>>>>>
>>>>>>> Here is there message about why they shut it down.
>>>>>>>
>>>>>>> Any ideas about what was happening and how to get this back up and
>>>>>>> running?
>>>>>>>
>>>>>>> From Kent Gardner at UMASSD:
>>>>>>>
>>>>>>> It appears that the SMAST host system that is running Thredds was
>>>>>>> generating a storm of DNS requests to our campus name server. When
>>>>>>> Mike shut Thredds down and disabled the tomcat account the storm
>>>>>>> stopped.
>>>>>>>
>>>>>>> I can think of no legitimate reason why Thredds would be doing this.
>>>>>>> The only thing that remotely comes to mind would be someone trying to
>>>>>>> look up IP numbers in a log file to get the host name for
>>>>>>> informational purposes. Has anyone come across this behavior before in
>>>>>>> Thredds/Tomcat?
>>>>>>>
>>>>>>> Also looking in /tmp we see the following:
>>>>>>>
>>>>>>> ls -al /tmp|grep tomcat
>>>>>>>
>>>>>>> drwxr-xr-x 2 tomcat tomcat 4096 Apr 15 19:42 adiandian
>>>>>>>
>>>>>>> -rwxr-xr-x 1 tomcat tomcat 5 Apr 18 13:11 bill.lock
>>>>>>>
>>>>>>> drwxr-xr-x 3 tomcat tomcat 4096 Apr 11 14:32 dEDVea
>>>>>>>
>>>>>>> drwxr-xr-x 3 tomcat tomcat 4096 Apr 14 10:30 dvcdNo
>>>>>>>
>>>>>>> drwxr-xr-x 3 tomcat tomcat 4096 Apr 8 09:35 fkuQAx
>>>>>>>
>>>>>>> -rwxr-xr-x 1 tomcat tomcat 5 Apr 18 13:11
>>>>>>> gates.lock
>>>>>>>
>>>>>>> drwxr-xr-x 2 tomcat tomcat 4096 Apr 18 21:52
>>>>>>> hsperfdata_tomcat
>>>>>>>
>>>>>>> drwxr-xr-x 2 tomcat tomcat 4096 Mar 28 23:59 httpdlog
>>>>>>>
>>>>>>> --wx--Sr-- 1 tomcat tomcat 51 Apr 16 11:46
>>>>>>> notify.file
>>>>>>>
>>>>>>>
>>>>>>> I do not know of any files that Thredds/Tomcat would put in /tmp. Does
>>>>>>> anyone know if any of these files are legitimate?
>>>>>>>
>>>>>>> As far a game plan goes I will need to confer with Mike. At the very
>>>>>>> least we need to scan for and delete all suspicious files, and change
>>>>>>> the password on the tomcat account. After that we start things up and
>>>>>>> monitor the network traffic. "
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Rich
>>>>>>>
>>>>>>> --
>>>>>>> Dr. Richard P. Signell
>>>>>> Ticket Details
>>>>>> ===================
>>>>>> Ticket ID: IXX-362335
>>>>>> Department: Support THREDDS
>>>>>> Priority: Normal
>>>>>> Status: Open
>>>>>>
>>>>
>>>> Ticket Details
>>>> ===================
>>>> Ticket ID: IXX-362335
>>>> Department: Support THREDDS
>>>> Priority: Normal
>>>> Status: Open
>>>>
>>>
>>
>>
>> Ticket Details
>> ===================
>> Ticket ID: IXX-362335
>> Department: Support THREDDS
>> Priority: Normal
>> Status: Open
>>
>
>
Ticket Details
===================
Ticket ID: IXX-362335
Department: Support THREDDS
Priority: Normal
Status: Open