[THREDDS #IXX-362335]: Urgent: UMASS Production Tomcat/THREDDS server shut down due to flood of DNS requests

Rich,

Is there a competent sys admin for this TDS instance?  I'm not really a 
security expert, but yet, upgrading OpenSSL to eliminate the 
vulnerability would seem to be sufficient.  I can't speak to the 
necessity of re-compiling the APR or Tomcat Native library. Checking 
against a service that identifies vulnerabilities would seem to be a 
good idea.

Turning off OpenSSL and remote access (i.e., "locking down" the server) 
is probably still the safest bet.  This is done in the 
{$tomcat_home}/conf/server.xml file.

-Lansing

On 4/22/2014 9:40 AM, Signell, Richard wrote:
> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut down 
> due to flood of DNS requests
>
> I saw this comment:
>
> "Confirmed my app was vulnerable according to filippo.io/Heartbleed
> After upgrading OpenSSL, simply restarting Tomcat eliminated the
> vulnerability. Re-compiling APR or Tomcat Native was not necessary"
>
> Do you agree that this simple approach is sufficient?
>
> -Rich
>
> On Tue, Apr 22, 2014 at 11:35 AM, Unidata THREDDS Support
> <address@hidden> wrote:
>> My experience is with Windows, since that is what I have.  I'll try to
>> find out what is included in downloads for other o/s. Information
>> directly from apache is here:
>>
>> https://issues.apache.org/bugzilla/show_bug.cgi?id=56363
>>
>> and a discussion on security.stackexchange is here:
>>
>> http://security.stackexchange.com/questions/55139/does-the-heartbleed-vulnerability-affect-apache-tomcat-servers-using-tomcat-nati
>>
>> -Lansing
>>
>> On 4/22/2014 9:31 AM, Signell, Richard wrote:
>>> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut down 
>>> due to flood of DNS requests
>>>
>>> Does this pertain to Tomcat/THREDDS on Windows machines only, or all 
>>> machines?
>>>
>>> On Tue, Apr 22, 2014 at 11:28 AM, Unidata THREDDS Support
>>> <address@hidden> wrote:
>>>> Hi Rich,
>>>>
>>>> We noticed that with the most recent jdk and tomcat downloads (1.7u55
>>>> and 7u53, respectively), our Windows machines had an old SSL, as
>>>> evidenced here:
>>>>
>>>> Apr 21, 2014 2:34:22 PM org.apache.catalina.core.AprLifecycleListener 
>>>> initializeSSL
>>>> INFO: OpenSSL successfully initialized (OpenSSL 1.0.1e 11 Feb 2013)
>>>>
>>>> Clearly, the OpenSSL library predates the Heartbleed fix.  We asked Jen 
>>>> about this, and her advice was to keep the APRLifecycleListener commented 
>>>> out (as we have on our production machines).  This avoids the 
>>>> vulnerability until Apache Tomcat updates.
>>>>
>>>> -Lansing
>>>>
>>>> On 4/22/2014 9:24 AM, Signell, Richard wrote:
>>>>> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut 
>>>>> down due to flood of DNS requests
>>>>>
>>>>> Are there things we should do to respond to Heartbleed on public
>>>>> THREDDS servers?
>>>>>
>>>>> For folks running public IPython notebook servers, we received this
>>>>> advice:"If you run a public #IPython notebook server over HTTPS, shut
>>>>> it down, upgrade OpenSSL, regenerate your TLS certificate & restart."
>>>>>
>>>>> Does the same apply to tomcat/thredds?
>>>>>
>>>>> -Rich
>>>>>
>>>>> On Mon, Apr 21, 2014 at 5:12 PM, Unidata THREDDS Support
>>>>> <address@hidden> wrote:
>>>>>> Hi Rich, all,
>>>>>>
>>>>>> Nothing is jumping out at me. Do you have the access log and 
>>>>>> threddsServlet.log files for the time period when the DNS requests were 
>>>>>> being generated?
>>>>>>
>>>>>> The only /tmp file listed below that looks legit (i.e., likely produced 
>>>>>> by the TDS) is the hsperfdata_tomcat/ directory (more below [1]). The 
>>>>>> only other one that seems like it might be OK is the httpdlog file. But 
>>>>>> even that seems a bit odd unless you are running an Apache server as the 
>>>>>> tomcat user. Is the tomcat user only used to run Tomcat instances 
>>>>>> running TDS?
>>>>>>
>>>>>>
>>>>>> [1] The hsperfdata_tomcat/ directory is the only one from the file 
>>>>>> listing below that we have on our main server. It, according to [2], is 
>>>>>> created by Java and used to allow/improve monitoring capabilities with 
>>>>>> jconsole and the like.
>>>>>>
>>>>>> [2] 
>>>>>> http://stackoverflow.com/questions/76327/how-can-i-prevent-java-from-creating-hsperfdata-files
>>>>>>
>>>>>>> Thredds guys,
>>>>>>>
>>>>>>> UMASS shutdown their production tomcat/thredds and disabled the tomcat
>>>>>>> user on Saturday, which of course is causing an interuption in ocean
>>>>>>> forecast products in New England used by the US Coast Guard, US IOOS
>>>>>>> and the local Weather Service Offices.
>>>>>>>
>>>>>>> Here is there message about why they shut it down.
>>>>>>>
>>>>>>> Any ideas about what was happening and how to get this back up and 
>>>>>>> running?
>>>>>>>
>>>>>>>    From Kent Gardner at UMASSD:
>>>>>>>
>>>>>>> It appears that the SMAST host system that is running Thredds was
>>>>>>> generating a storm of DNS requests to our campus name server. When
>>>>>>> Mike shut Thredds down and disabled the tomcat account the storm
>>>>>>> stopped.
>>>>>>>
>>>>>>> I can think of no legitimate reason why Thredds would be doing this.
>>>>>>> The only thing that remotely comes to mind would be someone trying to
>>>>>>> look up IP numbers in a log file to get the host name for
>>>>>>> informational purposes. Has anyone come across this behavior before in
>>>>>>> Thredds/Tomcat?
>>>>>>>
>>>>>>> Also looking in /tmp we see the following:
>>>>>>>
>>>>>>> ls -al /tmp|grep tomcat
>>>>>>>
>>>>>>> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 15 19:42 adiandian
>>>>>>>
>>>>>>> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 bill.lock
>>>>>>>
>>>>>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 11 14:32 dEDVea
>>>>>>>
>>>>>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 14 10:30 dvcdNo
>>>>>>>
>>>>>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr  8 09:35 fkuQAx
>>>>>>>
>>>>>>> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 
>>>>>>> gates.lock
>>>>>>>
>>>>>>> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 18 21:52 
>>>>>>> hsperfdata_tomcat
>>>>>>>
>>>>>>> drwxr-xr-x   2 tomcat       tomcat           4096 Mar 28 23:59 httpdlog
>>>>>>>
>>>>>>> --wx--Sr--   1 tomcat       tomcat             51 Apr 16 11:46 
>>>>>>> notify.file
>>>>>>>
>>>>>>>
>>>>>>> I do not know of any files that Thredds/Tomcat would put in /tmp. Does
>>>>>>> anyone know if any of these files are legitimate?
>>>>>>>
>>>>>>> As far a game plan goes I will need to confer with Mike. At the very
>>>>>>> least we need to scan for and delete all suspicious files, and change
>>>>>>> the password on the tomcat account. After that we start things up and
>>>>>>> monitor the network traffic. "
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Rich
>>>>>>>
>>>>>>> --
>>>>>>> Dr. Richard P. Signell
>>>>>> Ticket Details
>>>>>> ===================
>>>>>> Ticket ID: IXX-362335
>>>>>> Department: Support THREDDS
>>>>>> Priority: Normal
>>>>>> Status: Open
>>>>>>
>>>>
>>>> Ticket Details
>>>> ===================
>>>> Ticket ID: IXX-362335
>>>> Department: Support THREDDS
>>>> Priority: Normal
>>>> Status: Open
>>>>
>>>
>>
>>
>> Ticket Details
>> ===================
>> Ticket ID: IXX-362335
>> Department: Support THREDDS
>> Priority: Normal
>> Status: Open
>>
>
>



Ticket Details
===================
Ticket ID: IXX-362335
Department: Support THREDDS
Priority: Normal
Status: Open