[THREDDS #IXX-362335]: Urgent: UMASS Production Tomcat/THREDDS server shut down due to flood of DNS requests

My experience is with Windows, since that is what I have.  I'll try to 
find out what is included in downloads for other o/s. Information 
directly from apache is here:

https://issues.apache.org/bugzilla/show_bug.cgi?id=56363

and a discussion on security.stackexchange is here:

http://security.stackexchange.com/questions/55139/does-the-heartbleed-vulnerability-affect-apache-tomcat-servers-using-tomcat-nati

-Lansing

On 4/22/2014 9:31 AM, Signell, Richard wrote:
> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut down 
> due to flood of DNS requests
>
> Does this pertain to Tomcat/THREDDS on Windows machines only, or all machines?
>
> On Tue, Apr 22, 2014 at 11:28 AM, Unidata THREDDS Support
> <address@hidden> wrote:
>> Hi Rich,
>>
>> We noticed that with the most recent jdk and tomcat downloads (1.7u55
>> and 7u53, respectively), our Windows machines had an old SSL, as
>> evidenced here:
>>
>> Apr 21, 2014 2:34:22 PM org.apache.catalina.core.AprLifecycleListener 
>> initializeSSL
>> INFO: OpenSSL successfully initialized (OpenSSL 1.0.1e 11 Feb 2013)
>>
>> Clearly, the OpenSSL library predates the Heartbleed fix.  We asked Jen 
>> about this, and her advice was to keep the APRLifecycleListener commented 
>> out (as we have on our production machines).  This avoids the vulnerability 
>> until Apache Tomcat updates.
>>
>> -Lansing
>>
>> On 4/22/2014 9:24 AM, Signell, Richard wrote:
>>> New Client Reply: Urgent: UMASS Production Tomcat/THREDDS server shut down 
>>> due to flood of DNS requests
>>>
>>> Are there things we should do to respond to Heartbleed on public
>>> THREDDS servers?
>>>
>>> For folks running public IPython notebook servers, we received this
>>> advice:"If you run a public #IPython notebook server over HTTPS, shut
>>> it down, upgrade OpenSSL, regenerate your TLS certificate & restart."
>>>
>>> Does the same apply to tomcat/thredds?
>>>
>>> -Rich
>>>
>>> On Mon, Apr 21, 2014 at 5:12 PM, Unidata THREDDS Support
>>> <address@hidden> wrote:
>>>> Hi Rich, all,
>>>>
>>>> Nothing is jumping out at me. Do you have the access log and 
>>>> threddsServlet.log files for the time period when the DNS requests were 
>>>> being generated?
>>>>
>>>> The only /tmp file listed below that looks legit (i.e., likely produced by 
>>>> the TDS) is the hsperfdata_tomcat/ directory (more below [1]). The only 
>>>> other one that seems like it might be OK is the httpdlog file. But even 
>>>> that seems a bit odd unless you are running an Apache server as the tomcat 
>>>> user. Is the tomcat user only used to run Tomcat instances running TDS?
>>>>
>>>>
>>>> [1] The hsperfdata_tomcat/ directory is the only one from the file listing 
>>>> below that we have on our main server. It, according to [2], is created by 
>>>> Java and used to allow/improve monitoring capabilities with jconsole and 
>>>> the like.
>>>>
>>>> [2] 
>>>> http://stackoverflow.com/questions/76327/how-can-i-prevent-java-from-creating-hsperfdata-files
>>>>
>>>>> Thredds guys,
>>>>>
>>>>> UMASS shutdown their production tomcat/thredds and disabled the tomcat
>>>>> user on Saturday, which of course is causing an interuption in ocean
>>>>> forecast products in New England used by the US Coast Guard, US IOOS
>>>>> and the local Weather Service Offices.
>>>>>
>>>>> Here is there message about why they shut it down.
>>>>>
>>>>> Any ideas about what was happening and how to get this back up and 
>>>>> running?
>>>>>
>>>>>   From Kent Gardner at UMASSD:
>>>>>
>>>>> It appears that the SMAST host system that is running Thredds was
>>>>> generating a storm of DNS requests to our campus name server. When
>>>>> Mike shut Thredds down and disabled the tomcat account the storm
>>>>> stopped.
>>>>>
>>>>> I can think of no legitimate reason why Thredds would be doing this.
>>>>> The only thing that remotely comes to mind would be someone trying to
>>>>> look up IP numbers in a log file to get the host name for
>>>>> informational purposes. Has anyone come across this behavior before in
>>>>> Thredds/Tomcat?
>>>>>
>>>>> Also looking in /tmp we see the following:
>>>>>
>>>>> ls -al /tmp|grep tomcat
>>>>>
>>>>> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 15 19:42 adiandian
>>>>>
>>>>> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 bill.lock
>>>>>
>>>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 11 14:32 dEDVea
>>>>>
>>>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 14 10:30 dvcdNo
>>>>>
>>>>> drwxr-xr-x   3 tomcat       tomcat           4096 Apr  8 09:35 fkuQAx
>>>>>
>>>>> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 gates.lock
>>>>>
>>>>> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 18 21:52 
>>>>> hsperfdata_tomcat
>>>>>
>>>>> drwxr-xr-x   2 tomcat       tomcat           4096 Mar 28 23:59 httpdlog
>>>>>
>>>>> --wx--Sr--   1 tomcat       tomcat             51 Apr 16 11:46 notify.file
>>>>>
>>>>>
>>>>> I do not know of any files that Thredds/Tomcat would put in /tmp. Does
>>>>> anyone know if any of these files are legitimate?
>>>>>
>>>>> As far a game plan goes I will need to confer with Mike. At the very
>>>>> least we need to scan for and delete all suspicious files, and change
>>>>> the password on the tomcat account. After that we start things up and
>>>>> monitor the network traffic. "
>>>>>
>>>>> Thanks,
>>>>> Rich
>>>>>
>>>>> --
>>>>> Dr. Richard P. Signell
>>>> Ticket Details
>>>> ===================
>>>> Ticket ID: IXX-362335
>>>> Department: Support THREDDS
>>>> Priority: Normal
>>>> Status: Open
>>>>
>>>
>>
>>
>> Ticket Details
>> ===================
>> Ticket ID: IXX-362335
>> Department: Support THREDDS
>> Priority: Normal
>> Status: Open
>>
>
>



Ticket Details
===================
Ticket ID: IXX-362335
Department: Support THREDDS
Priority: Normal
Status: Open