[THREDDS #IXX-362335]: Urgent: UMASS Production Tomcat/THREDDS server shut down due to flood of DNS requests

Hi Rich, all,

Nothing is jumping out at me. Do you have the access log and threddsServlet.log 
files for the time period when the DNS requests were being generated?

The only /tmp file listed below that looks legit (i.e., likely produced by the 
TDS) is the hsperfdata_tomcat/ directory (more below [1]). The only other one 
that seems like it might be OK is the httpdlog file. But even that seems a bit 
odd unless you are running an Apache server as the tomcat user. Is the tomcat 
user only used to run Tomcat instances running TDS?


[1] The hsperfdata_tomcat/ directory is the only one from the file listing 
below that we have on our main server. It, according to [2], is created by Java 
and used to allow/improve monitoring capabilities with jconsole and the like.

[2] 
http://stackoverflow.com/questions/76327/how-can-i-prevent-java-from-creating-hsperfdata-files

> Thredds guys,
> 
> UMASS shutdown their production tomcat/thredds and disabled the tomcat
> user on Saturday, which of course is causing an interuption in ocean
> forecast products in New England used by the US Coast Guard, US IOOS
> and the local Weather Service Offices.
> 
> Here is there message about why they shut it down.
> 
> Any ideas about what was happening and how to get this back up and running?
> 
> From Kent Gardner at UMASSD:
> 
> It appears that the SMAST host system that is running Thredds was
> generating a storm of DNS requests to our campus name server. When
> Mike shut Thredds down and disabled the tomcat account the storm
> stopped.
> 
> I can think of no legitimate reason why Thredds would be doing this.
> The only thing that remotely comes to mind would be someone trying to
> look up IP numbers in a log file to get the host name for
> informational purposes. Has anyone come across this behavior before in
> Thredds/Tomcat?
> 
> Also looking in /tmp we see the following:
> 
> ls -al /tmp|grep tomcat
> 
> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 15 19:42 adiandian
> 
> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 bill.lock
> 
> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 11 14:32 dEDVea
> 
> drwxr-xr-x   3 tomcat       tomcat           4096 Apr 14 10:30 dvcdNo
> 
> drwxr-xr-x   3 tomcat       tomcat           4096 Apr  8 09:35 fkuQAx
> 
> -rwxr-xr-x   1 tomcat       tomcat              5 Apr 18 13:11 gates.lock
> 
> drwxr-xr-x   2 tomcat       tomcat           4096 Apr 18 21:52 
> hsperfdata_tomcat
> 
> drwxr-xr-x   2 tomcat       tomcat           4096 Mar 28 23:59 httpdlog
> 
> --wx--Sr--   1 tomcat       tomcat             51 Apr 16 11:46 notify.file
> 
> 
> I do not know of any files that Thredds/Tomcat would put in /tmp. Does
> anyone know if any of these files are legitimate?
> 
> As far a game plan goes I will need to confer with Mike. At the very
> least we need to scan for and delete all suspicious files, and change
> the password on the tomcat account. After that we start things up and
> monitor the network traffic. "
> 
> Thanks,
> Rich
> 
> --
> Dr. Richard P. Signell


Ticket Details
===================
Ticket ID: IXX-362335
Department: Support THREDDS
Priority: Normal
Status: Open