[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[THREDDS #IXX-362335]: Urgent: UMASS Production Tomcat/THREDDS server shut down due to flood of DNS requests
- Subject: [THREDDS #IXX-362335]: Urgent: UMASS Production Tomcat/THREDDS server shut down due to flood of DNS requests
- Date: Mon, 21 Apr 2014 15:12:04 -0600
Hi Rich, all,
Nothing is jumping out at me. Do you have the access log and threddsServlet.log
files for the time period when the DNS requests were being generated?
The only /tmp file listed below that looks legit (i.e., likely produced by the
TDS) is the hsperfdata_tomcat/ directory (more below [1]). The only other one
that seems like it might be OK is the httpdlog file. But even that seems a bit
odd unless you are running an Apache server as the tomcat user. Is the tomcat
user only used to run Tomcat instances running TDS?
[1] The hsperfdata_tomcat/ directory is the only one from the file listing
below that we have on our main server. It, according to [2], is created by Java
and used to allow/improve monitoring capabilities with jconsole and the like.
[2]
http://stackoverflow.com/questions/76327/how-can-i-prevent-java-from-creating-hsperfdata-files
> Thredds guys,
>
> UMASS shutdown their production tomcat/thredds and disabled the tomcat
> user on Saturday, which of course is causing an interuption in ocean
> forecast products in New England used by the US Coast Guard, US IOOS
> and the local Weather Service Offices.
>
> Here is there message about why they shut it down.
>
> Any ideas about what was happening and how to get this back up and running?
>
> From Kent Gardner at UMASSD:
>
> It appears that the SMAST host system that is running Thredds was
> generating a storm of DNS requests to our campus name server. When
> Mike shut Thredds down and disabled the tomcat account the storm
> stopped.
>
> I can think of no legitimate reason why Thredds would be doing this.
> The only thing that remotely comes to mind would be someone trying to
> look up IP numbers in a log file to get the host name for
> informational purposes. Has anyone come across this behavior before in
> Thredds/Tomcat?
>
> Also looking in /tmp we see the following:
>
> ls -al /tmp|grep tomcat
>
> drwxr-xr-x 2 tomcat tomcat 4096 Apr 15 19:42 adiandian
>
> -rwxr-xr-x 1 tomcat tomcat 5 Apr 18 13:11 bill.lock
>
> drwxr-xr-x 3 tomcat tomcat 4096 Apr 11 14:32 dEDVea
>
> drwxr-xr-x 3 tomcat tomcat 4096 Apr 14 10:30 dvcdNo
>
> drwxr-xr-x 3 tomcat tomcat 4096 Apr 8 09:35 fkuQAx
>
> -rwxr-xr-x 1 tomcat tomcat 5 Apr 18 13:11 gates.lock
>
> drwxr-xr-x 2 tomcat tomcat 4096 Apr 18 21:52
> hsperfdata_tomcat
>
> drwxr-xr-x 2 tomcat tomcat 4096 Mar 28 23:59 httpdlog
>
> --wx--Sr-- 1 tomcat tomcat 51 Apr 16 11:46 notify.file
>
>
> I do not know of any files that Thredds/Tomcat would put in /tmp. Does
> anyone know if any of these files are legitimate?
>
> As far a game plan goes I will need to confer with Mike. At the very
> least we need to scan for and delete all suspicious files, and change
> the password on the tomcat account. After that we start things up and
> monitor the network traffic. "
>
> Thanks,
> Rich
>
> --
> Dr. Richard P. Signell
Ticket Details
===================
Ticket ID: IXX-362335
Department: Support THREDDS
Priority: Normal
Status: Open