DNS status (fwd)

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================

---------- Forwarded message ----------
Date: Fri, 4 Aug 2000 09:22:00 -0600 (MDT)
From: Greg Woods <address@hidden>
To: address@hidden
Subject: DNS status

At approximately 12:22pm Thursday Aug. 3, something caused the named process
on both fl-phx (128.117.98.3) and bubby (128.117.8.94) to crash. Service
on fl-phx was restored shortly after 1400, but it was not noticed that
bubby's named had also crashed until this morning. Service has now been
restored on bubby. It is clear that some sort of "denial of service"
incident occurred, but we don't know if it was accidental (some machine
going nuts and sending many queries in rapid succession) or intentional
(a hacker doing it). These machines are the ones that have a hole in the
security perimeter for DNS, so that they can send queries to external
servers. They depend on named itself to refuse queries that come from
outside, so it is possible that someone outside could have flooded the
DNS port on these machines with garbage. It's also possible that some
machine that normally uses these machines as forwarders accidentally
did something similar. Because only these two servers were affected,
and not other machines that also have exposed DNS servers, I suspect
it was an accident caused by a machine somewhere at FL. We will need 
to keep an eye on this. (A planned upgrade of BIND on these systems
may or may not also prevent the crash from happening).

Normal monitoring procedures on the systems will not detect if a single
daemon goes down but the machine is otherwise still operational, which
is why this took so long to get noticed.

In addition to the above, there is some sort of problem with fl-phx
this morning that has caused the machine to drop into single user mode
(this prevents me from bringing the backup machine online because the
primary machine is still active on that IP address; it can be pinged).
I am still investigating that situation. However, those of you at FL
(inside the security perimeter) who have your DNS configured as
recommended, with divisional servers forwarding first to fl-phx
(128.117.98.3) and then to bubby (128.117.8.94) should not notice
anything amiss as long as bubby's named (and the link to the Mesa) stays up.

I will be installing named monitoring scripts on fl-phx and bubby
as soon as I can.

--Greg