[LDM #CSZ-262645]: errant LDM REQUESTs on sirocco.srcc.lsu.edu

Hi Jeffry,

We wanted to thank you for your clear and concise description of the cause
and solution the the firewall-related problem that was interfering with
LSU/SRCC LDMs from reliably getting data from machines outside of the
LSU campus.  The information provided will likely help some other site
down the road!

Thanks again...

Jeffry Handal wrote:
> Summary:
> Issue at LSU were caused by the campus firewall.
> 
> Equipment:
> Juniper SRX-5800 with IDP
> 
> Technical Details of Issue:
> All traffic at LSU is subject to an IDP to block p2p. Since January
> 18th, the IDP was turned off because it was causing high CPU
> utilization, slowing the entire campus traffic. Unfortunately, there was
> a rule that still would point to an inactive IDP. The way this works,
> the flows that match this IDP policy will be asked to redirect to the
> IDP module. They will go into a queue waiting for IDP inspection.
> Because the IDP module is not enabled, they are momentarily stuck there
> waiting for timeout. All this waiting will being to clog up the
> buffer-queue, which eventually triggers the log message we saw on the
> SRX (i.e. Feb 22 14:03:24  csc-118-l7-srx5800-edgefrwl fpc1 Cobar: XLR1
> flow_held_mbuf 500, raise above 500, 1000th time. ). When the queue gets
> full, packets were dropped. Most connections will not see a problem
> because TCP will recover and start a new connection. For other
> applications, this may look like a DoS due to the constant creation of
> new connections, e.g. Unidata.
> 
> LSU future plans:
> * Feature request to Juniper: if IDP is turned off, send a warning
> message or error trigger that a rule is still pointing to the IDP even
> though it is off.
> 
> * We are in the process of creating a Science DMZ that will no be
> subject to a campus firewall.
> 
> Thanks to everyone involved for the support and patience in this
> process. By far, hardware issues are the hardest to troubleshoot; here,
> we went down to the buffer level to diagnose the problem.
> 
> Regards,
> 
> Jeffry J.Handal
> Convergence Specialist, M.S.E.E., P.E.
> University Networking and Infrastructure - LSU
> 200 Computing Services Building
> Baton Rouge, LA 70803
> Office:       (225)578-1966
> Fax:  (225)578-6400

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: CSZ-262645
Department: Support LDM
Priority: Normal
Status: Open