[IDD #JLJ-308670]: NEXRAD Level II outage

James,

> We just experienced a full outage of all our NEXRAD Level II data that we pull
> from Unidata via LDM. We're now trying to determine whether the problem was at
> our end or the Unidata end.
> 
> We lost data at 20:43:17Z and it returned at 21:30:05Z. Our logs contained 
> many
> messages like the following during the outage:
> 
> Feb 26 21:16:25 llwxldm1 idd.unidata.ucar.edu[12796] NOTE: LDM-6 desired
> product-class: 20130226210125.139 TS_ENDT {{NEXRAD2,  "(.*)"},{NONE,
> "SIG=e8cdcd0c6992e8d6e3a46eda90eb93f4"}}
> Feb 26 21:16:25 llwxldm1 idd.unidata.ucar.edu[12796] INFO: Resolving
> idd.unidata.ucar.edu to 128.117.140.3 took 0.011059 seconds
> Feb 26 21:16:25 llwxldm1 idd.unidata.ucar.edu[12796] INFO: Connected to 
> upstream
> LDM-6 on host idd.unidata.ucar.edu using port 388
> Feb 26 21:16:25 llwxldm1 idd.unidata.ucar.edu[12796] ERROR: Disconnecting due 
> to
> LDM failure; Upstream LDM says we're not allowed to receive requested 
> products:
> 20130226210125.139 TS_ENDT {{NEXRAD2,  "(.*)"},{NONE,  
> "SIG=e8cdcd0c6992e8d6e3a46
> eda90eb93f4"}}
> Feb 26 21:16:25 llwxldm1 idd.unidata.ucar.edu[12796] INFO: Sleeping 13 seconds
> before retrying...
> 
> Can you tell us if there were any problems at Unidata during this time? If 
> not,
> were there problems upstream at NWS, do you think?

It looks like the downstream LDM process responsible for receiving NEXRAD-2 
data at your end decided that it needed to reconnect and, consequently, closed 
the connection. The closure, however, doesn't appear to have been propagated to 
the matching upstream LDM at our site. Consequently, all subsequent 
re-connection attempts were rebuffed until the matching upsteam LDM finally 
received a broken connection signal.

This might indicate a problem with our Linux Virtual Server (LVS) 
implementation (idd.unidata.ucar.edu is actually a cluster of computers served 
by LVS).

I'm investigating further and will let you know if I find anything.

Please keep me apprised of any more problems at your end.

> Thanks.
> 
> ---------------------------+---------------------------
> James M. Pelagatti (Jamie) | MIT Lincoln Laboratory
> Software Engineer        | Group 43 (Weather Sensing)
> (781) 981-1886           | 244 Wood St., Room S1-611
> FAX: (781) 981-0632      | Lexington, MA 02420-9108
> mailto:address@hidden  | http://www.ll.mit.edu

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: JLJ-308670
Department: Support LDM
Priority: Normal
Status: Closed