[IDD #JLJ-308670]: NEXRAD Level II outage

Jamie,

> I can fill in the background a bit for you though: The initial problem was an
> LDM queue error:
> 
> Feb 26 20:43:17 llwxldm1 idd.unidata.ucar.edu[11304] ERROR: pq_del_oldest:
> signature 54d80cc875df7b6f08ef920d4301df55: Not Found
> Feb 26 20:43:17 llwxldm1 idd.unidata.ucar.edu[11304] ERROR: pq_insert() 
> failed:
> Invalid argument: bfc927f1ff82336f5bdfe668e0c8f5c1    35219 20130226204316.841
> NEXRAD2 891006  L2-BZIP2/KEPZ/20130226204214/891/6/I/V06/0
> Feb 26 20:43:17 llwxldm1 idd.unidata.ucar.edu[11304] INFO: Exiting
> Feb 26 20:43:17 llwxldm1 rpc.ldmd[10789] NOTE: child 11304 exited with status 
> 10

I'd say your product-queue looks like it was corrupted, somewhow.

> Then the LDM went idle. When I saw this, I restarted it. But I got errors 
> again:
> 
> Feb 26 21:13:40 llwxldm1 pqexpire[12700] ERROR: pq_seqdel: xdr_prod_info() 
> failed
> Feb 26 21:13:40 llwxldm1 pqexpire[12700] ERROR: pq_seqdel failed: Input/output
> error (errno = 5)
> Feb 26 21:13:40 llwxldm1 pqexpire[12700] NOTE: Exiting

The pqexpire(1) utility is obsolete and unnecessary -- even for LDM 6.6.5. You 
shouldn't be using it.

> Then, after receiving two packets, I got these:
> 
> Feb 26 21:13:41 llwxldm1 idd.unidata.ucar.edu[12701] ERROR: pq_del_oldest:
> signature b729ad4065f9a3e3ce202400a4a296ea: Not Found
> Feb 26 21:13:41 llwxldm1 idd.unidata.ucar.edu[12701] ERROR: pq_insert() 
> failed:
> Invalid argument: 4fd830c6d703339aaa13d6c974e29f1d    53372 20130226205840.786
> NEXRAD2 447014  L2-BZIP2/KDTX/20130226205728/447/14/I/V03/0
> Feb 26 21:13:41 llwxldm1 idd.unidata.ucar.edu[12701] INFO: Exiting
> Feb 26 21:13:41 llwxldm1 rpc.ldmd[12698] NOTE: child 12701 exited with status 
> 10
> Feb 26 21:14:37 llwxldm1 rpc.ldmd[12698] NOTE: Exiting
> 
> Then I removed the product queue and recreated it and restarted the LDM and
> that's when I started seeing the connection errors below.
> 
> ---------------------------+---------------------------
> James M. Pelagatti (Jamie) | MIT Lincoln Laboratory
> Software Engineer        | Group 43 (Weather Sensing)
> (781) 981-1886           | 244 Wood St., Room S1-611
> FAX: (781) 981-0632      | Lexington, MA 02420-9108
> mailto:address@hidden  | http://www.ll.mit.edu


Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: JLJ-308670
Department: Support LDM
Priority: Normal
Status: Closed