[CONDUIT #APE-897679]: Missing data on idd.unidata.ucard.edu feed

Hi Chi and Justin,

We got back word from Penn State regarding the CONDUIT hiccup seen on March
8.  Here are the relevant parts of Art Person's reply:

> > We noticed some burps in NCEP data generally on March 8, but NCEP was
> > apparently having some processing problems that day which seemed to smooth
> > out.  Below is a listing of the 00Z and 12Z decoded gempak data for March
> > 7 and 8.  The 00Z and 06Z runs for the 8th look fairly normal, but the 12Z
> > run looks a little skimpy.  18Z the 8th and 00Z the 9th (not posted) look
> > fairly normal again:
> >
> > -rw-rw-r--   2 ldm      ldmsys   148244480 Mar  7 00:26 geavg_2011030700.gem
> > -rw-rw-r--   2 ldm      ldmsys   189058560 Mar  7 00:24 gec00_2011030700.gem
> > -rw-rw-r--   2 ldm      ldmsys   192459264 Mar  7 00:23 gep01_2011030700.gem
> > -rw-rw-r--   2 ldm      ldmsys   191659520 Mar  7 00:24 gep02_2011030700.gem
> > ...
> > -rw-rw-r--   2 ldm      ldmsys   148463616 Mar  8 01:00 geavg_2011030800.gem
> > -rw-rw-r--   2 ldm      ldmsys   188037632 Mar  8 00:59 gec00_2011030800.gem
> > -rw-rw-r--   2 ldm      ldmsys   192128000 Mar  8 00:59 gep01_2011030800.gem
> > -rw-rw-r--   2 ldm      ldmsys   193584128 Mar  8 00:59 gep02_2011030800.gem
> > ...
> > -rw-rw-r--   2 ldm      ldmsys   148010496 Mar  7 06:27 geavg_2011030706.gem
> > -rw-rw-r--   2 ldm      ldmsys   188246016 Mar  7 06:26 gec00_2011030706.gem
> > -rw-rw-r--   2 ldm      ldmsys   193233920 Mar  7 06:23 gep01_2011030706.gem
> > -rw-rw-r--   2 ldm      ldmsys   193673216 Mar  7 06:24 gep02_2011030706.gem
> > ...
> > -rw-rw-r--   2 ldm      ldmsys   148224512 Mar  7 12:24 geavg_2011030712.gem
> > -rw-rw-r--   2 ldm      ldmsys   189570048 Mar  7 12:23 gec00_2011030712.gem
> > -rw-rw-r--   2 ldm      ldmsys   193326080 Mar  7 12:24 gep01_2011030712.gem
> > -rw-rw-r--   2 ldm      ldmsys   193821184 Mar  7 12:24 gep02_2011030712.gem
> > -rw-rw-r--   2 ldm      ldmsys   191723520 Mar  7 12:23
> 
> > I do notice a change in transmission volume characteristics around 18Z on
> > March 8 in the Unidata graphs:
> >
> > http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?CONDUIT+idd-ingest.meteo.psu.edu
> 
> > It looks like our CONDUIT source switched from ncepldm1 to ncepldm0 around
> > 18Z the 8th.  Prior to 18Z, we were primary/alternate flip-flopping
> > between ncepldm1 and idd.unidata.ucar.edu (which we use for an alternate),
> > but after 18Z, delivery from ncepldm0 must have been better because the
> > flip-flopping seems to have ceased.  I also notice that there have been no
> > flip-flop messages in our logs for the woc since 18Z March 8 through now.
> > However, there were flip-flop messages on March 7 and earlier dates.
> > They seemed to increase in frequency beginning on February 27, 2011 and
> > all were primary/alternate flip-flops to ncepldm1.
> 
> re:
> > Here are the log entries for the ncepldm1.woc.noaa.gov on
> > idd-ingest.meteo.psu.edu for March 8:
> >
> > Mar 08 17:02:32 idd-ingest ncepldm1.woc.noaa.gov[31726] ERROR: readtcp(): 
> > select() timeout on socket 4
> > Mar 08 17:02:32 idd-ingest ncepldm1.woc.noaa.gov[31726] ERROR: 
> > one_svc_run(): RPC layer closed connection
> > Mar 08 17:02:32 idd-ingest ncepldm1.woc.noaa.gov[31726] ERROR: 
> > Disconnecting due to LDM failure; Connection to upstream LDM closed
> > Mar 08 17:02:32 idd-ingest ncepldm1.woc.noaa.gov[31726] NOTE: LDM-6 desired 
> > product-class: 20110308160232.557 TS_ENDT {{CONDUIT,  ".*"},{NONE, 
> > "SIG=da621d69457602862a27f7e7677371ae"}}
> > Mar 08 17:02:32 idd-ingest ncepldm1.woc.noaa.gov[31726] NOTE: Upstream 
> > LDM-6 on ncepldm1.woc.noaa.gov is willing to be a primary feeder
> > Mar 08 17:17:31 idd-ingest ncepldm1.woc.noaa.gov[31726] ERROR: 
> > Disconnecting due to LDM failure; Upstream LDM died
> > Mar 08 17:17:31 idd-ingest ncepldm1.woc.noaa.gov[31726] NOTE: LDM-6 desired 
> > product-class: 20110308161731.506 TS_ENDT {{CONDUIT,  ".*"},{NONE, 
> > "SIG=0e95dce1e3c492a55437f1050c60be08"}}
> ...
> 

Evidently, Art did not see any of the DNS problems that we (Unidata) saw.  His 
LDM log
file, however, confirms problems getting data from ncepldm2.woc.noaa.gov on 
March 8.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: APE-897679
Department: Support CONDUIT
Priority: Normal
Status: Closed