Re: 20080216: IDD burstiness (cont.)

Hi Tom,

Thanks for your follow up.

> The spikes being seen by all are undoubtedly "second trip" or 
> "recirculation" products being received on idd.unidata.ucar.edu from 
> mistral.srcc.lsu.edu.  The products being received on 
> idd.unidata.ucar.edu from dvb1.ssec.wisc.edu uniformly have very low 
> latencies.  It would seem that LDM queues for data being received on 
> both oliver.unidata.ucar.edu and emo.unidata.ucar.edu -- the 
> accumulators for the idd.unidata.ucar.edu cluster -- are not large 
> enough to account for/reject "old" products from mistral.srcc.lsu.edu. 
> This is understandable given that both olvier and emo are ingesting the 
> entire volume of data available on the IDD multiply redundantly from 
> upstream sites.

Ah, makes sense.

> The recirculation argument does not, however, account for Jeff's report 
> of burstiness in the NNEXRAD products.

I have noticed similiar problems with IDS products that started yesterday 
afternoon.  I can't attest to the burstiness, just that seemingly random 
products come really late (10-20 minutes).

The last concrete example was:

553
WWUS43 KDMX 162149
WSWDMX

This didn't hit my LDM until 5 minutes later.

> I think that if you study the latency plots in conjunction with the 
> volume plots, you will see that the entire content of the IDS|DDPLUS 
> datastream is being sent to downstreams with low latency.  The hiccup is 
> the set of`high-latency, second-trip products being intoduced into the 
> system from mistral.srcc.lsu.edu.

I dunno, I have lots of users complaining to me and have noticed similiar 
issues with COD's website.  I have lots of NWS users, so they will squawk 
to me when a product doesn't hit iembot, since they issued the product and 
can sense latency in real time :)

> By the way, the explanation for why we are now seeing the relatively 
> high latency products from mistral on oliver/emo is that the two 
> NOAAPort ingesters here at UCAR are not working at the moment. A service 
> call was placed this morning; we expect to get the problem diagnosed and 
> resolved on Tuesday morning when the service is made.

Ahh, I also noticed this ADM today:

NOUS72 KNCF 111452
ADMNCF
THE NWSTG IS EXPERIENCING TECHNICAL DIFFICULTIES WITH A FRONT END
PROCESSOR.  THEY ARE TROUBLSHOOTING THE ISSUE AND HOPE TO HAVE
THE PROBLEM RESOLVED IN A TIMELY FASHION.  THANK YOU FOR YOUR
PATIENCE.

I tried to verify any problems with other data vendors and couldn't get 
any affirmative responses.

thanks,
  daryl