20080216: IDD burstiness (cont.)

Hi Daryl,

Strange topic for a Saturday evening/night :-)

re:
>Just off-list note, I have been seeing problems since yesterday too:
>
>http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+pircsl4.agr
> on.iastate.edu
>
>Its bursting :)

>Its on unidata2.ssec and idd.unidata as well:

>http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+idd.unidata.ucar.edu

The spikes being seen by all are undoubtedly "second trip" or
"recirculation" products being received on idd.unidata.ucar.edu from
mistral.srcc.lsu.edu.  The products being received on
idd.unidata.ucar.edu from dvb1.ssec.wisc.edu uniformly have very low
latencies.  It would seem that LDM queues for data being received on
both oliver.unidata.ucar.edu and emo.unidata.ucar.edu -- the
accumulators for the idd.unidata.ucar.edu cluster -- are not large
enough to account for/reject "old" products from mistral.srcc.lsu.edu.
This is understandable given that both olvier and emo are ingesting the
entire volume of data available on the IDD multiply redundantly from
upstream sites.

The recirculation argument does not, however, account for Jeff's
report of burstiness in the NNEXRAD products.  I have been trying
to ascertain if his problem is really one of non-receipt of the
products in his LDM queue or one of his machine not processing
the products out of his queue.  Since Jeff's NNEXRAD latencies do not
show a burstiness in receipt, I am led to believe that there is
some sort of a processing bottleneck on his machine.  I am a bit
frustrated that I have not been able to make this distinction clear
to Jeff, but, then again, he has not (to my knowledge) attended
an LDM workshop, so he may not understand the distinction between
the two phenomena. 

I think that if you study the latency plots in conjunction with the
volume plots, you will see that the entire content of the IDS|DDPLUS
datastream is being sent to downstreams with low latency.  The hiccup
is the set of`high-latency, second-trip products being intoduced into
the system from mistral.srcc.lsu.edu.

By the way, the explanation for why we are now seeing the relatively
high latency products from mistral on oliver/emo is that the two
NOAAPort ingesters here at UCAR are not working at the moment.
A service call was placed this morning; we expect to get the problem
diagnosed and resolved on Tuesday morning when the service is made.

Cheers,

Tom
--
+----------------------------------------------------------------------------+
* Tom Yoksas                                            UCAR Unidata Program *
* (303) 497-8642 (last resort)                                 P.O. Box 3000 *
* address@hidden                                  Boulder, CO 80307 *
* Unidata WWW Service                            http://www.unidata.ucar.edu/*
+----------------------------------------------------------------------------+