[IDD #PLZ-432021]: Latency at Oswego

Hi Bob,

re:
> I have noticed a dramatic increase in data loss during the past week or
> so here at SUNY Oswego.  Our ingest machine is met-07.oswego.edu.  Do
> you have any suggestions?  Most of the losses are occurring during the
> afternoon.  We are using idd.cise-nsf.gov as our primary feed, and
> idd.unidata.ucar.edu as our alternate.

I looked at the latencies being reported by met-07.oswego.edu and see that the
numbers for UNIWISC, FSL2, and NNEXRAD while not spectacular are not nearly
as bad as for HDS and IDS|DDPLUS:

FSL2 latency
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?FSL2+met-07.oswego.edu

NNEXRAD latency
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NNEXRAD+met-07.oswego.edu

UNIWISC latency
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?UNIWISC+met-07.oswego.edu

Since the latency plots for IDS|DDPLUS are exactly the same as for HDS, I
suspected that you are requesting both of them on a single request for 'WMO'.
I confirmed this by looking in the ~ldm/etc/ldmd.log file on the IDD cluster
node that is feeding you the WMO data.

Since the volume of UNIWISC, NNEXRAD, and FSL2 are all much smaller than WMO,
I suspect that there is some sort of volume limitation being imposed on your
ingest.  Because of this, and since you most likely are wanting the 
observational
data to be received as soon as possible, I recommend that you split your
WMO request into separate requests for IDS|DDPLUS and HDS.  Here is the example
for the split for your current WMO request to idd.unidata.ucar.edu:

change:

request WMO ".*" idd.unidata.ucar.edu

to:

request IDS|DDPLUS ".*" idd.unidata.ucar.edu
request HDS ".*" idd.unidata.ucar.edu

Make the change to your WMO requests to both idd.unidata.ucar.edu and 
idd.cise-nsf.gov.

I am expecting that splitting off the IDS|DDPLUS feed request will result in 
dramatically
lower latencies for the IDS|DDPLUS data.  I am not, however, as optomistic for 
the
HDS data since it has the greatest volume of all of the feeds you are requesting
by a considerable amount:

Cumulative volume summary Graph
http://www.unidata.ucar.edu/cgi-bin/rtstats/rtstats_summary_volume?met-07.oswego.edu+GRAPH

Given the huge difference in the latencies between a reasonably high volume
datastream like NNEXRAD and that for WMO, and given your observation that this 
started
very recently, I am left with the impression that there have been some 
modifications
made to the Oswego network that is acting to limit your data ingestion.  We 
refer
to this generally as 'packet shaping'.  If the latency for the HDS feed remains 
high
after you have split your WMO request as I indicated above, I recommend that 
you contact
the networking group on your campus to see if they recently implemented 'packet 
shaping'.
If they have, you might have to make a case for having them open up port 388 
traffic
so you can continue to receive the data you need for education and research.

Please let us know if you need a more detailed explanation of 'packet shaping' 
from us.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: PLZ-432021
Department: Support IDD
Priority: Normal
Status: Closed