20050222: FW: 20050217: Need help to resolve long latency issue on IDD/LDM (fwd)

>From: "Darren Yezo" <address@hidden>
>Organization: Kean University
>Keywords: 200502221655.j1MGtmv2010630 IDD packet shaping NOAAPORT

Hi Darren,

I apologize for taking so long to respond to you on this topic...

>I have been talking to Dr Yoh for a bit of time on this and I think I it
>will be easier to discuss this with you directly.

OK.

>Maybe there was some
>lost communication somewhere..  We are quite aware that we can throttle
>and unthrottle bandwidth by port.

I did not interpret the original email from Shing to mean that
he knew that you were using packet shaping techniques to limit
traffic on IDD ingest streams.

>The reason we limited hurri.kean.edu
>is because at the time the unidata computer was taking up over 10% of
>our networks bandwidth.. We are not the size of LSU. Their 20 mbps
>connection you mentioned in the email is more then our ENTIRE non dorm
>internet connection.

I did intend the comment about LSU to be interpreted as meaning that
Kean has the same kind of resources as LSU but was hording them.
Rather, I was trying to relay an experience where our communications
with an LSU departmental contact resulted in a denial of the existence
of "packet shaping".  After a conference call with LSU departmental and
IT representatives, department folks were made aware of "packet
shaping" being done by their IT group.  After we talked with the IT
folks about what was in the data being transferred, they expressed a
great deal of interest in seeing it flow freely.  The comment they made
was to the effect that was what their network was designed and funded
for.  Given this, they gladly "bored" a 20 Mbps hole through which the
IDD data could freely flow.

Given the LSU experience and our working with IT folks at other
insitutions, I wanted to make sure that Shing was effectively
communicating that the use of the bandwidth was for research and
education, not for things that plague many instutions like MP3
downloads.

>So while the unidata information is for research we
>also need our internet connection for online class, web registration,
>etc...  

We agree wholeheartedly.

>So other then turning off packet shaping on port 388 is there anything
>else you can recommend? 

There are a couple of approaches that can be explored.  In my previous
email to Shing, I suggested a way that the LDM could be configured
to fly under the packet shaping barrier.  This approach would not, 
however, save you bandwidth as the aggregate amount of data would
stay the same while not being slowed by packet shaping constraints.

What would help the situation (other than Kean getting a bigger
Internet pipe :-) is:

- for Shing to review exactly which data in the HDS feed they
  really need/want.  It is our observation that a lot of sites
  don't actually use all of the data that they are receiving, so
  elmininating that data from feed requests would result in a
  lower bandwidth use

- installation of a NOAAPORT data ingest system at Shing's department
  and pipe the output to machines on the departmental LAN which
  would result in using more internal bandwidth (which should be more
  like 100 Mbps) and less external bandwidth

The second option will cost some amount of money for Shing's
department/ Kean University.  The cost might be mitigated by use of
equipment it may have used back when Unidata universities' only way to
get real time data was through a satellite link.  For instance, Kean
might still have the satellite installation pad, dish, and LNB; if it
does it may not need to spend any money on replacement equipment (if it
is in good working order).

In order for a new satellite reception system to work well a site
survey should be done to make sure that the dish location is not
subject to Terrestrial Interference (TI).  If it is, a new site would
need to be found, and that site would need to be close enough to a
location where a computer with network connection and clean power
(UPS) could be housed.

In addition, Kean would need to purchase a Novra S75 DVB-S receiver
(www.novra.com) for about $320 and possibly a tuned cavity filter (I
can dig up the info on what the NWS is deploying) and line amplifier
(Norsat LA-30).  The tuned cavity filter would be needed if TI is a
problem at the satellite dish site; the line amp is needed if the tuned
cavity filter is used.

Finally, Shing could get our NOAAPORT ingest code and run it on a
machine that is exclusively dedicated to the ingest and movement of
data to the departmental LAN using the LDM (no decoders; no
applications; no web serving; etc.).  The connection between the Novra
and PC (running Linux or Solaris x86) would be through a dedicated
Ethernet port (the machine will need to have two: one to connect it to
the department LAN, and the other to accept the UDP multicast stream
coming out of the Novra S75 receiver.

The data coming out of the Novra would represent the full NOAAPORT
broadcast, so it would be the majority of data that Shing would want
from the IDD.  Streams that he would probably still want to get from
the IDD include GOES imagery in the UNIWISC stream, NEXRAD composite
imagery from the FNEXRAD stream, and lightning data from the NLDN
stream.  He would already be getting all of the IDS|DDPLUS, HRS,
NNEXRAD, NIMAGE, and new NGRID datastreams.  The volume of the NOAAPORT
broadcast data is currently on the order of 612 MB/hr (megabytes per
hour) with peaks of over 1.3 GB/hr (gigabytes per hour).  While the
internal bandwidth use at Kean would likely climb, the amount being
transferred through Kean's link to the outside world would decrease
dramatically.

The question to you and Shing are whether or not pursuing a NOAAPORT
reception system makes sense and could be supported on an ongoing
basis?  Satellite reception systems need attention from time to time,
so this is not a install once and forget option.

Once again, I apologize for taking so long to get back to you on
your inquiry.

Cheers,

Tom

>-----Original Message-----
>From: Shing Yoh [mailto:address@hidden] 
>Sent: Friday, February 18, 2005 11:01 AM
>To: address@hidden
>Cc: address@hidden
>Subject: 20050217: Need help to resolve long latency issue on IDD/LDM
>(fwd)
>
>Darren,
>
>       Following is the email exchange and I have with Unidata.  It has
>my understanding of the situation and their responses.  The bottom line
>is that they feel that the packet shaper software should be able to
>leave 
>the IDD/LDM traffic passes without delay.  If we can not open
>IP port 388 for just hurri.kean.edu, how about the idea that just open
>IP port 388 for the whole network ?  Anyhow, I believe that this email
>should give you and your software support people a better idea
>our situation and need.  Please let me know if there is anything that
>you need to have or to know in order for us to resolve this issue.
>
>Thanks
>
>Shing
>
>---------- Forwarded message ----------
>Date: Thu, 17 Feb 2005 22:42:41 -0700
>From: Unidata Support <address@hidden>
>To: Shing Yoh <address@hidden>
>Cc: address@hidden, address@hidden,
>address@hidden
>Subject: 20050217: Need help to resolve long latency issue on IDD/LDM
>
>>From: Shing Yoh <address@hidden>
>>Organization: Kean University
>>Keywords: 200502172036.j1HKaUv2005158 IDD packet shaping
>
>Hi Shing,
>
>>Since a few months ago when packet shaper software was installed on our
>>campus network, we encountered serious issue on ingesting model data
>>(missing data and high latency).
>
>The classic signature of packet shaping is high latencies for feeds
>that contain a lot of data (e.g., HDS), and low latencies for feeds
>that do not (e.g., IDS|DDPLUS).
>
>>From the information that I can gather
>>so far from the Unidata web pages and email archive, I had talked to
>the
>>IT people on the campus and had already taken the following steps :
>>
>>(1) I have changed the product queue size in ldmadmin
>$pq_size=1000000000
>>and restart the ldm.  The file size of ldm.pq is running at 1015898112
>>byte.
>
>The LDM queue size will have no effect on the latencies you see.  The
>only thing it might do is help keep older data in the queue longer.
>I am not sure that this would be important for your use, but it might
>be.
>
>>(2) I have talked to the IT people and I was told that we are currently
>>set at 800kps for bandwidth to our server "hurri.kean.edu".  The
>setting
>>is for the all ports and traffic to our server since the software can
>not
>>be set for just ldm IP port 388.
>
>This is the first time that I have heard of a packet shaper that can not
>be turned off for activity on a certain port.  I would argue that if
>they can turn off packet shaping for an individual port, then they
>should do so for port 388 since it is only used for the LDM and the
>LDM is moving scientific data needed for education and research.  It
>is not moving MP3s or movie sreams.
>
>>(3) The above rule for traffic to our server has priority 10, the
>highest.
>
>This is good, but not sufficient.
>
>>Unfortunately, with all these changes, our latency for HDS model data
>>is still at 4000s (with a much reduced set of model data then what we
>used
>>to ingest).
>
>Three things that I need to mention here:
>
>1) the clock on your system is not being maintained.  Please take a look
>    at the latency plot for your IDS|DDPLUS ingestion:
>
>http://my.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+hurri.
>kean.edu
>
>    The stair step in baseline latency is an indication that your clock
>is
>    off by about 50 seconds, and that it is drifging at a rate of about
>1 second
>    every 12 hours.  I strongly recommend that you setup ntpd on your
>machine
>    so that the clock be maintained correctly.  An accurate clock is
>essential
>    to receiving all data in a the IDD especially when the LDM is
>stopped
>    and restarted for some reason.
>
>2) your system _is_ showing the classic packet shaping signs.  The
>latency
>    for IDS|DDPLUS is low, and the latency for HDS is high.  This means
>    that the input to hurri is not limited to 800 Kbps in total, but,
>    rather, each connection is limited individually.  This makes me
>believe
>    that your IT folks _could_ turn off the packet shaping for all
>traffic
>    on port 388.  I also have the sneaking suspicion that they could
>    change the limit for port 388 traffic individually, but I can't
>guarantee
>    that would be the case.
>
>3) given that the packet shaping is being applied on a
>connection-by-connection
>    basis, you could split up your HDS data request into a number of
>    requests each one of which would be asking for a smaller portion of
>    the whole feed.  Before doing this, however, I urge you to continue
>    to fight the good fight with your IT folks and press for there being
>    no packet shaping for port 388 traffic.
>
>    If you can not win the war with your IT folks, the first thing you
>    should do is determine which of the grids in the HDS feed are the
>    most important to you.  If there are any grids that you absolutely
>    do not use, then eliminate them from your data request to cut the
>    volume.  Then take the remaining set of data you want, and try
>    to break it up into a number of requests each of which will be
>    small enough to fly under the packet shaping radar.
>
>>I am running out of ideas to try or suggestions to IT people.  Are
>there
>>any suggestions from Unidata that I or our IT people should try or
>>investigate in order to resolve this long latency issue ?
>
>Again, I am not convinced that your IT people are aware of their ability
>to alter the packet shaping profile by port.  We ran into a packet
>shaping
>situation at LSU, and had a conference call between the SRCC folks (the
>users of the LDM), the campus IT folks, and several of the UPC technical
>staff.  After the IT staff were made aware of the kind of dat athat
>is being ingested, and its use in research and education, they bored a
>hole in their packet shaper that allows transfer rates of 20 Mbps.  The
>same thing could be done for Kean.
>
>>By the way, the packet shaper software that they are using is the
>>NetEnforcer by Allot Communications, Version 5.1.   Any help will be
>>appreciated.
>
>Thanks for passing along this information.  Our experience to date
>has been with sites using a system from Packeteer.  It might well
>be that NetEnforcer does not have as many controls as the system
>from Packeteer.  I am suspicious that this is not the case, however.
>It may boil down to your IT folks not knowing all that they can
>do with NetEnforcer.
>
>>Dr. Shing Yoh
>>Dept. Geology & Meteorology            K     K EEEEEEE    A    N     N
>>Kean University                        K    K  E         A A   NN    N
>>1000 Morris Avenue                     K   K   E        A   A  N N   N
>>Union, New Jersey 07083                KKKK    EEEEEE  A     A N  N  N
>>Voice : 908 737 3692                   K  K    E       AAAAAAA N   N N
>>Fax   : 908 737 3699                   K   K   E       A     A N    NN
>>Email : address@hidden                  K    K  E       A     A N     N
>>Web   : http://hurri.kean.edu/~yoh     k     K EEEEEEE A     A N     N
>
>Cheers,
>
>Tom Yoksas
>--
>************************************************************************
>Unidata User Support                                    UCAR Unidata
>(303)497-8643                                                  P.O. Box
>address@hidden                                   Boulder, CO
>------------------------------------------------------------------------
>Unidata WWW Service
>------------------------------------------------------------------------
>NOTE: All email exchanges with Unidata User Support are recorded in the
>Unidata inquiry tracking system and then made publicly available
>through the web.  If you do not want to have your interactions made
>available in this way, you must let us know in each email you send to
>us.
>
>
>
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.