20030718: HDS inest troubleshooting (cont.) and UNIWISC mods

>From: David Raymond <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200307041809.h64I9TLd018509 IDD LDM-6

Hi Dave,

re: proposed UNIWISC changes
>I like the proposed changes, though the increase in data flow may
>be a bit daunting.

The good news is that a site can choose to not ingest the new bands and
continue to get only the hourly images.  The minimum increase for a
site will end up being about 4.6 MB per hour.

>Actually, I would like to see the southern
>hemisphere from GOES west as well, since it is of interest from
>the point of view of tropical oceanic meteorology.

I will experiment with compositing the GOES-West Northern Hemisphere
and Southern Hemisphere sectors to see what they look like (size,
coverage, etc.).  I suspect that the west composites would be a bit
smaller than the east ones since the southern scan from GOES-10 is
smaller than the same for GOES-12, but I need to generate some
numbers.  If the composite's size was the same as the eastern one, the
increase in data on disk would increase from 56 MB per hour kept to
approx. 81 MB per hour.

Back to the HDS testing on heron.  The ingest test from rainbow to
heron as been running for about a day, and the results are that the
latencies from rainbow to heron are significantly _worse_ than from
yin.engin.umich.edu to heron.  This was totally unexpected since
rainbow is on the same regional network as UCAR, so it has the same
huge pipe to the internet (I2) as we do.  I ran some traceroute tests
to heron, and see that we (Boulder) are only 14 ms away whereas yin is
over 35 ms away.  My tentative conclusion is that whatever is causing
the latency (packet shaper) must be setup to act differently for feeds
from yin than from rainbow.

If you read this note this weekend, I would appreciate you changing
your HDS feed from rainbow to seistan.srcc.lsu.edu.  Yesterday
afternoon I setup seistan to allow feed requests from heron, so you
should be good to go.  For the record, I don't expect the HDS ingest to
improve with this switch, but I am curious to see if the latencies from
LSU match those from rainbow.

The next phase in the latency investigation is contacting the IT group
at NMT to see if they are running a packet shaper (and if they will
admit that they are doing so), and, if so, if they will bore a hole in
it for port 388 traffic.  Will you be willing to contact these folks?

We are more than willing to participate in any discussions with your IT
group.  This is exactly what we did with LSU while investigating
outbound data feed problems there.  A conference call among LSU/SRCC
personnel, LSU IT personnel, and Unidata developers resulted in LSU
creating a single use "pipe" that was allocated 200 Mbps for LDM use
(LSU is a top level IDD relay, so the bandwidth needed can be large).

Tom