20030719: HDS ingest troubleshooting (cont.)

>From: David Raymond <address@hidden>
>Organization: NMT
>Keywords: 200307041809.h64I9TLd018509 IDD LDM-6

Hi Dave,

re: switch HDS feed to seistan.srcc.lsu.edu

>I am making the change at this instant.

Thanks for the change.

re: packet shaping at NMT

>They are running a packet shaper and I have already talked to them and
>they made some accommodation.  I will talk to them again if necessary.

The latencies for the HDS stream are huge.  The latencies for the
UNIWISC stream are way larger than they should/could be ranging up
to 200 seconds per product.  Most "tuned" sites are seeing latencies
for UNIWISC in the one to a few seconds range.

>What is in HDS?  Are the raw radiosondes in there or in IDS/DDPLUS?  I
>may not actually need it.

HDS contains just the NCEP model output.  All observational data like
METARs, rawinsondes, ship/buoy reports, etc. are in the IDS|DDPLUS
stream.  The HDS volume (which ranges up to 300 MB per hour) is a lot
more than either UNIWISC (which is up to 6 MB per hour currently) or
IDS|DDPLUS (which is about 10 MB per hour).  So, you can see that the
ingest of the model data swamps the ingest of your other feeds in terms
of volume.  This situation will worsen as more and more gridded fields
are added to NOAAPORT.

By the way, after switching the HDS feed from yin to rainbow, the
volume of HDS data you are receiving dropped from 300 MB/hr to about 50
MB/hr.  The bad latencies experienced in feeding from rainbow is
causing the loss of up to 250 MB/hr.

Tom

>From address@hidden Sun Jul 20 17:03:00 2003

Hi Tom,

I am turning off HDS, as we currently don't use it.

Dave

>From address@hidden Mon Jul 21 13:40:35 2003

>I am turning off HDS, as we currently don't use it.

OK, sounds reasonable.

One last change:

Since it appears that the NMT packet shaping is done on a per host
basis, and since your reception of IDS|DDPLUS from yin.engin.umich.edu
is showing very small latencies, and since the HDS latencies you were
seeing while ingesting from yin were much smaller than what you saw
from rainbow and seistan, I think that it would be wise to move your
UNIWISC request from rainbow to yin:

request UNIWISC ".*"    yin.engin.umich.edu

However, since you were not ingesting this feed from yin at the
beginning of the LDM-6 upgrade process, it is possible that U Mich has
not allowed heron to feed UNIWISC.  So, you will need to run a notifyme
command as 'ldm' before you change your ldmd.conf request line:

<login as 'ldm'>
notifyme -vxl- -f UNIWISC -o 3600 -h yin.engin.umich.edu

If you are allowed to do the notifyme, you are also allowed to feed
that stream from yin.  I know that yin has the UNIWISC data because I
just ran the same 'notifyme' command from a machine here at the UPC.

I figure that once you switch your UNIWISC feed to yin, you will be
ingesting as well as can be expected given the packet shaping that is
being done on your campus.

Cheers,

Tom

>From address@hidden Mon Jul 21 14:04:24 2003

Hi Tom,

I just switched to yin as you suggest.

Dave