Re: 20021015: 20021015: Data falling behind on thelma...

Steve,

On Tue, 15 Oct 2002, Unidata Support wrote:

> On closer inspection, the graphs weren't updating this morning due
> to an unrelated change on one of our machines here. Today
> is julian 288. The 1200 second jump was just before 0Z last night.
> The graphs are now updating. For comparison, you can see 
> marzipan.atmos.washington.edu
> feeds NMC2 from thelma and is doing OK, and sunny.atmos.washington.edu
> is feeding the HDS, NNEXRAD etc from thelma. Again looks OK.
>
> You NNEXRAD feed from SSEC still shows similarly poor perfomance as
> your feeds to Thelma. Your NEXRD2 via ou.edu looks the same as what we see 
> here
> on thelma. So, it appears that your traffic out to certain hosts is affected 
> more
> than others.
>
> I tried running traceroutes and ldmping from my side to ldm.meteo.psu.edu
> and got long timeouts at first, then later was able to get through.
> It would appear that the proplem lies in your network. Many campus
> IT departments are playing with traffic controls, so you might ask if
> they are throttling throughput.

No indications that Penn State is throttling anything, but we did find the
problem (apparently)...  our College upgraded its network switches and
when they switched our facility over to a new switch, it didn't
autonegotiate properly and was running 100Mbit *half* duplex...  They
fixed it to *full* duplex (which our switch was hard-wired for) and things
resumed normal operation.  What threw me was that we got some data well
from some sources but not others...  I guess you can never rule anything
out with networks!

                                    Art.

>
> Steve Chiswell
>
>
>
>
> >From: "Arthur A. Person" <address@hidden>
> >Organization: UCAR/Unidata
> >Keywords: 200210151749.g9FHnP121406
>
> >Steve,
> >
> >On Tue, 15 Oct 2002, Unidata Support wrote:
> >
> >> It looks like your machine experienced a 1200 second interuption.
> >
> >That's because I had rpc.ldmd -m set to 1200 seconds when I remade my
> >queue awhile (couple weeks?) ago.  It was running that way and reclassing,
> >no doubt, but that was a symptom that data was falling behind.  I
> >restarted the ldm to run with -m 10800 which is probably the bump you saw
> >as an interuption.
> >
> >> Your HDS feed shows the jump up, and has now gone back to normal:
> >> http://www.unidata.ucar.edu/staff/chiz/latency/stats/HDS_ldm.meteo.psu.edu.g
> > if
> >>
> >> The same 1200 second jump occurs in your NNEXRAD and NMC2 feeds from
> >> SSEC and Thelma respectively:
> >> http://www.unidata.ucar.edu/staff/chiz/latency/stats/NNEXRAD_ldm.meteo.psu.e
> > du.gif
> >> http://www.unidata.ucar.edu/staff/chiz/latency/stats/NMC2_ldm.meteo.psu.edu.
> > gif
> >>
> >> You can view all your rtstats being sent here at:
> >> http://www.unidata.ucar.edu/staff/chiz/latency/stats/
> >>
> >>
> >> The 1200 second jump does not appear in thelma's reception of NMC2 from
> >> outside here, so it would appear that it is on your end, especially since
> >> the jump appears in both your feed from SSEC and from here.
> >>
> >> Since your HDS has gone back to ~0, I would look for the other feeds to 
> >> begi
> > n to
> >> recover too....but they are larger and will take more time.
> >>
> >> In the mean time, if you don't see improvement, let us know. We can keep
> >> track of the latency graphs on this end.
> >
> >I'm not sure why some of the graphs show the data on time, but here's what
> >I'm seeing with an ldmadmin watch:
> >
> >   Current time:  17:37Z
> >   Transmit times of some streams:
> >       NEXRD2:        17:36Z
> >       NNEXRAD:       16:28Z
> >       NMC2:          16:13Z
> >       HDS|DDPLUS:    15:52Z
> >
> >When I lifted the -m window to 10800, I would have expected to see
> >IDS|DDPLUS|HDS catch up or at least hold steady, but instead it's fallen
> >much farther behind.  NEXRD2 still remains on time.  Any ideas?
> >
> >                                 Art.
> >
> >> >From: "Arthur A. Person" <address@hidden>
> >> >Organization: UCAR/Unidata
> >> >Keywords: 200210151610.g9FGAH113439
> >>
> >> >Hi...
> >> >
> >> >I'm seeing data from thelma falling beind... currently ~35 minutes behind.
> >> >Other sources my NEXRD2 source is within a few minutes but NNEXRAD is also
> >> >~35 minutes behind (from ssec).  Are you noticing any slowness on
> >> >thelma?... or should I begin beating on our networking people?
> >> >Traceroutes to thelma seem okay...
> >> >
> >> >                                   Art.
> >> >
> >> >Arthur A. Person
> >> >Research Assistant, System Administrator
> >> >Penn State Department of Meteorology
> >> >email:  address@hidden, phone:  814-863-1563
> >> >
> >>
> >> ****************************************************************************
> >  <
> >> Unidata User Support                                    UCAR Unidata 
> >> Program
> >  <
> >> (303)497-8643                                                  P.O. Box 
> >> 3000
> >  <
> >> address@hidden                                   Boulder, CO 80307
> >  <
> >> ----------------------------------------------------------------------------
> >  <
> >> Unidata WWW Service                        http://www.unidata.ucar.edu/
> >  <
> >> ****************************************************************************
> >  <
> >>
> >
> >Arthur A. Person
> >Research Assistant, System Administrator
> >Penn State Department of Meteorology
> >email:  address@hidden, phone:  814-863-1563
> >
>
> **************************************************************************** <
> Unidata User Support                                    UCAR Unidata Program <
> (303)497-8643                                                  P.O. Box 3000 <
> address@hidden                                   Boulder, CO 80307 <
> ---------------------------------------------------------------------------- <
> Unidata WWW Service                        http://www.unidata.ucar.edu/      <
> **************************************************************************** <
>

Arthur A. Person
Research Assistant, System Administrator
Penn State Department of Meteorology
email:  address@hidden, phone:  814-863-1563