Re: 19990113: idd: delayed metars & NOAAPORT

On Wed, 13 Jan 1999, Unidata Support wrote:

> >To: address@hidden
> >From: Tom McDermott <address@hidden>
> >Subject: Re: 19981223: idd: delayed metars & NOAAPORT
> >Organization: .
> >Keywords: 199901131548.IAA14251
> 
> 
> On Wed, 23 Dec 1998, Robb Kambic wrote:
> > On Wed, 23 Dec 1998, Unidata Support wrote:
> > 
> > > 
> > > ------- Forwarded Message
> > > 
> > > >From: Tom McDermott <address@hidden>
> > > >Subject: idd: delayed metars & NOAAPORT
> > > >Organization: SUNY Brockport
> > > >Keywords: 199812231947.MAA02806 IDD NOAAPORT
> > > 
> > > Dear Unidata Support,
> > > 
> > > We are having a chronic problem here at SUNY Brockport with delayed
> > > METARS since the switch to NOAAPORT took place.  For obs made at 50
> > > minutes past the hour, we had formerly been receiving before the hour.
> > > Now they are coming in anywhere from 1 to 3 hours late.
> > > 
> > > This first occurred when the switch to NOAAPORT took place, and has been
> > > consistently happening ever since.  Network connectivity to our feed at
> > > Cornell seems to be OK, no dropped packets, normal transmission times.
> > > However, the problem does seem to lie somewhere between us, since SUNY
> > > Albany, which is also fed from Cornell, is getting them on time.
> > > 
> > > The only thing we have to go on is the number of disconnects from Cornell
> > > in our logs, which usually averages around 270 to 280 per 24hr period, or
> > > around 12/hr.  The messages in the logs at Cornell (accessible from
> > > Unidata's web page) are a little more informative than ours are.  There's
> > > a lot of lines like this (I've edited out line not relevant to our server
> > > vortex):
> > 
> > Tom,
> > 
> > That's a lot of disconnects. It's also the reason for the late data,
> > surprizing that your site is not loosing more data. Here's what I think
> > what is happening. Before, NOAAport Brockport was almost running close to
> > it's network capacity, now it's saturated. That's the reason for the
> > disconnect( time outs) and the LDM RECLASS statements. I would contact
> > your network people to see if they can arrange a solution and further
> > restrict the incoming data.  You also could check the status of your
> > machine to see if it cpu loaded, disk loaded, etc.  
> 
> 
> Robb,
> 
> Here's what I've found regarding this problem:
> 
> 1.  CPU and disk are not loaded, so I ruled this out immediately.
> 
> 2.  Network saturation seemed a plausible diagnosis.  Failing over to
> SUNY Albany resulted in no improvement whatsover, which would be
> consistent with network saturation.
> 
> 3.  However, applying the Unidata filter for NOAAPORT made no difference
> either, in terms of the number of disconnects and delayed products, which
> was a bit surprising if it was a network capacity problem.
> 
> 4.  Last week I spoke with our network people about this problem.  They
> were skeptical it was a network capacity problem since semester had ended
> and very few people were at work.  They gave me a figure of 23% of
> capacity for outbound packets and 3% for inbound packets on 12/23/98.  As
> a test, on last Tues, 1/5/99, I changed back our REQUEST for UNIDATA to be 
> ".*" and we ran statistics for our campus internet connection (maximum
> utilization was 30%) and on the subnet our ldm server is on (maximum
> utilization was 9%).  We will have to see what happens when classes
> resume, but it seems unlikely that network saturation was the cause of the
> problem during the time period we are talking about.  (The semester had
> ended when NOAAPORT started.
> 
> 5.  Looking at Cornell's ldm logs, I noticed that in almost every case,
> the RPC timeout messages that preceded the disconnects referred to
> products with headers beginning with "SDXX" (mainly), or else "SDUS".
> I then did a watch on the pq, and you could see that at certain times the
> only things being received were these 2 products, nothing else was getting
> through.  So I added a pattern to the Unidata filter REQUEST statement
> which excluded products beginning with "SD".  The results were immediate:
> the METARs were received on time and all the other delayed products, such
> as models, were received on time as well.  To confirm that the "SD"
> products were the culprit, I changed our REQUEST statement for all UNIDATA
> streams except HDS to be ".*" and for HDS everything except products
> beginning with "SD".  We ran with this for several days last week.  We
> received all products on time.  The number of disconnects from Cornell
> went down to about 2/hr on average.  Since then I've further refined the
> HDS REQUEST to exclude other products that we aren't currently using such
> as ^P, ^AG, ^IS, ^VE, etc.  Since none of our active downstreams sites is
> requesting HDS, I didn't see a problem with this.  We're now averaging
> about 1 disconnect per hour.
> 
Tom,

That's great research and interesting results, I'll remember it if any
other site have similar problems. It still implies network
saturation at some point in your connection or a router problem. I did a
notifyme:

wcfields.unidata.ucar.edu.rkambic> notifyme -vl - -f HDS -p "^SD" \
-h zero.unidata.ucar.edu
 

The results didn't show much.  Average product size is 1-2 k and the
number of products/min was low, ranging from 1-20.  There is no way that
data rate would saturate a network.  The only guess and it's a far one,
would be coincidence of receiving a SD(XX|US) product an a saturation
point. No other sites have problems receiving these products



> So I guess my question is: do you have any ideas why we would be having
> such a problem receiving these "SD.." products?  I'm sort of surprised
> that we seem to be the only site experiencing this.  Not being able to
> receive them is not a problem currently, but if in the future these Radar
> Coded Messages were to be supported by gempak, we might want to get them.

I wouldn't worry about that now. It seems that John Steer(sp) and Dewain
Wallace are try to get Brockport on the vBNS.  I would talk to them, they
seemed a little confused about the data rates needed for feeds. Keep me
inform of any other events that you solve or notice.

Thanks,
Robb...


> 
> Info about our server:  SparcStation 10 with dual 75mhz processors, 256MB,
> running Solaris 2.6 and ldm 5.0.6.
> 
> Tom 
> ------------------------------------------------------------------------------
> Tom McDermott                         Email: address@hidden
> System Administrator                  Phone: (716) 395-5718
> Earth Sciences Dept.                  Fax: (716) 395-2416
> SUNY College at Brockport
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================