Re: 19991103: LDM 5.0.8 and 'expanding input buffer size' problemwithAFOS feed

Gregory,

After looking at the log messages, it doesn't appear your queue is 250
megabytes.  Look in the data dir, what's the:

% ls -l ldm.pq

If it's 250 megabytes then up it again. There might be other products
coming in on your afos stream, etc.  If it isn't then change it in the
bin/ldmadmin file to 250.  Then the standard

% ldmadmin stop
% ldmadmin delqueue
% ldmadmin mkqueue
% ldmadmin start

I just put the commands here for my sanity sake.

I assume you are using the same ldmd.conf file for the ldm-5.0.8 ? Are you
sure that the parity is corect for afos?

Nov 04 16:09:17 leslie dds[20383]: TERMIOS "/dev/ttyC11": 9600 baud, none
parity
Nov 04 16:09:17 leslie afos[20382]: TERMIOS "/dev/ttyC13": 9600 baud, none
parity
Nov 04 16:09:17 leslie pps[20384]: TERMIOS "/dev/ttyC15": 9600 baud, even
parity

Also, if you comment out the dds and pps does it make any difference?

Robb...


On Thu, 4 Nov 1999, Gregory Grosshans wrote:

> I increased the LDM queue to 250 MB with LDM 5.0.8 and LDM continues to not 
> see
> any AFOS products, but it does see DDS and PPS.  When I switched back to 
> 5.0.2 AFOS
> products were recognized right away.
> 
> I've attached the ldmd.log.1 file for you to look at.  You will notice LDM is 
> logging
> products from PPS and DDS but only continues to expand the input buffer
> size for the AFOS feed.
> 
> I noticed the 'Expanding input buffer size' is generated from pqing, in 
> particular
> fxbuf.c.  However, I haven't look enough at fxbuf.c or the rest of the pqing 
> code
> to understand whats happening.
> 
> Our AFOS feed is actually from our local AFOS.  We also have AWIPS, and I'm 
> ingesting
> the NOAAPORT data into another LDM server.  I'm trying to upgrade my LDM 
> servers
> to 5.0.8 to support the /p option and wanted to do the two machines that 
> ingest actual data
> feeds before I upgrade those machines that actually do something with the 
> data.
> 
> Any other suggestions?
> 
> Thanks,
> Gregg
> 
> Gilbert Sebenste wrote:
> 
> > Hey Robb and Greg,
> >
> > > > I should also specify that LDM isn't seeing any AFOS products, the 
> > > > buffer is
> > > > increasing.  I've seen the buffer
> > > > increase before on other data feeds while LDM was still ingesting data. 
> > > >  That is, you
> > > > can see the data coming
> > > > in via 'ldmadmin watch'.
> > > > >
> > > > > Gregory,
> > > > >
> > > > > This is strange, the only thing I can think of is that the 5.0.8 
> > > > > version
> > > > > is receiving more AFOS data.  The irix system sometimes get into 
> > > > > confused
> > > > > states when the queue is expanded.  I would make the queue 150 
> > > > > megabytes
> > > > > and try again. If the message still appear, make the queue large 
> > > > > until the
> > > > > messages disappear.  The comments in RCS for the afos_message.c state 
> > > > > that
> > > > > the routines where made more lenient to allow more products to be
> > > > > accepted.
> > > > >
> > > > > Robb...
> >
> > I assume you are getting the full bandwidth of AFOS via AWIPS.
> >
> > Indeed, I had placed a request in to allow the LDM to be able to handle
> > the products better...and it inadvertantly allowed more products to come
> > through. I, too, had to increase my queue...right now, I am just getting
> > the NOAA Wather Wire Service, as paranoid mode has struck the Chicago NWS
> > and they won't let anyone through their firewall anymore. If you can set
> > me up with a test feed, I can check on it here (my machine is
> > weather3.admin.niu.edu).
> >
> > What I suspect is happening is what Robb mentioned above. Presently, I
> > have my machine configured so that my queue is 250 MB. Here's a sample out
> > of my pqact.conf of how I am saving the data (see below for more
> > comments):
> >
> > #
> > #
> > #                   A F O S  S E C T I O N
> > #
> > #
> > #-----------------------------------------------------------------------------
> > # State Zone Forecasts by Specific State (Illinois).
> > AFOS    ^CHIZFPIL(.*) ([0-3][0-9])([0-2][0-9])
> >         FILE    domestic/state_zone_forecasts/IL.(\2:mmm)\2.tmp
> > AFOS    ^CHIZFPIL(.*) ([0-3][0-9])([0-2][0-9])
> >         EXEC    prepend_file domestic/state_zone_forecasts/IL.(\2:mmm)\2
> > #-----------------------------------------------------------------------------
> > # State Forecasts by Specific State (Illinois).
> > AFOS    ^CHISFPIL(.*) ([0-3][0-9])([0-2][0-9])
> >         FILE    domestic/databases/FPUS01.(\3:yy)(\3:mmm)\3
> > AFOS    ^CHISFPIL(.*) ([0-3][0-9])([0-2][0-9])
> >         EXEC    make_index domestic/databases/FPUS01.(\3:yy)(\3:mmm)\3 \2 
> > (\3:yy)(\3:mmm)\3\4
> > #-----------------------------------------------------------------------------
> > # NOWCAST for Chicago, Illinois.
> > AFOS    ^CHINOWCHI ([0-3][0-9])([0-2][0-9])
> >         FILE    domestic/databases/FXUS21.(\3:yy)(\3:mmm)\3
> > AFOS    ^CHINOWCHI ([0-3][0-9])([0-2][0-9])
> >         EXEC    make_index domestic/databases/FXUS21.(\3:yy)(\3:mmm)\3 \2 
> > (\3:yy)(\3:mmm)\3\4
> > #-----------------------------------------------------------------------------
> > # State Weather Roundup for Colorado.
> > AFOS    ^DENSWRCO(.*) ([0-3][0-9])([0-2][0-9])
> >         FILE    domestic/state_roundups/CO.(\2:mmm)\2\3
> > #-----------------------------------------------------------------------------
> > # Severe storm outlook narratives (day 1, day 2, and mesoscale).
> > AFOS    ^MKCSWODY1(.*) ([0-3][0-9])([0-2][0-9])
> >         FILE    domestic/convective/ACUS01.(\2:mmm)\2
> > AFOS    ^MKCSWODY1(.*) ([0-3][0-9])([0-2][0-9])
> >         EXEC    make_index domestic/convective/\1.(\2:mmm)\2 KMKC  
> > (\2:yy)(\2:mmm)\2\3
> > AFOS    ^MKCSWODY2(.*) ([0-3][0-9])([0-2][0-9])
> >         FILE    domestic/convective/ACUS02.(\2:mmm)\2
> > AFOS    ^MKCSWODY2(.*) ([0-3][0-9])([0-2][0-9])
> >         EXEC    make_index domestic/convective/\1.(\2:mmm)\2 KMKC  
> > (\2:yy)(\2:mmm)\2\3
> > AFOS    ^MKCMCD(.*) ([0-3][0-9])([0-2][0-9])
> >         FILE    domestic/convective/ACUS03.(\2:mmm)\2
> > AFOS    ^MKCMCD(.*) ([0-3][0-9])([0-2][0-9])
> >         EXEC    make_index domestic/convective/\1.(\2:mmm)\2 KMKC  
> > (\2:yy)(\2:mmm)\2\3
> > #-----------------------------------------------------------------------------
> > # Tornado warnings.
> > AFOS    ^(.*)TOR(.*) ([0-3][0-9])([0-2][0-9])
> >         FILE    domestic/databases/WF.(\3:yy)(\3:mmm)\3
> > AFOS    ^(.*)TOR(.*) ([0-3][0-9])([0-2][0-9])
> >         EXEC    make_index domestic/databases/WF.(\3:yy)(\3:mmm)\3 \2 
> > (\3:yy)(\3:mmm)\3\4
> > #Severe thunderstorm and tornado watches.
> > AFOS    ^MKCSEL(.*) ([0-3][0-9])([0-2][0-9])
> >         FILE    domestic/watches/severe_watch.(\3:mmm)\3
> > AFOS    ^MKCSEL(.*) ([0-3][0-9])([0-2][0-9])
> >         EXEC    make_index domestic/watches/severe_watch.(\3:mmm)\3 \2 
> > (\3:yy)(\3:mmm)\3\4
> > #Flash flood statements.
> > AFOS    ^(.*)FFS(.*) ([0-3][0-9])([0-2][0-9])
> >         FILE    domestic/watches/flood_stmt.(\3:mmm)\3
> > AFOS    ^(.*)FFS(.*) ([0-3][0-9])([0-2][0-9])
> >         EXEC    make_index domestic/watches/flood_stmt.(\3:mmm)\3 \2 
> > (\3:yy)(\3:mmm)\3\4
> >
> > -----------------------------------------------------------------------------
> > As you can see, the additional ([0-3][0-9])([0-2][0-9]) comes from the
> > fact that the second line of the AFOS PIL is put on the first...which lets
> > me be much more flexible in what I can do in naming the file. If, however,
> > you don't have that line in there...put a (.*) and that should take care
> > of it.
> >
> > The problem is that there are also inconsistencies, even among SPC
> > products. The ACUS1 and 2 products have the second line, while the meso
> > discussions do not. As a result, they get saved in a different filename,
> > despite the similar entry of those of the ACUS1 and 2.
> >
> > Try 250 MB as the queue...and then go from there. Like I said, I can try
> > it to see if it works on mine...I am using Linux Redhat 6.0 on a PII 450
> > MHZ machine on a partial T3, so I can handle it here.
> >
> > *******************************************************************************
> > Gilbert Sebenste                                                     
> > ********
> > Internet: address@hidden    (My opinions only!)                     ******
> > Staff Meteorologist, Northern Illinois University                      ****
> > Work phone: 815-753-5492                                                ***
> > *******************************************************************************
> >
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================