Re: 19991103: LDM 5.0.8 and 'expanding input buffer size' problemwithAFOS feed

Thanks for the input Gilbert

Robb...



On Thu, 4 Nov 1999, Gilbert Sebenste wrote:

> Hey Robb and Greg,
> 
> > > I should also specify that LDM isn't seeing any AFOS products, the buffer 
> > > is
> > > increasing.  I've seen the buffer
> > > increase before on other data feeds while LDM was still ingesting data.  
> > > That is, you
> > > can see the data coming
> > > in via 'ldmadmin watch'.
> > > >
> > > > Gregory,
> > > >
> > > > This is strange, the only thing I can think of is that the 5.0.8 version
> > > > is receiving more AFOS data.  The irix system sometimes get into 
> > > > confused
> > > > states when the queue is expanded.  I would make the queue 150 megabytes
> > > > and try again. If the message still appear, make the queue large until 
> > > > the
> > > > messages disappear.  The comments in RCS for the afos_message.c state 
> > > > that
> > > > the routines where made more lenient to allow more products to be
> > > > accepted.
> > > >
> > > > Robb...
> 
> I assume you are getting the full bandwidth of AFOS via AWIPS.
> 
> Indeed, I had placed a request in to allow the LDM to be able to handle
> the products better...and it inadvertantly allowed more products to come
> through. I, too, had to increase my queue...right now, I am just getting
> the NOAA Wather Wire Service, as paranoid mode has struck the Chicago NWS
> and they won't let anyone through their firewall anymore. If you can set
> me up with a test feed, I can check on it here (my machine is
> weather3.admin.niu.edu).
> 
> What I suspect is happening is what Robb mentioned above. Presently, I
> have my machine configured so that my queue is 250 MB. Here's a sample out
> of my pqact.conf of how I am saving the data (see below for more
> comments):
> 
> #
> #
> #                   A F O S  S E C T I O N
> #
> #
> #-----------------------------------------------------------------------------
> # State Zone Forecasts by Specific State (Illinois).
> AFOS  ^CHIZFPIL(.*) ([0-3][0-9])([0-2][0-9])
>       FILE    domestic/state_zone_forecasts/IL.(\2:mmm)\2.tmp
> AFOS  ^CHIZFPIL(.*) ([0-3][0-9])([0-2][0-9])
>       EXEC    prepend_file domestic/state_zone_forecasts/IL.(\2:mmm)\2
> #-----------------------------------------------------------------------------
> # State Forecasts by Specific State (Illinois).
> AFOS  ^CHISFPIL(.*) ([0-3][0-9])([0-2][0-9])
>       FILE    domestic/databases/FPUS01.(\3:yy)(\3:mmm)\3
> AFOS  ^CHISFPIL(.*) ([0-3][0-9])([0-2][0-9])
>       EXEC    make_index domestic/databases/FPUS01.(\3:yy)(\3:mmm)\3 \2 
> (\3:yy)(\3:mmm)\3\4
> #-----------------------------------------------------------------------------
> # NOWCAST for Chicago, Illinois.
> AFOS  ^CHINOWCHI ([0-3][0-9])([0-2][0-9])
>       FILE    domestic/databases/FXUS21.(\3:yy)(\3:mmm)\3
> AFOS  ^CHINOWCHI ([0-3][0-9])([0-2][0-9])
>       EXEC    make_index domestic/databases/FXUS21.(\3:yy)(\3:mmm)\3 \2 
> (\3:yy)(\3:mmm)\3\4
> #-----------------------------------------------------------------------------
> # State Weather Roundup for Colorado.
> AFOS  ^DENSWRCO(.*) ([0-3][0-9])([0-2][0-9])
>       FILE    domestic/state_roundups/CO.(\2:mmm)\2\3
> #-----------------------------------------------------------------------------
> # Severe storm outlook narratives (day 1, day 2, and mesoscale).
> AFOS  ^MKCSWODY1(.*) ([0-3][0-9])([0-2][0-9])
>       FILE    domestic/convective/ACUS01.(\2:mmm)\2
> AFOS  ^MKCSWODY1(.*) ([0-3][0-9])([0-2][0-9])
>       EXEC    make_index domestic/convective/\1.(\2:mmm)\2 KMKC  
> (\2:yy)(\2:mmm)\2\3
> AFOS  ^MKCSWODY2(.*) ([0-3][0-9])([0-2][0-9])
>       FILE    domestic/convective/ACUS02.(\2:mmm)\2
> AFOS  ^MKCSWODY2(.*) ([0-3][0-9])([0-2][0-9])
>       EXEC    make_index domestic/convective/\1.(\2:mmm)\2 KMKC  
> (\2:yy)(\2:mmm)\2\3
> AFOS  ^MKCMCD(.*) ([0-3][0-9])([0-2][0-9])
>       FILE    domestic/convective/ACUS03.(\2:mmm)\2
> AFOS  ^MKCMCD(.*) ([0-3][0-9])([0-2][0-9])
>       EXEC    make_index domestic/convective/\1.(\2:mmm)\2 KMKC  
> (\2:yy)(\2:mmm)\2\3
> #-----------------------------------------------------------------------------
> # Tornado warnings.
> AFOS  ^(.*)TOR(.*) ([0-3][0-9])([0-2][0-9])
>       FILE    domestic/databases/WF.(\3:yy)(\3:mmm)\3
> AFOS  ^(.*)TOR(.*) ([0-3][0-9])([0-2][0-9])
>       EXEC    make_index domestic/databases/WF.(\3:yy)(\3:mmm)\3 \2 
> (\3:yy)(\3:mmm)\3\4
> #Severe thunderstorm and tornado watches.
> AFOS  ^MKCSEL(.*) ([0-3][0-9])([0-2][0-9])
>       FILE    domestic/watches/severe_watch.(\3:mmm)\3
> AFOS  ^MKCSEL(.*) ([0-3][0-9])([0-2][0-9])
>       EXEC    make_index domestic/watches/severe_watch.(\3:mmm)\3 \2 
> (\3:yy)(\3:mmm)\3\4
> #Flash flood statements.
> AFOS  ^(.*)FFS(.*) ([0-3][0-9])([0-2][0-9])
>       FILE    domestic/watches/flood_stmt.(\3:mmm)\3
> AFOS  ^(.*)FFS(.*) ([0-3][0-9])([0-2][0-9])
>       EXEC    make_index domestic/watches/flood_stmt.(\3:mmm)\3 \2 
> (\3:yy)(\3:mmm)\3\4
> 
> -----------------------------------------------------------------------------
> As you can see, the additional ([0-3][0-9])([0-2][0-9]) comes from the
> fact that the second line of the AFOS PIL is put on the first...which lets
> me be much more flexible in what I can do in naming the file. If, however,
> you don't have that line in there...put a (.*) and that should take care
> of it.
> 
> The problem is that there are also inconsistencies, even among SPC
> products. The ACUS1 and 2 products have the second line, while the meso
> discussions do not. As a result, they get saved in a different filename,
> despite the similar entry of those of the ACUS1 and 2.
> 
> Try 250 MB as the queue...and then go from there. Like I said, I can try
> it to see if it works on mine...I am using Linux Redhat 6.0 on a PII 450
> MHZ machine on a partial T3, so I can handle it here.
> 
> *******************************************************************************
> Gilbert Sebenste                                                     ********
> Internet: address@hidden    (My opinions only!)                     ******
> Staff Meteorologist, Northern Illinois University                      ****
> Work phone: 815-753-5492                                                ***
> *******************************************************************************
>  
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================