Re: 19991103: LDM 5.0.8 and 'expanding input buffer size' problemwithAFOS feed

Thanks for the patch!  LDM has been happily ingesting AFOS data since late 
yesterday afternoon.

Gregg

Robb Kambic wrote:

> Gregory,
>
> I actually found the patch file, added it as a attachment. Here's the tag
> on how to apply it:
>
> http://www.unidata.ucar.edu/glimpse/ldm/2821
>
> According to the patch, pqing.c 1.66 should work, then the PIL code can be
> added. I'll attach 1.66
>
> I forgot about David Copley problem, here's logs for pqing.c. It looks
> like the PIL code was added after NMC2 stuff.
>
> ----------------------------
> revision 1.68
> date: 1999/06/17 16:11:07;  author: steve;  state: Exp;  lines: +3 -2
> Replaced missing "#include <ldmconfig.h>" directive that was removed at
> the last checkin.
> ----------------------------
> revision 1.67
> date: 1999/06/03 21:12:38;  author: rkambic;  state: Exp;  lines: +27 -8
> PIL header addition
> ----------------------------
> revision 1.66
> date: 1999/02/09 03:17:09;  author: davis;  state: Exp;  lines: +6 -6
> Fix CONDUIT hack.
> ----------------------------
> revision 1.65
> date: 1998/10/16 19:28:49;  author: steve;  state: Exp;  lines: +3 -2
> Ported to HP-UX B.11.00.
> Modified configuration mechanism: now uses configuration file
>     "ldmconfig.h.in".
> ----------------------------
>
> You might have add the line to pqing.c:
>
> #include <ldmconfig.h>
>
> Robb...
>
> On Thu, 4 Nov 1999, Gregory Grosshans wrote:
>
> > Robb,
> >
> > I tried a feedtest and received the following:
> >
> > bin/feedtest -vxl - -b 9600 -p none -f AFOS /dev/ttyC13
> > Nov 04 21:09:31 feedtest[23658]: Starting Up
> >         $Revision: 1.68 $ built Aug 13 1999 11:23:12
> > Nov 04 21:09:31 feedtest[23658]: TERMIOS "/dev/ttyC13": 9600 baud, none 
> > parity
> >         garbage encountered while searching for header
> > Nov 04 21:10:03 feedtest[23658]: Expanding input buffer size to 20480
> > Nov 04 21:10:13 feedtest[23658]: Expanding input buffer size to 24576
> > Nov 04 21:10:16 feedtest[23658]: Interrupt
> > Nov 04 21:10:16 feedtest[23658]: Exiting
> > leslie 47:
> >
> > So I looked through the Unidata searchable web pages using 'afos'.  I found 
> > some email from
> > this past February between Unidata and David Copley.  He appeared to have 
> > the same problem
> > as I am only he was using 5.0.6.
> >
> > I called David and he said that Unidata noticed a bug in pqing.c and that 
> > Unidata provided a
> > patch and then LDM ingested AFOS
> > data just fine.  Looking at some other email it appears pqing was updated 
> > to support the
> > CONDUIT project.  Can you provide me the patch necessary for pqing to 
> > accept AFOS data
> > instead of the CONDUIT NMC2 data?
> >
> > Thanks,
> > Gregg
> >
> >
> > Robb Kambic wrote:
> >
> > > Gregory,
> > >
> > > I don't have any good solutions to your problem.  I created an source code
> > > release for ldm-5.0.2 in the standard ldm source ftp directory.  You could
> > > compile the source, test it and then modify the PIL fixes to the code. One
> > > word of caution, save all the other working releases so no overwrites
> > > occur.  At this time, I can't really expend any more time because AFOS
> > > stream is not really used in the IDD. If I think of any new ideas about
> > > the problems, I'll let you know.  I had numerous problems with HPUX, if
> > > fact I can't get it compiled on HPUX 11.0 because of a bug in the rpc
> > > library. For HPUX 11.0 users, they have to use the version compiled on
> > > HPUX 10.2   I sympathize with your situation, it's frustrating.
> > >
> > > Robb...
> > >
> > > On Thu, 4 Nov 1999, Gregory Grosshans wrote:
> > >
> > > > I'm running LDM 5.0.2, and trying to get 5.0.8 to work on an HP 712 
> > > > HPUX 10.20
> > > > machine.  Up until this past spring the LDM was running on an HP 715 
> > > > with HPUX 9.x.
> > > > The feeds into this machine are AFOS, DDS and PPS via serial 
> > > > connections.
> > > >
> > > > I just grabbed the HPUX 10.2 LDM 5.0.8 binaries off the web site and 
> > > > installed them and
> > > > set the queue at 100 MB.  Again I received
> > > > the "Expanding input buffer size" problem.
> > > >
> > > > I've attached ldmd.log.1 and ldmd.log.2.  ldmd.log.1 was created when I 
> > > > tried to run
> > > > the 5.0.8 binaries and you will notice the
> > > > expanding buffer size messages.  The interesting item at the end of 
> > > > this log is under
> > > > afos it lists 'nregions    35'.  While the PPS and
> > > > DDS feeds don't list an nregions at all.  Then if you look at the end 
> > > > of ldmd.log.2
> > > > (created from 5.0.2) under AFOS it doesn't list
> > > > 'nregions...'.    Do you know what causes LDM to generate this message?
> > > >
> > > > Outside of just a few HP's with Mux ports the other alternatives are 
> > > > configuring a
> > > > linux box.   Operations is just an HP shop.
> > > >
> > > > The router configuration would involve NCEP HQ so I'd like to wait 
> > > > until absolutely
> > > > necessary to go that route.
> > > >
> > > > Gregg
> > > >
> > > >
> > > >
> > > > Robb Kambic wrote:
> > > >
> > > > > Gregory,
> > > > >
> > > > > At this point, I'm really starting to speculate. Is it possible to 
> > > > > install
> > > > > the ldm on another box that's not IRIX?  My thinking is some 
> > > > > library/etc
> > > > > has changed to make the ldm behave abnormal. If we could duplicate 
> > > > > this on
> > > > > another platform then it's definitely a bug in the ldm.  Could be OS
> > > > > specific.
> > > > >
> > > > > Robb...
> > > > >
> > > > > On Thu, 4 Nov 1999, Gregory Grosshans wrote:
> > > > >
> > > > > > On the machine in question the only feeds are DDS, PPS and AFOS.  
> > > > > > On this machine
> > > > > > I've had a LDM queue of 25 MB for three years and it worked fine 
> > > > > > with all three
> > > > > > feed types.
> > > > > >
> > > > > > There is a seperate machine with LDM ingesting NOAAPORT and on this 
> > > > > > machine the
> > > > > > queue is 700+ MB since I'm ingesting three
> > > > > > channels of NOAAPORT data.
> > > > > >
> > > > > > I've actually set the size of the ldm queue in the 
> > > > > > $LDMHOME/bin/ldmadmin script to
> > > > > > 250000000.  I've verified several times that the queue is of 
> > > > > > sufficient length.
> > > > > > When I switch back to 5.0.2 I change the queue back to 25 MB and 
> > > > > > everything works
> > > > > > fine.
> > > > > >
> > > > > > I tried commenting out the PPS and DDS feed types, leaving only 
> > > > > > AFOS on and
> > > > > > starting 5.0.8 and still received the 'expanding input buffer 
> > > > > > size'.  While LDM
> > > > > > was running with just the AFOS feed I tried pqcat.  PQCAT started 
> > > > > > and exited right
> > > > > > away and indicated the number of products were 0.
> > > > > >
> > > > > > Gilbert, If I were able to have the routers and firewall changed to 
> > > > > > allow your LDM
> > > > > > access to this machine I don't understand how the AFOS
> > > > > > data would be able to get to you.  Right now there are no AFOS 
> > > > > > products entering
> > > > > > the queue, the only thing occurring is the 'Expanding input buffer 
> > > > > > size.'  If
> > > > > > there were AFOS products entering the queue they would also show up 
> > > > > > in one of the
> > > > > > three other LDM machines here.
> > > > > >
> > > > > > Actually, that may be a key, the buffer is expanding.  Robb, do you 
> > > > > > know how the
> > > > > > buffer plays into the LDM?   Does LDM load a product
> > > > > > into a new dynamic buffer prior to placing it in the queue and then 
> > > > > > does it free
> > > > > > the memory?\
> > > > > >
> > > > > > Thanks for your timely responses.
> > > > > >
> > > > > > Gregg
> > > > > >
> > > > > >
> > > > > > Gilbert Sebenste wrote:
> > > > > >
> > > > > > > On Thu, 4 Nov 1999, Robb Kambic wrote:
> > > > > > >
> > > > > > > > Gregory,
> > > > > > > >
> > > > > > > > After looking at the log messages, it doesn't appear your queue 
> > > > > > > > is 250
> > > > > > > > megabytes.  Look in the data dir, what's the:
> > > > > > > >
> > > > > > > > % ls -l ldm.pq
> > > > > > > >
> > > > > > > > If it's 250 megabytes then up it again. There might be other 
> > > > > > > > products
> > > > > > > > coming in on your afos stream, etc.  If it isn't then change it 
> > > > > > > > in the
> > > > > > > > bin/ldmadmin file to 250.  Then the standard
> > > > > > > >
> > > > > > > > % ldmadmin stop
> > > > > > > > % ldmadmin delqueue
> > > > > > > > % ldmadmin mkqueue
> > > > > > > > % ldmadmin start
> > > > > > > >
> > > > > > > > I just put the commands here for my sanity sake.
> > > > > > >
> > > > > > > DUH! I forgot he was also ingesting NOAAPORT. Whoa, that's 175 MB 
> > > > > > > right
> > > > > > > there. Up, up and away! Use 400 MB at least.
> > > > > > >
> > > > > > > > Also, if you comment out the dds and pps does it make any 
> > > > > > > > difference?
> > > > > > > >
> > > > > > > > Robb...
> > > > > > >
> > > > > > > I think that's the key here. NOAAPORT takes up 150 MB of my 
> > > > > > > queue/hour,
> > > > > > > and of course, we filter out scrambled NIDS and other stuff. Give 
> > > > > > > it a go,
> > > > > > > and then let us know!
> > > > > > >
> > > > > > > *******************************************************************************
> > > > > > > Gilbert Sebenste                                                  
> > > > > > >    ********
> > > > > > > Internet: address@hidden    (My opinions only!)                   
> > > > > > >   ******
> > > > > > > Staff Meteorologist, Northern Illinois University                 
> > > > > > >      ****
> > > > > > > Work phone: 815-753-5492                                          
> > > > > > >       ***
> > > > > > > *******************************************************************************
> > > > > > >
> > > > > >
> > > > >
> > > > > ===============================================================================
> > > > > Robb Kambic                                Unidata Program Center
> > > > > Software Engineer III                      Univ. Corp for Atmospheric 
> > > > > Research
> > > > > address@hidden                   WWW: http://www.unidata.ucar.edu/
> > > > > ===============================================================================
> > > >
> > >
> > > ===============================================================================
> > > Robb Kambic                                Unidata Program Center
> > > Software Engineer III                      Univ. Corp for Atmospheric 
> > > Research
> > > address@hidden                   WWW: http://www.unidata.ucar.edu/
> > > ===============================================================================
> >
>
> ===============================================================================
> Robb Kambic                                Unidata Program Center
> Software Engineer III                      Univ. Corp for Atmospheric Research
> address@hidden                   WWW: http://www.unidata.ucar.edu/
> ===============================================================================
>
>   ------------------------------------------------------------------------
>                     Name: pqing.diff
>    pqing.diff       Type: Plain Text (TEXT/PLAIN)
>                 Encoding: BASE64
>              Description: pqing.diff
>
>                  Name: pqing.c
>    pqing.c       Type: Plain Text (TEXT/PLAIN)
>              Encoding: BASE64
>           Description: pqing.c 1.66

begin:vcard 
n:Grosshans;Gregory
tel;fax:405-579-0700
tel;work:405-579-0720
x-mozilla-html:TRUE
url:www.spc.noaa.gov
org:DOC/NOAA/NWS/NCEP/Storm Prediction Center
adr:;;1313 Halley Circle                                ;Norman;OK;73069;
version:2.1
email;internet:address@hidden
x-mozilla-cpt:;0
fn:Gregory Grosshans
end:vcard