[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20040115: pqact(1) PIPE problems

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: 20040115: pqact(1) PIPE problems
  • Date: Wed, 14 Jan 2004 13:35:59 -0700
Karen,

> To: address@hidden
> From: "Karen Cooper" <address@hidden>
> Subject: pqact PIPE problems
> Organization: National Severe Storms Laboratory
> Keywords: 200401141757.i0EHvIp2008899 LDM

The above message contained the following:

> Running LDM 6.0.14 on Linux (RedHat) Workstation 3.  I am getting 104 
> NEXRD2 (Level II radar data) feeds from OU.  I can receive all of them 
> and write them out to files with no problem.
> 
> However, if I try, in pqact, to PIPE them through to Harry Edmon's 
> nexradII_bz program for decompression I get error messages which 
> increase in number depending on how many radars I am trying to 
> decompress.

What, exactly, are the error messages?

> (I've written to Harry asking if he has any ideas about 
> what could be causing the problem).

I know nothing about the "nexradII_bz" LDM decoder.  Hopefully, either
Chiz or Tom will be able to help.

> I can get ~20 radars and only see this error occassionally, but if I up 
> it to 30 radars, the errors become almost constant and if I do all 
> 104..... the system locks up.  Best I can tell, the decompression alg is 
> trying to do a read of the data in the pipe, but it is empty.  This is 
> not an error message, as the read function is returning "0" indicating 
> it read 0 bytes.

The getbuf() function also returns 0 on an error or a (10 minute) timeout.

> I was wondering if there was possibly some limit on the number of pipes 
> that could be open, and/or if anyone else has had a simliar problem with 
> having many pipes open at once.

There is a limit on the number of open pipes: it is reached when the
pipe() function returns EMFILE in the function pipe_open() of the module
pqact/filel.c.  When this limit is reached, the system-error message
"pipe" should be logged and the least-recently-used pipe will be closed
in order to open a new pipe.  This could be the source of your problem
if the maximum number of open pipes on your system is too small.

You should be able to check the limit via the command "ulimit -a" (which
is a shell built-in).  On our Linux 2.4.22 (Fedora Core release 1)
system, the limit is 1024, which should be more than sufficient.

Please let us know what you discover.

> BTW, we do not use the "-close" option in the PIPE because the NEXRD2
> data arrives in chunks until a volume scan is complete.
> 
> -- 
> address@hidden
> Phone:405-366-0434    
> Cell:405-834-8559
> SAIC/Systems Analyst
> National Severe Storms Laboratory

Regards,
Steve Emmerson

>From address@hidden  Wed Nov 16 10:10:55 2005

Thanks, Tom.  And, yes, that helps a LOT!

-- 
Gerry Creager -- address@hidden
Texas Mesonet -- AATLT, Texas A&M University    
Cell: 979.229.5301 Office: 979.458.4020
FAX:  979.847.8578 Pager:  979.228.0173
Office: 903A Eller Bldg, TAMU, College Station, TX 77843