Re: LDM write errors

On 29 Feb 2000, Ken Waters wrote:

> 
>      LDM gurus,
>      
>      We have problems with certain text products that seem to indicate that 
>      our EXEC'ed script gets fired off twice when a product is received 
>      once.  A look at the ldmd.log finds entries like:
>      
>      Feb 29 21:48:53 wwwldm notifyme[24340]:     8904 20000229214815.415    
>      AFOS 000  MEMFWFMRX 
>      Feb 29 21:48:54 wwwldm pqact[10524]: pipe_dbufput: 
>      -strip/home/ldm/processMEMFWFMRX write error 
>      Feb 29 21:48:54 wwwldm pqact[10524]: pipe_dbufput: 
>      -strip/home/ldm/processMEMFWFMRX write error 
>      

Ken,

Their are a couple of things you might want to change to help.  First the
above errors are caused by the PIPE action trying to write to the process
decoder.  Inspecting the process decoder, I didn't find any <STDIN>
filehandle reads, therefore the LDM is writing to the decoder but the
process decoder is not reading anything. The PIPE action should be changed
to an EXEC action or make the process decoder read the <STDIN> input.

From your pqact.conf file:

# Run the process script to do the proper actions for this new message

AFOS    ^(...)(...)(...|..)
        PIPE    -strip /home/ldm/process \1 \2 \3


With this type of pqact entry, a new process decoder is spawned for every
product because the parameters "\1 \2 \3" change for every product. The
LDM currently has a limit on the number of processes of 32:

/*
 * Tuning parameter: MAXENTRIES is the number of descriptors
 * you wish to allocate to this list. You need to leave enough around
 * for the error output and for rpc.
 */
#define MAXENTRIES 32

If estimate 2-3 products/second then there would be over 150 decoders
spawned / minute.  You could up the limit to 200 if your computer
platform/os can handle that many processes.  The downside is that your ldm
configuration would be no-standard.  The file to change is: pqact/filel.c
This is the reason that the process decoder is being spawned twice or more
for every product, the LDM was reaping the process decoder before it was
done.

Another solution:

You could read the product in your decoder, extract the header then do the
rest of the decoder processing.  That;s how the current UPC perl decoders
work. The major pro side is that there is only one decoder running.

Observation:

In the decoder process there are many "system("mv ... " type calls.  For
every system call a fork is done, this is time consuming and resourse
consuming to the LDM and OS.  It would be better to extract all the system
calls to another external script that could be run out of cron. If you
named your file something.new, then the external script could do the moves
external to the LDM. In my opinion this would be a much cleaner solution.

Robb...


>      Feb 29 21:51:54 wwwldm notifyme[24340]:     8904 20000229215116.736    
>      AFOS 000  MEMFWFMRX 
>      Feb 29 21:51:55 wwwldm pqact[10524]: pipe_dbufput: 
>      -strip/home/ldm/processMEMFWFMRX write error 
>      Feb 29 21:51:56 wwwldm pqact[10524]: pipe_dbufput: 
>      -strip/home/ldm/processMEMFWFMRX write error 
>      
>      Note that I get two errors for each receipt of the product.  This 
>      really louses up my script as it is rotating versions (1 through 9) 
>      and the two instances of the script fall over each other.  The result 
>      is instead of having versions 1-2-3-4-5-6-7-8-9 I end up with either 
>      2-4-6-8 or 1-3-5-7-9.  
>      
>      What I really need to know is what is causing these write errors.  The 
>      error seems to be product specific.  Namely it always happens to 
>      certain products (like MEMFWFMRX, in this case).
>      
>      I have attached some relevant files.
>      
>      Thanks for any help.
>      
>      Ken Waters
>      NWS Southern Region SSD
>      (817) 978-2671
> 

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================