[LDM #BOP-725250]: LDM - write errors to processed data files; tried everything else!

Hi Mike,

> Institution: Champlin Park High School
> Package Version: LDM 6.5.0
> Operating System: Linux 2.18...  Fedora Core 5
> Hardware Information: dual processor Opteron 248\\\'s in network; 1 T RAID; 
> 2G RAM

> Inquiry: I must be haunted!  Everything's working jes' hunky-dory except I 
> continue to
> get these write errors.  I double-checked read/write/execute permissions; 
> checked spelling
> of file names; checked file locations but everything looks fine.

Consider the first FSL2 action:

FSL2    ^FSL\.CompressedNetCDF\.MADIS\.metar\.(.*)      PIPE    -close  
/usr/local/ldm/MADIS_data/store_MADIS.sh 
/usr/local/ldm/MADIS_data/point/metar/netcdf \1

Questions:

- does /usr/local/ldm/MADIS_data/store_MADIS.sh exist (silly, but I have to ask)

- is the execute permission set on /usr/local/ldm/MADIS_data/store_MADIS.sh

- is /usr/local/ldm/MADIS_data/point/metar/netcdf the name of a directory in 
which
  you want output written?  If yes, does the directory structure down to and 
including
  the final subdirectory (netcdf) exist AND is writable by the user running 
your LDM?

- what is the content of /usr/local/ldm/MADIS_data/store_MADIS.sh?  It could be 
that the
  script is trying to run one or more applications that are not being found in 
the PATH
  that is in scope when the script is run

The 'ERROR: pbuf_flush (6) write: Broken pipe' errors that I see in ldmd.log 
output suggest
that the script is exiting for whatever reason before the entire product has 
been read
from STDIN.  When this happens, pqact senses that the reader of the 
STDOUT-STDIN pipe
being used to feed the product to the decode action has gone away prematurely.  
It
will then try to execute the action once more.  If there is a problem with your
store_MADIS.sh script, or if the script can not write to the output directory, 
this
will result in a pair of errors for each product that pqact is trying to 
process (i.e.,
alot of log messages).

> Is it a configuration error in pqact.conf?  I read and re-read the "tab" 
> versus "space" material
> and I just can't see it on the screen.  The logs suggest it's a configuration 
> issue.  BUT
> pqact.conf looks ok.

I agree that your pqact.conf actions look OK.  Please be aware that you can 
always do a
gross check of pqact.conf actions using ldmadmin:

<as 'ldm'>
ldmadmin pqactcheck

> I'm sorry to keep bothering you guys; this seems like it should be so easy...

It should be easy to setup pqact.conf processing actions, but the situation gets
complicated when the task is some sort of a shell script (or Perl script) that
is expecting that programs it expects to run can be found by virtue of the
PATH that is in scope when the script is run.  Also, some users get confused
by the need to create the output directory hierarchy since a simple FILE action
will create subdirectories as needed.  If your output directories do not already
exist, the store_MADIS.sh script you are trying to run must be smart enough to
create the needed directories.

One last comment that has nothing to do with the problem you are seeing:

A bug was found in the FILE -overwrite action in LDM-6.5.0.  A beta version 
containing
a fix can be found in the pub/ldm/beta directory of anonymous FTP on our FTP 
server
ftp.unidata.ucar.edu.  It is likely that the beta version will be released as a 
supported
version tomorrow (after folks that have reported the problem have had 1 day of 
experience
running the beta). Since you are not subscribed to our 'ldm-users' email list 
(at least I
couldn't find your email address in the list of subscribed members), you would 
not have
heard about this problem.  I suggest that you subscribe to this list so you can
stay up to date wrt LDM use/problems/tips.

You can (un)subscribe to any Unidata-maintained email list online at:

http://www.unidata.ucar.edu/content/support/mailinglist/mailing-list-form.html

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: BOP-725250
Department: Support LDM
Priority: Normal
Status: Closed