[LDM #UKU-190526]: LDM - PIPE usage in pqact.conf and bash scripts

Hi Mike,

re:
> I honestly don't know where to go next except to bed.  I've included
> log files from LDM logs, from the ldmfile3.bash script and the contents
> of ldmadmin watch screen.  You probably need to see the pqact.conf
> file, too, but there's a limit to the size.  Let me know.

There was no log file included with your last email, but I don't think
it is needed.  Please see below.

> Everything looks OK from the logs.  All the permissions are set
> correctly, i.e., for all users.  Copies of the required files and
> programs are in the directories: ncdump, pipencdump (the mkfifo file),
> etc.  I'm obviously receiving ingest products but nothing is being
> saved to the directories e.g.,
> /usr/local/ldm/MADIS_data/point/metar/netcdf et al.

OK.

> If products are ingested where are they going?  Maybe Einstein was
> wrong: there IS an ether and my files are disappearing into it!!

If the version of ldmfile3.sh is the same as was included in your
email of yesterday afternoon, then there are two errors in the script
which were likely my fault (sorry).  Both errors are mismatched quotes in
the 'if' constructs:

# Write stdin to the designated file
gzip -d > $DIRS/pipencdump
if [ $? != 0 ]; then
  echo $(date -u +'%b %d %T') $(basename $0)\[$$\]: Error gunzipping input, exit
 status = $?"
  exit
fi

# Create an ASCII version of the netCDF
ncdump $DIRS/pipencdump > $1
if [ $? != 0 ]; then
  echo $(date -u +'%b %d %T') $(basename $0)\[$$\]: Error writing ncdump output,
 exit status = $?"
  exit
fi

i.e.:
echo $(date -u +'%b %d %T') $(basename $0)\[$$\]: Error gunzipping input, exit 
status = $?"

should be:

echo "$(date -u +'%b %d %T') $(basename $0)\[$$\]: Error gunzipping input, exit 
status = $?"

and:

 echo $(date -u +'%b %d %T') $(basename $0)\[$$\]: Error writing ncdump output, 
exit status = $?"

should be:

echo "$(date -u +'%b %d %T') $(basename $0)\[$$\]: Error writing ncdump output, 
exit status = $?"

The result of the first mismatched quote will cause ncdump to not be run
since it will look like it is part of a big string.

Also, it will appear that nothing is running since the result of the 'gzip -d',
the file pipencdump, gets removed at the last step.

A corrected version of the script is:

#!/bin/bash

#---------------------------------------------------------------------
#
# Name: ldmfile.bash
#
# Purpose: file an LDM product and log the receipt of the product
# Note: modify the "log" file for a specific use!
#
# History: 20070412 - Created from example Bourne shell script
# provided by Unidata
# 20070416 - Modified by Tom at Unidata
# 20070418 - Modified by Mike at Industrial Weather
#------------------------------------------------------------------------

# set log file
LOG=/usr/local/ldm/MADIS_data/logs/MADIS_log
exec >>$LOG 2>&1

# Create directory structure
FNAME=$(basename $1)
DIRS=$(echo $1 | sed s/$FNAME//)
mkdir -p $DIRS

# Write stdin to the designated file
gzip -d > $DIRS/pipencdump
if [ $? != 0 ]; then
  echo "$(date -u +'%b %d %T') $(basename $0)\[$$\]: Error unzipping input, 
exit status = $?"
  exit
fi

# Create an ASCII version of the netCDF
ncdump $DIRS/pipencdump > $1
if [ $? != 0 ]; then
  echo "$(date -u +'%b %d %T') $(basename $0)\[$$\]: Error writing ncdump 
output, exit status = $?"
  exit
fi

# Success
echo $(date -u +'%b %d %T') $(basename $0)\[$$\]: FILE $1
rm -f $DIRS/pipencdump

# done
exit

 
I tested this script (logging to a different log file) using:

cat 19982991906.nc.gz | ./ldmfile3.sh /local/ldm/data/mike/MADIS_data

'19982991906.nc.gz' is a gzip-compressed netCDF file that I had lying around;
it does not contain MADIS data, but that is not important since your script
doesn't (yet) care what is in the gzipped netCDF file.

By the way, what is 'Industrial Weather'?  I have been under the impression that
you were doing this work as part of your position at Champlin Park Hight School.

Cheers,

Tom
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: UKU-190526
Department: Support LDM
Priority: Normal
Status: Closed