[LDM #QWL-862656]: Script to pqinsert files not working

Evan,

> I wrote a Perl script to use with my LDM to automatically insert files
> into the product queue. The script uses the Perl Inotify2 module to
> watch a specified list of directories. When a file is (completely)
> written in one of these directories, a routine is run to insert the
> corresponding file into the product queue once the file is closed. The
> watch operations and file insertions are logged to a file. Here is the
> script:
> 
> #!/usr/bin/perl -w
> 
> # ldm_notify.pl
> #
> # Script to watch specified directories and insert new complete
> # files into LDM product queue.
> 
> use strict;
> use warnings;
> use diagnostics;
> use Linux::Inotify2;
> 
> $|++;
> 
> my $inotify = new Linux::Inotify2 or die "Unable to create new inotify
> object: $!";
> 
> my $logfile = "/home/ldm/logs/pqinsert.log";
> my @time = localtime;
> 
> my $inotify = new Linux::Inotify2 or die "Unable to create new inotify
> object: $!";
> 
> my $logfile = "/home/ldm/logs/pqinsert.log";
> my @time = localtime;
> 
> my $path = "/home/ldm/temp/";
> my @site_array = ("ktst1", "ktst2");
> my $site_cnt = 0;
> 
> foreach my $site (@site_array) {
> 
> my $path_site = $path . $site;
> $inotify->watch($path_site, IN_CREATE, sub {
> 
> my $event = shift;
> my $name = $event->fullname;
> if (!($name =~m/.filepart/)) {
> 
> my $pqmessage = qx("pqinsert $name") if
> $event->IN_CLOSE;
> open Wlog, ">>$logfile";
> if (!($pqmessage)) {
> print Wlog "Inserted: $name\n";
> }
> else {
> print Wlog "$pqmessage\n";
> }
> close Wlog;
> }
> 
> }) or die "Watch creation failed: $!";
> 
> if (!($site_cnt)) {
> open Wlog, ">$logfile";
> printf Wlog "%4d-%02d-%02d %02d:%02d:%02d\n",
> $time[5]+1900, $time[4]+1, $time[3], $time[2], $time[1], $time[0];
> }
> else {
> open Wlog, ">>$logfile";
> }
> 
> print Wlog "Watch established on: $path_site\n";
> close Wlog;
> $site_cnt++;
> }
> 
> 1 while $inotify->poll;
> 
> I call this script using an EXEC entry in ldmd.conf:
> 
> # Execute product queue insertion script
> EXEC    "perl /home/ldm/etc/ldm_notify.pl"
> 
> I check the script is running by issuing a "ps -eaf | grep ldm" command
> after starting the LDM and it shows the script was run:
> 
> [ldm@frtt1 etc]$ ps -eaf | grep ldm
> root     12048  2223  0 Feb21 ?        00:00:00 sshd: ldm [priv]
> ldm      12055 12048  0 Feb21 ?        00:00:01 sshd: ldm@pts/1
> ldm      12058 12055  0 Feb21 pts/1    00:00:00 -bash
> root     12294  2223  0 Feb21 ?        00:00:00 sshd: ldm [priv]
> ldm      12300 12294  0 Feb21 ?        00:00:11 sshd: ldm@notty
> ldm      12303 12300  0 Feb21 ?        00:00:01
> /usr/libexec/openssh/sftp-server
> root     13265  2223  0 00:04 ?        00:00:00 sshd: ldm [priv]
> ldm      13271 13265  0 00:04 ?        00:00:00 sshd: ldm@pts/2
> ldm      13274 13271  0 00:04 pts/2    00:00:00 -bash
> ldm      14499     1  0 14:56 ?        00:00:00 rpc.ldmd -P 388 -q
> /home/ldm/data/ldm.pq /home/ldm/etc/ldmd.conf
> ldm      14501 14499  1 14:56 ?        00:00:00 pqact
> ldm      14502 14499  1 14:56 ?        00:00:00 rtstats -h
> rtstats.unidata.ucar.edu
> ldm      14503 14499  1 14:56 ?        00:00:00 perl
> /home/ldm/etc/ldm_notify.pl
> ldm      14504 14499  0 14:56 ?        00:00:00 rpc.ldmd -P 388 -q
> /home/ldm/data/ldm.pq /home/ldm/etc/ldmd.conf
> ldm      14505 14499  0 14:56 ?        00:00:00 rpc.ldmd -P 388 -q
> /home/ldm/data/ldm.pq /home/ldm/etc/ldmd.conf
> ldm      14506 14499  0 14:56 ?        00:00:00 rpc.ldmd -P 388 -q
> /home/ldm/data/ldm.pq /home/ldm/etc/ldmd.conf
> ldm      14507 14499  0 14:56 ?        00:00:00 rpc.ldmd -P 388 -q
> /home/ldm/data/ldm.pq /home/ldm/etc/ldmd.conf
> ldm      14508 14499  0 14:56 ?        00:00:00 rpc.ldmd -P 388 -q
> /home/ldm/data/ldm.pq /home/ldm/etc/ldmd.conf
> ldm      14509 14499  0 14:56 ?        00:00:00 rpc.ldmd -P 388 -q
> /home/ldm/data/ldm.pq /home/ldm/etc/ldmd.conf
> ldm      14510 14499  0 14:56 ?        00:00:00 rpc.ldmd -P 388 -q
> /home/ldm/data/ldm.pq /home/ldm/etc/ldmd.conf
> ldm      14511 12058  0 14:56 pts/1    00:00:00 ps -eaf
> ldm      14512 12058  0 14:56 pts/1    00:00:00 grep ldm
> 
> I then test the operation by copying a few of files into the watched
> directories. After this, the pqinsert.log file shows me:
> 
> 2009-02-22 14:56:24
> Watch established on: /home/ldm/temp/ktst1
> Watch established on: /home/ldm/temp/ktst2
> Inserted: /home/ldm/temp/ktst1/20080527-063022.cdf
> Inserted: /home/ldm/temp/ktst2/20080527-063924.cdf
> Inserted: /home/ldm/temp/ktst1/20080527-065139.cdf
> Inserted: /home/ldm/temp/ktst1/20080527-065506.cdf
> 
> I have another LDM set up on the same network to receive the EXP data
> from the LDM running this script. I added the appropriate ALLOW and
> REQUEST entries in the respective ldmd.conf files and tested transfer
> using the NEXRAD2 feed. I can see the the downstream LDM receives the
> NEXRAD2 feed and receives another EXP feed from the machine running this
> script via an "ldmadmin watch" command, but it doesn't receive the files
> I'm copying to the watched directories!
> 
> Can you please help me get this working?

You should verify that the data-products that the downstream LDM
requests are being inserted into the product-queue.  Compare the
log-file entry that the LDM on the upstream host makes in response
to the request by the downstream LDM with the output of an "ldmadmin
watch" for that type of data.  Verify that everything matches: that
the feedtypes match and that the product should be selected by the
regular expression and the timestamp interval.

> Many thanks,
> Evan

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: QWL-862656
Department: Support LDM
Priority: Normal
Status: Closed