[IDD #AIZ-345619]: LDM: pattern in ldmd.conf and pact.conf no longer works for CONDUIT??

Hi Christian,

re:
> I am ingesting data using LDM, and one of the feed I use is CONDUIT. I
> have been using it for years.  I am running ldm 6.13.10.  However,
> recently suddenly, I do not receive any CONDUIT products anymore.
> 
> Because of bandwidth limitations, I have a special CONDUIT regex request
> in ldmd.com.
> 
> I also tested with notifyme that I can receive the feed properly.
> 
> With notifyme:
> 
> notifyme -f CONDUIT -vl log -h iddb.unidata.ucar.edu
> I receive among all others, for example this header:
> 
> 20200715T121412.415671Z             notifyme[22262]
> notifyme.c:notifymeprog_5() INFO       40685 20200715102559.499768 CONDUIT 
> 283  data/nccf/com/gfs/prod/gfs.20200715/06/gfs.t06z.pgrb2.1p00.f237 
> !grib2/ncep/GFS/#000/202007150600F237/TMPK/700 hPa PRES! 000283
> 
> In my ldmd.conf, for CONDUIT (yes there is a tab character between request
> and CONDUIT):
> 
> request       CONDUIT
> "(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"
>  iddb.unidata.ucar.edu

OK, the first thing that I do to test extended regular expression(s) used
in LDM configuration file REQU$ST(s) and pattern-actoin file actions is use
'notifyme' to see if the REQUEST ERE will match anything in the feed.  For
instance:

notifyme -vl- -f CONDUIT -h iddb.unidata.ucar.edu -p 
"(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"
 -o 3600

I just ran this from the 'ldm' account on my personal development machine 
where I am running LDM v6.13.11, and the pattern matches lots of products.

re:
> In my pqact.conf (I have tabs after CONDUIT, and before & after FILE):
> CONDUIT         prod/gfs\.(........)/(..)/gfs.*pgrb2\.1p00\.f(.*) !grib2
> FILE    data/models/conduit/gfs/\1\2_\3_gfs.grib
> CONDUIT         prod/gfs\.(........)/(..)/gfs.*pgrb2\.1p00\.a.* !grib2
> FILE    data/models/conduit/gfs/\1_anl_gfs.grib

I just created the EXEC to run 'pqact' for CONDUIT products and an 
associated pattern-action file with these two actions as follows:

~ldm/etc/ldmd.conf:

EXEC    "pqact -f CONDUIT  etc/pqact.conf_conduit"

REQUEST CONDUIT 
"(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"
 iddb.unidata.ucar.edu

~ldm/etc/pqact.conf_conduit:

CONDUIT prod/gfs\.(........)/(..)/gfs.*pgrb2\.1p00\.f(.*) !grib2
        FILE    /data/ldm/pub/models/conduit/gfs/\1\2_\3_gfs.grib
CONDUIT prod/gfs\.(........)/(..)/gfs.*pgrb2\.1p00\.a.* !grib2
        FILE    /data/ldm/pub/models/conduit/gfs/\1_anl_gfs.grib

I cranked up my LDM, and I see files being created:

ls -alt /data/ldm/pub/models/conduit/gfs/
total 6920
-rw-r--r-- 1 ldm Unidata 6983837 Jul 15 10:42 2020071512_081_gfs.grib
drwxr-xr-x 2 ldm Unidata    4096 Jul 15 10:42 .
drwxr-xr-x 3 ldm Unidata    4096 Jul 15 10:42 ..
-rw-r--r-- 1 ldm Unidata   88554 Jul 15 10:42 2020071512_006_gfs.grib

Given that my tests worked, I don't know what could be causing the problem
in your setup.  The only things that I can think of are:

1) there is a space along with a tab between CONDUIT and the ERE or before/after
   FILE(s) in your pattern-action file

2) your LDM configuration file and/or pattern-action file was/were edited
   with a Windows editor (e.g., notepad) so the file is not in a *NIX
   format

Requests:

Can you send relevant lines from your LDM log file (feed starting up
line and any errors)?

Can you run the 'notifyme' that I included above and send the first "few"
(say 30 or so) lines of output (please don't send a large listing as it
is not needed).

re:
> But nothing gets through. I checked with ldmadmin watch -f CONDUIT and
> I get no headers. So it points out a problem in ldmd.conf, but I cannot
> figure it out, as I tried many variations in the regex. I also checked
> the regex and it is validated.

The output of the 'notifyme' command that I included above should affirm/
refute if you are receiving the products... I have seen instances where
an 'ldmadmin watch' doesn't show receipt of data while a 'notifyme' does
and visa versa.

re:
> I am also trying to get the status but it does not work (no output or even
> directory created):
> In ldmd.conf:
> 
> request         CONDUIT "status"        idd.ssec.wisc.edu
> 
> In pqact.conf:
> CONDUIT         ^.status\.(.*) [0-9][0-9][0-9][0-9][0-9][0-9]
> FILE    -close  data/models/conduit/status/\1
> 
> Any ideas...? I am a bit clueless attm...

Lets figure out one problem before attacking the other...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: AIZ-345619
Department: Support IDD
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.