[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[IDD #AIZ-345619]: LDM: pattern in ldmd.conf and pact.conf no longer works for CONDUIT??
- Subject: [IDD #AIZ-345619]: LDM: pattern in ldmd.conf and pact.conf no longer works for CONDUIT??
- Date: Wed, 15 Jul 2020 10:51:59 -0600
Hi Christian,
re:
> I am ingesting data using LDM, and one of the feed I use is CONDUIT. I
> have been using it for years. I am running ldm 6.13.10. However,
> recently suddenly, I do not receive any CONDUIT products anymore.
>
> Because of bandwidth limitations, I have a special CONDUIT regex request
> in ldmd.com.
>
> I also tested with notifyme that I can receive the feed properly.
>
> With notifyme:
>
> notifyme -f CONDUIT -vl log -h iddb.unidata.ucar.edu
> I receive among all others, for example this header:
>
> 20200715T121412.415671Z notifyme[22262]
> notifyme.c:notifymeprog_5() INFO 40685 20200715102559.499768 CONDUIT
> 283 data/nccf/com/gfs/prod/gfs.20200715/06/gfs.t06z.pgrb2.1p00.f237
> !grib2/ncep/GFS/#000/202007150600F237/TMPK/700 hPa PRES! 000283
>
> In my ldmd.conf, for CONDUIT (yes there is a tab character between request
> and CONDUIT):
>
> request CONDUIT
> "(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"
> iddb.unidata.ucar.edu
OK, the first thing that I do to test extended regular expression(s) used
in LDM configuration file REQU$ST(s) and pattern-actoin file actions is use
'notifyme' to see if the REQUEST ERE will match anything in the feed. For
instance:
notifyme -vl- -f CONDUIT -h iddb.unidata.ucar.edu -p
"(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"
-o 3600
I just ran this from the 'ldm' account on my personal development machine
where I am running LDM v6.13.11, and the pattern matches lots of products.
re:
> In my pqact.conf (I have tabs after CONDUIT, and before & after FILE):
> CONDUIT prod/gfs\.(........)/(..)/gfs.*pgrb2\.1p00\.f(.*) !grib2
> FILE data/models/conduit/gfs/\1\2_\3_gfs.grib
> CONDUIT prod/gfs\.(........)/(..)/gfs.*pgrb2\.1p00\.a.* !grib2
> FILE data/models/conduit/gfs/\1_anl_gfs.grib
I just created the EXEC to run 'pqact' for CONDUIT products and an
associated pattern-action file with these two actions as follows:
~ldm/etc/ldmd.conf:
EXEC "pqact -f CONDUIT etc/pqact.conf_conduit"
REQUEST CONDUIT
"(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"
iddb.unidata.ucar.edu
~ldm/etc/pqact.conf_conduit:
CONDUIT prod/gfs\.(........)/(..)/gfs.*pgrb2\.1p00\.f(.*) !grib2
FILE /data/ldm/pub/models/conduit/gfs/\1\2_\3_gfs.grib
CONDUIT prod/gfs\.(........)/(..)/gfs.*pgrb2\.1p00\.a.* !grib2
FILE /data/ldm/pub/models/conduit/gfs/\1_anl_gfs.grib
I cranked up my LDM, and I see files being created:
ls -alt /data/ldm/pub/models/conduit/gfs/
total 6920
-rw-r--r-- 1 ldm Unidata 6983837 Jul 15 10:42 2020071512_081_gfs.grib
drwxr-xr-x 2 ldm Unidata 4096 Jul 15 10:42 .
drwxr-xr-x 3 ldm Unidata 4096 Jul 15 10:42 ..
-rw-r--r-- 1 ldm Unidata 88554 Jul 15 10:42 2020071512_006_gfs.grib
Given that my tests worked, I don't know what could be causing the problem
in your setup. The only things that I can think of are:
1) there is a space along with a tab between CONDUIT and the ERE or before/after
FILE(s) in your pattern-action file
2) your LDM configuration file and/or pattern-action file was/were edited
with a Windows editor (e.g., notepad) so the file is not in a *NIX
format
Requests:
Can you send relevant lines from your LDM log file (feed starting up
line and any errors)?
Can you run the 'notifyme' that I included above and send the first "few"
(say 30 or so) lines of output (please don't send a large listing as it
is not needed).
re:
> But nothing gets through. I checked with ldmadmin watch -f CONDUIT and
> I get no headers. So it points out a problem in ldmd.conf, but I cannot
> figure it out, as I tried many variations in the regex. I also checked
> the regex and it is validated.
The output of the 'notifyme' command that I included above should affirm/
refute if you are receiving the products... I have seen instances where
an 'ldmadmin watch' doesn't show receipt of data while a 'notifyme' does
and visa versa.
re:
> I am also trying to get the status but it does not work (no output or even
> directory created):
> In ldmd.conf:
>
> request CONDUIT "status" idd.ssec.wisc.edu
>
> In pqact.conf:
> CONDUIT ^.status\.(.*) [0-9][0-9][0-9][0-9][0-9][0-9]
> FILE -close data/models/conduit/status/\1
>
> Any ideas...? I am a bit clueless attm...
Lets figure out one problem before attacking the other...
Cheers,
Tom
--
****************************************************************************
Unidata User Support UCAR Unidata Program
(303) 497-8642 P.O. Box 3000
address@hidden Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage http://www.unidata.ucar.edu
****************************************************************************
Ticket Details
===================
Ticket ID: AIZ-345619
Department: Support IDD
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata
inquiry tracking system and then made publicly available through the web. If
you do not want to have your interactions made available in this way, you must
let us know in each email you send to us.