[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[IDD #AIZ-345619]: LDM: pattern in ldmd.conf and pact.conf no longer works for CONDUIT??
- Subject: [IDD #AIZ-345619]: LDM: pattern in ldmd.conf and pact.conf no longer works for CONDUIT??
- Date: Thu, 16 Jul 2020 11:39:07 -0600
Hi Christian,
re: extended regular expression in a 'notifyme' invocation
> > I just ran this from the 'ldm' account on my personal development machine
> > where I am running LDM v6.13.11, and the pattern matches lots of products.
>
> I also tested the same and it also matched lots of products.
OK. The next step that I would do is to have the 'notifyme' invocation
using the ERE pointing at your upstream feed site running in one terminal
and have an 'ldmamdin watch -f CONDUIT' running in a second terminal. This way
you could see if your REQUEST is actually resulting in the receipt
of the products that you want.
Assuming that the above test shows that you are, in fact, receiving the
products you want, the next step is to:
- verify that you have an LDM configuration file EXEC line that will cause
the CONDUIT products received to be acted on by the pattern-action file
actions that don't appear to be working
- assuming that the received products are being received, and there is a
'pqact' instance running that should process the products, I would
increase the logging level of the 'pqact' instance
This is done by sending a USR2 signal to the 'pqact' instance that is
running the actions that should process the products.
NB: the effects of running 'kill -s USR2 <pqact pid>' is incremental and
cyclical: sending one signal increases the log level from notice to info
and sending a second will cause the 'pqact' instance to issue debug
log messages. Sending a third signal will return the 'pqact' instance
back to its original logging level.
re: the pattern-action file actions that I used in my test were:
> > ~ldm/etc/pqact.conf_conduit:
> >
> > CONDUIT prod/gfs\.(........)/(..)/gfs.*pgrb2\.1p00\.f(.*) !grib2
> > FILE /data/ldm/pub/models/conduit/gfs/\1\2_\3_gfs.grib
> > CONDUIT prod/gfs\.(........)/(..)/gfs.*pgrb2\.1p00\.a.* !grib2
> > FILE /data/ldm/pub/models/conduit/gfs/\1_anl_gfs.grib
> >
>
> I also redid the same in my installation.
OK.
Can you send your LDM configuration file so I can verify that you are, in
fact, trying to process the received products via an 'EXEC "pqact..."'
entry?
re: my test produced the disk files specified
>
> I did not have any file created... it is weird because I am migrating to
> new servers, and the LDM instance there has the same ldmd.conf and
> pqact.conf and the files are being created.
Hmm... So, there is something different about the setup on your older
LDM machine(s). Quite frankly, the only thing that comes to mind is
there not being an LDM configuration file EXEC line that will tell a
'pqact' instance to process the products.
re:
> So I checked that the directory permissions were fine, and they are... but
> I still need those older servers to still work for the next few weeks
> because the migration is not over yet.
I understand. I want to solve the mystery no matter what :-)
re: my testing worked
>
> I re-typed those 2 pattern-action and it did not change the results, with
> no file being created.
What is the output of the following run while your LDM is running:
<as 'ldm'>
ps -eaf | grep pqact
re: LDM configuration or pattern-action file(s) edited with a Windows editor?
> I only edit directly using emacs on the CentOS linux system...
OK.
re: relevant lines from your LDM log file (feed starting up
line and any errors)?
>
>
> 20200716T134650.077475Z ldmd[18482] ldmd.c:main() NOTE
> Starting Up (version: 6.13.10; built: Mar 8 2019 10:41:08)
> 20200716T134650.078550Z ldmd[18482]
> ldmd.c:create_ldm_tcp_svc() NOTE Using local address 0.0.0.0:388
> 20200716T134650.103817Z idd.meteo.psu.edu[18498]
> LdmConfFile.c:requester_exec() NOTE Starting Up(6.13.10):
> idd.meteo.psu.edu:388 20200716134432.103690 TS_E NDT {{NEXRAD3,
> "/p(N0Z)(CXX)"}}
> 20200716T134650.104574Z pqact[18487] pqact.c:main() NOTE
> Starting Up
> ...
> 20200716T134650.115367Z idd.ssec.wisc.edu[18504] requester6.c:req6_new()
> NOTE LDM-6 desired product-class: 20200716134432.115283 TS_ENDT {{CONDUIT,
> "status"},{NONE, "SIG=dd53be508c5457add3ab29bf89b86c1a"}}
> 20200716T134650.107628Z striker.atmos.albany.edu[18495]
> requester6.c:req6_new() NOTE LDM-6 desired product-class:
> 20200716134432.107552 TS_ENDT {{LIGHTNING, ".*"},{NONE,
> "SIG=e5e9dce70e99b19eacaeced2bd199906"}}
> 20200716T134650.116877Z iddb.unidata.ucar.edu[18503]
> LdmConfFile.c:requester_exec() NOTE Starting Up(6.13.10):
> iddb.unidata.ucar.edu:388 20200716134432.116801 TS_ENDT {{CONDUIT,
> "(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"}}
> 20200716T134650.117428Z iddb.unidata.ucar.edu[18503] requester6.c:req6_new()
> NOTE LDM-6 desired product-class: 20200716134432.117255 TS_ENDT {{CONDUIT,
> "(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"},{NONE,"SIG=523332ba202d0b22560d67494aeffb4c"}}
> 20200716T134650.121885Z pqact[18491] pqact.c:main() NOTE
> Starting Up
> 20200716T134650.125146Z pqact[18491] pqact.c:main() NOTE
> Starting from insertion-time 2020-07-16 13:46:46.910676 UTC
> 20200716T134650.126128Z rtstats[18488] rtstats.c:main() NOTE
> Starting Up (18482)
...
OK, it is hard to tell from the listing sent if there is a 'pqact' instance that
is trying to run actions on the CONDUIT data. The contents of your LDM
configuration
file and/or the 'ps -eaf | grep pqact' output should tell us/me if this is, in
fact
the case.
re:
>
> >
> > Can you run the 'notifyme' that I included above and send the first "few"
> > (say 30 or so) lines of output (please don't send a large listing as it
> > is not needed).
> [ldm@vkepler logs]$ notifyme -vl- -f CONDUIT -h iddb.unidata.ucar.edu -p
> "(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"
> -o 10800
> 20200716T134929.130382Z notifyme[27581] notifyme.c:main() NOTE
> Starting Up: iddb.unidata.ucar.edu:
> 20200716104929.129996 TS_ENDT {{CONDUIT,
> "(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"}}
> 20200716T134929.130497Z notifyme[27581] ldm5_clnt.c:forn5() NOTE
> LDM-5 desired product-class:
> 20200716104929.129996 TS_ENDT
> {{CONDUIT,"(gfs.*grb2.1p00.(f|a).*).*(PMSL|GUST|HGHT|P..M|RELH/2.m|RELH.*hPa|WX|OMEG.*hPa|TMPK|TM[XN]K03|[UV]REL|CLD|FXSH|CIN|CAPE|LIFT|PWTR|PRES/0.*NONE)"}}
> 20200716T134929.132425Z notifyme[27581] error.c:err_log() INFO
> Resolving iddb.unidata.ucar.edu to 128.117.130.3 took 0.001793 seconds
> 20200716T134929.263367Z notifyme[27581] ldm5_clnt.c:forn_signon()
> NOTE NOTIFYME(iddb.unidata.ucar.edu): OK
> 20200716T134933.838864Z notifyme[27581]
> notifyme.c:notifymeprog_5() INFO 46159 20200716122819.498183 CONDUIT
> 017
> data/nccf/com/gfs/prod/gdas.20200716/06/gdas.t06z.pgrb2.1p00.anl!grib2/ncep/SSIGFS/#000/202007160600F000/UREL/3
> hPa PRES! 000017
> 20200716T134933.902886Z notifyme[27581]
> notifyme.c:notifymeprog_5() INFO 60923 20200716122819.543308 CONDUIT
> 037 data/nccf/com/gfs/prod/gdas.20200716/06/gdas.t06z.pgrb2.1p00.anl
> !grib2/ncep/SSIGFS/#000/202007160600F000/HGHT/20 hPa PRES! 000037
> 20200716T134934.068210Z notifyme[27581]
> notifyme.c:notifymeprog_5() INFO 64054 20200716122819.690882 CONDUIT
> 097 data/nccf/com/gfs/prod/gdas.20200716/06/gdas.t06z.pgrb2.1p00.anl
> !grib2/ncep/SSIGFS/#000/202007160600F000/HGHT/200 hPa PRES! 000097
> 20200716T134934.131771Z notifyme[27581]
> notifyme.c:notifymeprog_5() INFO 52132 20200716122819.759393 CONDUIT
> 117 data/nccf/com/gfs/prod/gdas.20200716/06/gdas.t06z.pgrb2.1p00.anl
> !grib2/ncep/SSIGFS/#000/202007160600F000/VREL/250 hPa PRES! 000117
> 20200716T134934.297414Z notifyme[27581]
> notifyme.c:notifymeprog_5() INFO 63707 20200716122819.477299 CONDUIT
> 005 data/nccf/com/gfs/prod/gdas.20200716/06/gdas.t06z.pgrb2.1p00.anl
> !grib2/ncep/SSIGFS/#000/202007160600F000/HGHT/1 hPa PRES! 000005
> 20200716T134934.360755Z notifyme[27581]
> notifyme.c:notifymeprog_5() INFO 78081 20200716122819.939182 CONDUIT
> 237 data/nccf/com/gfs/prod/gdas.20200716/06/gdas.t06z.pgrb2.1p00.anl
> !grib2/ncep/SSIGFS/#000/202007160600F000/UREL/700 hPa PRES! 000237
> ...
This listing shows that the upstream is getting the products you want.
re:
> > re:
> > > But nothing gets through. I checked with ldmadmin watch -f CONDUIT and
> > > I get no headers. So it points out a problem in ldmd.conf, but I cannot
> > > figure it out, as I tried many variations in the regex. I also checked
> > > the regex and it is validated.
If this is still true, try switching your CONDUIT feed REQUEST to a
different upstream relay. We have on rare occasion seen upstream processes
stop sending data to the downstream to which it has an active connection.
Changing the REQUEST to a different upstream would definitely result
in a different machine feeding you. NB: simply restarting your LDM will
not result in your feed REQUEST(s) going to a different real-server backend
machine of the cluster/machine that you have been/are REQUESTing from.
Last question for this email:
Is the machine you are having problems with autan.sca.uqam.ca?
If yes, I can say that the real-time stats that it is reporting back to us
indicates that CONDUIT data is being received now:
https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_vol_nc?CONDUIT+autan.sca.uqam.ca
re:
> > Lets figure out one problem before attacking the other...
>
> Yes, thanks!!
No worries.
Cheers,
Tom
--
****************************************************************************
Unidata User Support UCAR Unidata Program
(303) 497-8642 P.O. Box 3000
address@hidden Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage http://www.unidata.ucar.edu
****************************************************************************
Ticket Details
===================
Ticket ID: AIZ-345619
Department: Support IDD
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata
inquiry tracking system and then made publicly available through the web. If
you do not want to have your interactions made available in this way, you must
let us know in each email you send to us.