[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[LDM #FWJ-984418]: Access to the data-product received
- Subject: [LDM #FWJ-984418]: Access to the data-product received
- Date: Mon, 15 Oct 2012 08:50:03 -0600
Hi Aurelien,
re:
> I am a new ldm user (new to unix too...) and I managed to install and
> configure ldm
> 6.11.1. When I launch ldm it seems like I am receiving data-products from my
> upstream
> ldm (ldmadmin watch, notifyme and pqmon are giving me the same information
> about the
> products received):
>
> ldm@ubuntu:~$ ldmadmin watch
> (Type ^D when finished)
> Oct 15 04:45:22 pqutil INFO: 284584 20121015035540.131 EXP 000
> ncar.ral.CoSPA.Shadow.BlendNcarPcDigitalVil.2012-10-15T03:45:00.PT07:30.nc
> Oct 15 04:45:22 pqutil INFO: 133862 20121015035548.029 EXP 000
> ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:15.nc
> Oct 15 04:45:22 pqutil INFO: 133467 20121015035548.147 EXP 000
> ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:30.nc
> Oct 15 04:45:22 pqutil INFO: 132706 20121015035548.267 EXP 000
> ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:45.nc
> Oct 15 04:45:23 pqutil INFO: 283923 20121015035550.251 EXP 000
> ncar.ral.CoSPA.Shadow.BlendNcarPcDigitalVil.2012-10-15T03:45:00.PT07:45.nc
>
> ldm@ubuntu:~$ notifyme -vl- -f ANY
> Oct 15 04:46:33 notifyme[3997] NOTE: Starting Up: localhost:
> 20121015044633.684 TS_ENDT {{ANY, ".*"}}
> Oct 15 04:46:33 notifyme[3997] NOTE: LDM-5 desired product-class:
> 20121015044633.684 TS_ENDT {{ANY, ".*"}}
> Oct 15 04:46:33 notifyme[3997] INFO: Resolving localhost to 127.0.0.1 took
> 0.000187 seconds
> Oct 15 04:46:33 notifyme[3997] NOTE: NOTIFYME(localhost): OK
> Oct 15 04:48:47 notifyme[3997] INFO: 6425 20121015044837.494 EXP 000
> ncar.ral.convwx.AirspaceCapProb.AFWA_CDF_WAF_30.2012-10-15T00:00:00PT13:00.grb2
> Oct 15 04:48:47 notifyme[3997] INFO: 7097 20121015044840.530 EXP 000
> ncar.ral.convwx.AirspaceCapProb.AFWA_CDF_WAF_20.2012-10-15T00:00:00PT13:00.grb2
>
> pqmon
> Oct 15 04:49:48 pqmon NOTE: Starting Up (4013)
> Oct 15 04:49:48 pqmon NOTE: nprods nfree nempty nbytes maxprods
> maxfree minempty maxext age
> Oct 15 04:49:48 pqmon NOTE: 677 1 121392 120686288 677
> 1 121392 379316528 299
> Oct 15 04:49:48 pqmon NOTE: Exiting
OK, these definitely indicate that you are receiving products.
re:
> But then I have been unable to figure out where the data is actually written.
> Should it
> be automatically downloaded by lmd in the repository lmd/var/data (there is
> nothing in
> that folder) or should I write an action in the pqact.conf file to have
> access to it?
You need an appropriate action in a pattern-action file (like
~ldm/etc/pqact.conf).
The important actions that your action will perform is creating a file in
the directory you specify with the name that you also need to specify.
re:
> I tried that option
The action you created in the pqact.conf file that you attached does not
specify a name for the file(s) that are to be written to disk. I suggest
a more usable action below.
re:
> but I would like to have access to the files downloaded directly,
> not by writing them into another file as FILE seems to do (the files that I
> will
> download once the installation is well configured are grib2 files).
Two things here:
- you will need to have the products that have been transferred to you
extracted out of the LDM product queue and then written to disk
- the names of the files you are receiving suggest that the files are
in netCDF format, not GRIB2
Perhaps you are requesting the wrong set of data from 128.117.200.219?
re:
> Sorry if the answer is obvious, but I am discovering ldm as well as unix and
> I didn't
> find anything on the web...
No worries.
re:
> I attached the conf files if you need them.
Thank you for attaching the ldmd.conf and pqact.conf files. They are most
useful in helping us to help you!
The action you wrote in pqact.conf:
#'ncar\.ral\.CoSPA.Live.BlendNcarPcDigitalVil\.(...)(.*PT0[1-8]:00\.nc)'
ANY ncar.ral FILE -close -log /datafromldm/
correctly tells 'pqact' (the LDM routine that processes products
out of an LDM queue) that you want to FILE the products. What is
missing/incorrect is the name for the files that will be written to
disk.
Please try the following example invocation to see if it does what
you want/need:
#
# Example product header received:
# ncar.ral.CoSPA.Shadow.BlendNcarPcDigitalVil.2012-10-15T03:45:00.PT07:30.nc
#
EXP (.*)
FILE -close -log
NCAR/radar/\1
NB:
I changed ANY to EXP since your 'ldmadmin watch' listing shows that the
products are coming in on the EXP feedtype.
Some of the white space in this example pattern-action file action are
tabs, not spaces. This action can be read (<tab> means a tab character):
EXP<tab>(.*)
<tab>FILE<tab>-close -log
<tab>NCAR/radar/\1
After replacing the action that you currently have with this example,
you should:
- verify that the pattern-action file does not contain errors:
<as 'ldm'>
-- edit ~ldm/etc/pqact.conf
ldmadmin pqactcheck
If there are no errors indicated, you can proceed; if there are
errors, you will need to re-edit pqact.conf and fix the error(s).
- if the LDM is running, you can the 'pqact' to reread your pattern-action
file patterns so that the new one will be used:
<as 'ldm'>
ldmadmin pqactHUP
The result of the new action should be the writing of files in the
~ldm/var/data/NCAR/radar directory, and the names of the files will
be their product IDs. For example:
...
ncar.ral.CoSPA.Shadow.BlendNcarPcDigitalVil.2012-10-15T03:45:00.PT07:30.nc
ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:15.nc
ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:30.nc
ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:45.nc
ncar.ral.CoSPA.Shadow.BlendNcarPcDigitalVil.2012-10-15T03:45:00.PT07:45.nc
...
If you don't like how the products are being named, you can fashion a
more complex action that extracts bits of information from the product IDs
and uses that information to name the files. We can discuss this further
after your LDM starts writing files to disk.
re:
> Thank you for your time!
No worries.
Cheers,
Tom
--
****************************************************************************
Unidata User Support UCAR Unidata Program
(303) 497-8642 P.O. Box 3000
address@hidden Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage http://www.unidata.ucar.edu
****************************************************************************
Ticket Details
===================
Ticket ID: FWJ-984418
Department: Support LDM
Priority: Normal
Status: Closed