[LDM #FWJ-984418]: Access to the data-product received

Hi Aurelien,

re:
> I am a new ldm user (new to unix too...) and I managed to install and 
> configure ldm
> 6.11.1. When I launch ldm it seems like I am receiving data-products from my 
> upstream
> ldm (ldmadmin watch, notifyme and pqmon are giving me the same information 
> about the
> products received):
> 
> ldm@ubuntu:~$ ldmadmin watch
> (Type ^D when finished)
> Oct 15 04:45:22 pqutil INFO:   284584 20121015035540.131     EXP 000 
> ncar.ral.CoSPA.Shadow.BlendNcarPcDigitalVil.2012-10-15T03:45:00.PT07:30.nc
> Oct 15 04:45:22 pqutil INFO:   133862 20121015035548.029     EXP 000 
> ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:15.nc
> Oct 15 04:45:22 pqutil INFO:   133467 20121015035548.147     EXP 000 
> ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:30.nc
> Oct 15 04:45:22 pqutil INFO:   132706 20121015035548.267     EXP 000 
> ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:45.nc
> Oct 15 04:45:23 pqutil INFO:   283923 20121015035550.251     EXP 000 
> ncar.ral.CoSPA.Shadow.BlendNcarPcDigitalVil.2012-10-15T03:45:00.PT07:45.nc
> 
> ldm@ubuntu:~$ notifyme -vl- -f ANY
> Oct 15 04:46:33 notifyme[3997] NOTE: Starting Up: localhost: 
> 20121015044633.684 TS_ENDT {{ANY,  ".*"}}
> Oct 15 04:46:33 notifyme[3997] NOTE: LDM-5 desired product-class: 
> 20121015044633.684 TS_ENDT {{ANY,  ".*"}}
> Oct 15 04:46:33 notifyme[3997] INFO: Resolving localhost to 127.0.0.1 took 
> 0.000187 seconds
> Oct 15 04:46:33 notifyme[3997] NOTE: NOTIFYME(localhost): OK
> Oct 15 04:48:47 notifyme[3997] INFO:     6425 20121015044837.494     EXP 000  
> ncar.ral.convwx.AirspaceCapProb.AFWA_CDF_WAF_30.2012-10-15T00:00:00PT13:00.grb2
> Oct 15 04:48:47 notifyme[3997] INFO:     7097 20121015044840.530     EXP 000  
> ncar.ral.convwx.AirspaceCapProb.AFWA_CDF_WAF_20.2012-10-15T00:00:00PT13:00.grb2
> 
> pqmon
> Oct 15 04:49:48 pqmon NOTE: Starting Up (4013)
> Oct 15 04:49:48 pqmon NOTE: nprods nfree  nempty      nbytes  maxprods  
> maxfree  minempty    maxext  age
> Oct 15 04:49:48 pqmon NOTE:    677     1  121392   120686288       677        
> 1    121392 379316528 299
> Oct 15 04:49:48 pqmon NOTE: Exiting

OK, these definitely indicate that you are receiving products.

re:
> But then I have been unable to figure out where the data is actually written. 
> Should it
> be automatically downloaded by lmd in the repository lmd/var/data (there is 
> nothing in
> that folder) or should I write an action in the pqact.conf file to have 
> access to it?

You need an appropriate action in a pattern-action file (like 
~ldm/etc/pqact.conf).
The important actions that your action will perform is creating a file in
the directory you specify with the name that you also need to specify.

re:
> I tried that option

The action you created in the pqact.conf file that you attached does not
specify a name for the file(s) that are to be written to disk.  I suggest
a more usable action below.

re:
> but I would like to have access to the files downloaded directly,
> not by writing them into another file as FILE seems to do (the files that I 
> will
> download once the installation is well configured are grib2 files).

Two things here:

- you will need to have the products that have been transferred to you
  extracted out of the LDM product queue and then written to disk

- the names of the files you are receiving suggest that the files are
  in netCDF format, not GRIB2

  Perhaps you are requesting the wrong set of data from 128.117.200.219?

re:
> Sorry if the answer is obvious, but I am discovering ldm as well as unix and 
> I didn't
> find anything on the web...

No worries.

re:
> I attached the conf files if you need them.

Thank you for attaching the ldmd.conf and pqact.conf files.  They are most
useful in helping us to help you!

The action you wrote in pqact.conf:

#'ncar\.ral\.CoSPA.Live.BlendNcarPcDigitalVil\.(...)(.*PT0[1-8]:00\.nc)'
ANY     ncar.ral        FILE    -close -log     /datafromldm/

correctly tells 'pqact' (the LDM routine that processes products
out of an LDM queue) that you want to FILE the products.  What is
missing/incorrect is the name for the files that will be written to
disk.

Please try the following example invocation to see if it does what
you want/need:

#
# Example product header received:
# ncar.ral.CoSPA.Shadow.BlendNcarPcDigitalVil.2012-10-15T03:45:00.PT07:30.nc
#

EXP     (.*)
       FILE     -close  -log
       NCAR/radar/\1

NB:
 
I changed ANY to EXP since your 'ldmadmin watch' listing shows that the
products are coming in on the EXP feedtype.

Some of the white space in this example pattern-action file action are
tabs, not spaces.  This action can be read (<tab> means a tab character):

EXP<tab>(.*)
<tab>FILE<tab>-close -log
<tab>NCAR/radar/\1

After replacing the action that you currently have with this example,
you should:

- verify that the pattern-action file does not contain errors:

<as 'ldm'>
-- edit ~ldm/etc/pqact.conf

ldmadmin pqactcheck

  If there are no errors indicated, you can proceed; if there are
  errors, you will need to re-edit pqact.conf and fix the error(s).

- if the LDM is running, you can the 'pqact' to reread your pattern-action
  file patterns so that the new one will be used:

<as 'ldm'>

ldmadmin pqactHUP

The result of the new action should be the writing of files in the 
~ldm/var/data/NCAR/radar directory, and the names of the files will
be their product IDs.  For example:

 ...
ncar.ral.CoSPA.Shadow.BlendNcarPcDigitalVil.2012-10-15T03:45:00.PT07:30.nc
ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:15.nc
ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:30.nc
ncar.ral.CoSPA.Shadow.BlendNcarPcEchoTops.2012-10-15T03:45:00.PT07:45.nc
ncar.ral.CoSPA.Shadow.BlendNcarPcDigitalVil.2012-10-15T03:45:00.PT07:45.nc
 ...

If you don't like how the products are being named, you can fashion a
more complex action that extracts bits of information from the product IDs
and uses that information to name the files.  We can discuss this further
after your LDM starts writing files to disk.

re:
> Thank you for your time!

No worries.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: FWJ-984418
Department: Support LDM
Priority: Normal
Status: Closed