[LDM #TAA-591344]: Temporal Subexpression Replacement

Hi Terri,

re:
> I am having issues reading a file from the queue and transforming the
> name using the information in the documentation on temporal subexpression
> replacement.

OK.

re:
> The files only come in once/hour so experimenting is taking too long.

Gotcha.

re:
> Could you give me some pointers? I'm using ldm-6.8.1.

We'll try.

re:
> The files come in and look like this (prefix and xx are really
> project-specific names):
> 
> prefix/xx_obs_20130807180000.CSV

To be clear, are you saying that the LDM/IDD Product IDs look like

prefix/xx_obs_20130807180000.CSV

I assume yes.

re:
> In my pact.conf, I can get this to work:
> 
> EXP     ^prefix/xx_obs_(.*).CSV FILE    -overwrite      -close  
> data/raw_data/obs/aaaa/bbbb_obs.\1.csv
> 
> And I get a file in the /home/ldm/data/raw_data/obs directory with the
> name changed to bbbb_obs.20130807180000.csv. This just changed xx to
> bbbb and uppercase CSV to lowercase csv with the full date string.

OK, very good.

re:
> What I would really like to do is to break up the date string to have it look
> like this:
> 
> bbbb_obs.yyyymmdd.hhMM.csv (throwing away the seconds)

Here is an LDM pattern-action file extended regular expression that should
work:

# prefix/xx_obs_20130807180000.CSV
EXP ^prefix/xx_obs_(....)(..)(..)(..)(..)..\.CSV
   FILE -overwrite -close
   data/raw_data/obs/aaaa/bbbb_obs.\1\2\3.\4\5.csv

Comments:

- of course, certain white spaces in LDM pattern-action file extended
  regular expressions are tabs (for anyone reading this ticket who
  doesn't already know this).  The example entry should be read:

EXP<tab>^prefix/xx_obs_(....)(..)(..)(..)(..)..\.CSV
<tab>FILE<tab>-overwrite<tab>-close
<tab>data/raw_data/obs/aaaa/bbbb_obs.\1\2\3.\4\5.csv

- there is more than one way to accomplish what you have said is your
  goal.  Another example that should work is:

# prefix/xx_obs_20130807180000.CSV
EXP ^prefix/xx_obs_(....)(....)(....)..\.CSV
   FILE -overwrite -close
   data/raw_data/obs/aaaa/bbbb_obs.\1\2.\3.csv

re:
> I've tried various combinations of parenthesized expressions, but am
> not coming up with the correct solution. Thanks for any help you can
> give me. Seems like a simple thing.

The best/easiest way to test an extended regular expression that you
want to use in an LDM pattern-action file is through the use of
'notifyme'.  This is true if the products being tested come in
frequently OR if they come in infrequently AND your LDM queue
is large enough to hold at least one product for the time between
receipt of products:

<as 'ldm'>
notifyme -vl- -f EXP -p "^prefix/xx_obs_(....)(....)(....)..\.CSV" -o 3600

The '-p' flag to 'notifyme' specifies the pattern that you want to
test. The '-o' flag says how far back in seconds you want to test
for products in your LDM queue.  If the queue is large enough to keep
an product until another is received, you can test, retest, retest, etc.

NB: depending on the shell you use, you may have to escape the parenthesis
in the 'notifyme' invocation (i.e., \( and \) instead of (), etc.).

On another subject completely:  Do you think that Global Weather would
be interested in relaying high volume CONDUIT data to commercial
entities that want to get the data (high resolution NCEP model output
that is not provided in the NOAAPort SBN) via the LDM/IDD?  We have
gotten a handful of requests for a CONDUIT feed by such organizations,
but we are not funded to provide this kind of support.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: TAA-591344
Department: Support LDM
Priority: Normal
Status: Closed