Re: 20050810:Re: IRaDS & LDM

Hi Marc and A.J.,

Your question was forwarded to me, being a LEAD team member myself.

First, why do you want to decode this data?  Are you aware that nexradII 
is a decompressor?  Generally it's beneficial to only decompress this 
data on demand because the overall compressed volume is very large and 
the uncompressed volume is huge.  Our visualization tools such as the 
IDV decompress it dynamically, so we generally do not recommend 
decompressing this as it comes in.

But, to answer your question, when a decoder is spawned depends on the 
products that come in the data stream.  Whether a decoder persists or 
not and when it terminates depends on how the decoder was coded. 
Generally we recommend that decoders stay open and read from stdin in 
order to avoid the overhead of restarting them.  The nexradII decoder is 
slightly different in that it accumulates pieces of a volume scan, 
decompresses them, and creates a file containing a decompressed volume 
scan, after which it terminates.

To say more, spawning of a decoder depends on the products that match 
the regular expression that filters the products sent to the decoder. 
As long as products come in that match the regular expression and don't 
cause new substitutions in the template, in this case the 
"data/pub/decoded/nexradII/raw/\2/\3.raw" string, they will be fed to 
that same instance of the decoder if it is still running.

So, in this example, when new products come in from a different station 
or with a different date, this causes different substitutions in the \2 
and \3 pieces of the template.  Thus, a new instance of the decoder will 
be spawned with the new values substituted into the template, creating a 
new argument for this particular invocation.  In this data stream the 
volume scan is sent in a sequence of 100 radial chunks.  Since all the 
products for a particular volume scan have the same station ID and date, 
this results in a new instance of the decoder for each volume scan from 
a particular station, and a resulting file that is the complete 
decompressed volume scan for each instance of the decoder.

Regarding shutting down, this decoder looks for a final product to know 
when to end the decompression, close the file, and shut down.  (Thus, 
order of arrival is important with this decoder.)

Does this answer your questions?

Anne


Unidata Support wrote:
> ------- Forwarded Message
>
>
> > To: Andrew J Ragusa <address@hidden>
> > cc: address@hidden,
> > cc: address@hidden
> > From: Marcus A Christie <address@hidden>
> > Subject: Re: IRaDS & LDM
> > Organization: UCAR/Unidata
> > Keywords: 200508102225.j7AMP6jo020518
>
>
> Andrew J Ragusa wrote:
>
> > I'm A.J. working with Professor Plale over at IU.  I had a question on the
> > nexradII data and the LDM decoder.  Mainly, how does LDM launch the
> > decoder? Is the decoder launched every time a new chunk of a file comes
> > in, or is it a persistent pipe that starts when the first chunk of file
> > is received and finished when the last chunk is received??
> >
> > Thanks
> >
> > AJ Ragusa
> > address@hidden
> > 630-697-2470
>
>
> Just to further elaborate, this is in relation to the LDM pqact action 
> PIPE.  The full command, if it helps, is:
>
> NEXRD2 
> ^L2-([^/]*)/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9][0-9][0-9])
>          PIPE    nexradII data/pub/decoded/nexradII/raw/\2/\3.raw
>
>
> Marc

--
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://my.unidata.ucar.edu/
****************************************************