Re: 20050810:Re: IRaDS & LDM

Hi Marc and A.J.,

Comments below...

Marcus A Christie wrote:
> Anne Wilson wrote:
>
> > Hi Marc and A.J.,
> >
> > Your question was forwarded to me, being a LEAD team member myself.
>
> Thanks, Anne, for a quick and thorough response!
>
> > First, why do you want to decode this data?  Are you aware that 
> > nexradII is a decompressor?  Generally it's beneficial to only 
> > decompress this data on demand because the overall compressed volume 
> > is very large and the uncompressed volume is huge.  Our visualization 
> > tools such as the IDV decompress it dynamically, so we generally do 
> > not recommend decompressing this as it comes in.
>
> This is mostly AJ's project, so I'll let him clarify, but what we wanted 
> to do was get some performance numbers on what kind of computational 
> capacity is needed to 1) decompress these files as whole files, and 2) 
> decompress these files in a piecemeal way like LDM.  So first we're 
> trying to ascertain whether LDM spawns a decoder for each piece of a 
> radar file, or one for all the pieces, in which case the CPU needs for 1 
> should be about the same as the CPU needs for 2.  AJ is working with 
> Beth on a way to collect metadata regarding these radar files so that 
> they can do queries, and as needed decompress these files.

I just happen to be fairly familiar with this particular decoder.  (Not 
true for the other decoders!)   I hope my discussion made clear that for 
this decoder spawns one instance of the decoder for all the pieces of a 
volume scan.

Regarding the collection of metadata, there is a collaborative project 
going on here, which also involves the University of Iowa and CAUHSI, to 
generate metadata for this level II radar data.  The web page is still 
rather thin at the moment 
(http://www.iihr.uiowa.edu/~hml/projects/nexrad-itr/), but the idea (as 
I understand it) is to generate metadata for this data and then relay 
the metadata rather than the data itself.  The metadata generation may 
actually involve mining so that detected features can be logged in the 
metadata.

But, I presume you and A.J. are trying to generate metadata for the LEAD 
metadata schema?   I'm not sure how much that would overlap with the 
above effort, but it's might be worth looking into.


> > But, to answer your question, when a decoder is spawned depends on the 
> > products that come in the data stream.  Whether a decoder persists or 
> > not and when it terminates depends on how the decoder was coded. 
> > Generally we recommend that decoders stay open and read from stdin in 
> > order to avoid the overhead of restarting them.  The nexradII decoder 
> > is slightly different in that it accumulates pieces of a volume scan, 
> > decompresses them, and creates a file containing a decompressed volume 
> > scan, after which it terminates.
> >
> > To say more, spawning of a decoder depends on the products that match 
> > the regular expression that filters the products sent to the decoder. 
> > As long as products come in that match the regular expression and 
> > don't cause new substitutions in the template, in this case the 
> > "data/pub/decoded/nexradII/raw/\2/\3.raw" string, they will be fed to 
> > that same instance of the decoder if it is still running.
>
> When would the instance of nexradII not still be running?  Would the 
> nexradII decoder ever terminate before it has received the entire product?

No, the nexradII decoder would not terminate before it received the 
entire product.  In my discussion I was confabulating info about that 
decoder along with decoders in general.


> > So, in this example, when new products come in from a different 
> > station or with a different date, this causes different substitutions 
> > in the \2 and \3 pieces of the template.  Thus, a new instance of the 
> > decoder will be spawned with the new values substituted into the 
> > template, creating a new argument for this particular invocation.  In 
> > this data stream the volume scan is sent in a sequence of 100 radial 
> > chunks.  Since all the products for a particular volume scan have the 
> > same station ID and date, this results in a new instance of the 
> > decoder for each volume scan from a particular station, and a 
> > resulting file that is the complete decompressed volume scan for each 
> > instance of the decoder.
> >
> > Regarding shutting down, this decoder looks for a final product to 
> > know when to end the decompression, close the file, and shut down.  
> > (Thus, order of arrival is important with this decoder.)
> >
> > Does this answer your questions?
> >
> > Anne
>
>
> Thanks Anne.  This helps a lot.
>
> Marc

You're welcome!

Anne

--
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://my.unidata.ucar.edu/
****************************************************