[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: 20050810:Re: IRaDS & LDM
- Subject: Re: 20050810:Re: IRaDS & LDM
- Date: Thu, 11 Aug 2005 10:40:08 -0600
Hi Marc and A.J.,
Comments below...
Marcus A Christie wrote:
Anne Wilson wrote:
Hi Marc and A.J.,
Your question was forwarded to me, being a LEAD team member myself.
Thanks, Anne, for a quick and thorough response!
First, why do you want to decode this data? Are you aware that
nexradII is a decompressor? Generally it's beneficial to only
decompress this data on demand because the overall compressed volume
is very large and the uncompressed volume is huge. Our visualization
tools such as the IDV decompress it dynamically, so we generally do
not recommend decompressing this as it comes in.
This is mostly AJ's project, so I'll let him clarify, but what we wanted
to do was get some performance numbers on what kind of computational
capacity is needed to 1) decompress these files as whole files, and 2)
decompress these files in a piecemeal way like LDM. So first we're
trying to ascertain whether LDM spawns a decoder for each piece of a
radar file, or one for all the pieces, in which case the CPU needs for 1
should be about the same as the CPU needs for 2. AJ is working with
Beth on a way to collect metadata regarding these radar files so that
they can do queries, and as needed decompress these files.
I just happen to be fairly familiar with this particular decoder. (Not
true for the other decoders!) I hope my discussion made clear that for
this decoder spawns one instance of the decoder for all the pieces of a
volume scan.
Regarding the collection of metadata, there is a collaborative project
going on here, which also involves the University of Iowa and CAUHSI, to
generate metadata for this level II radar data. The web page is still
rather thin at the moment
(http://www.iihr.uiowa.edu/~hml/projects/nexrad-itr/), but the idea (as
I understand it) is to generate metadata for this data and then relay
the metadata rather than the data itself. The metadata generation may
actually involve mining so that detected features can be logged in the
metadata.
But, I presume you and A.J. are trying to generate metadata for the LEAD
metadata schema? I'm not sure how much that would overlap with the
above effort, but it's might be worth looking into.
But, to answer your question, when a decoder is spawned depends on the
products that come in the data stream. Whether a decoder persists or
not and when it terminates depends on how the decoder was coded.
Generally we recommend that decoders stay open and read from stdin in
order to avoid the overhead of restarting them. The nexradII decoder
is slightly different in that it accumulates pieces of a volume scan,
decompresses them, and creates a file containing a decompressed volume
scan, after which it terminates.
To say more, spawning of a decoder depends on the products that match
the regular expression that filters the products sent to the decoder.
As long as products come in that match the regular expression and
don't cause new substitutions in the template, in this case the
"data/pub/decoded/nexradII/raw/\2/\3.raw" string, they will be fed to
that same instance of the decoder if it is still running.
When would the instance of nexradII not still be running? Would the
nexradII decoder ever terminate before it has received the entire product?
No, the nexradII decoder would not terminate before it received the
entire product. In my discussion I was confabulating info about that
decoder along with decoders in general.
So, in this example, when new products come in from a different
station or with a different date, this causes different substitutions
in the \2 and \3 pieces of the template. Thus, a new instance of the
decoder will be spawned with the new values substituted into the
template, creating a new argument for this particular invocation. In
this data stream the volume scan is sent in a sequence of 100 radial
chunks. Since all the products for a particular volume scan have the
same station ID and date, this results in a new instance of the
decoder for each volume scan from a particular station, and a
resulting file that is the complete decompressed volume scan for each
instance of the decoder.
Regarding shutting down, this decoder looks for a final product to
know when to end the decompression, close the file, and shut down.
(Thus, order of arrival is important with this decoder.)
Does this answer your questions?
Anne
Thanks Anne. This helps a lot.
Marc
You're welcome!
Anne
--
***************************************************
Anne Wilson UCAR Unidata Program
address@hidden P.O. Box 3000
Boulder, CO 80307
----------------------------------------------------
Unidata WWW server http://my.unidata.ucar.edu/
****************************************************