Re: 20091021: Nexrad Level II decoder

Hiya,

I been revising the makeRadarII script to make as many complete products 
as possible. I started a log on incomplete products, I discovered that 
eventually the complete product arrives, it could be as much as 30 
minutes after the first piece arrives. So here's what I've been doing, 
if it's an incomplete product, I wait until the next product 
arrives and then  make a product in the live directory from the pieces 
that have already arrived. Most of the time the 
pieces are contiguous up to some point. Then I keep checking everytime an 
end piece arrives if I can make a complete product for the incomplete 
pieces. When all the pieces have arrived, a complete product is made and
all the pieces are removed. This makes partial products available soon 
after the first piece arrives but eventually the partial products are 
replaced with a complete product. I've been running the script all day and 
there are no incomplete products that are over 30 minutes old.  I feel 
confident that the old script should be replace with this new script, what 
do you think?


Robb...

~ldm/logs/makeRadarII.log:

KFTG 20091030 1621 Partial product created
KFTG 20091030 1651 Partial product created
KFTG 20091030 1721 Partial product created
KFTG 20091030 1908 Partial product created
KFTG 20091030 2004 Partial product created
KFTG 20091030 2008 Partial product created
KFTG 20091030 2034 Partial product created
KFTG 20091030 2038 Partial product created
KFTG 20091030 2103 Partial product created
KFTG 20091030 2108 Partial product created


On Thu, 22 Oct 2009, Jeff Weber wrote:

> Thanks for the detailed response :)
>
> Heck, why meet, Robb...is this something you think you could handle?
>
> If not, let's get together Thurs aft
>
> Thanks,
>
> Jeff
> ---
>
> On Wed, 21 Oct 2009, Tom Yoksas wrote:
>
> > Hi Jeff,
> >
> > re:
> > > I would like to meet to discuss the writing of a decoder to insure proper
> > > order and complete receipt of the full scan of level II radar files.
> > >
> > > Anytime thurs or fri works for me.
> >
> > I have an Africa Initiative meeting from 11:30 - ?? tomorrow (Thursday),
> > but any time after that or on Friday is OK with me.
> >
> > It seems to me that the pieces of a Level II volume scan can be written
> > to disk and then assembled after all have been received.  The tricky
> > part is that one needs to receive the final piece to know how many
> > pieces there are/should be and then know how long to wait before
> > deciding that all of the pieces have not been received before exiting.
> >
> > Here is a snippit of a 'notifyme' listing for KFTG today (it just so
> > happens that the sequence of pieces for this volume scan are in
> > order):
> >
> > Oct 21 19:08:26 notifyme[3769] INFO:    12632 20091021190220.884 NEXRAD2 88001  
> > L2-BZIP2/KFTG/20091021190220/88/1/S/V03/0
> > Oct 21 19:08:27 notifyme[3769] INFO:    41835 20091021190222.965 NEXRAD2 88002  
> > L2-BZIP2/KFTG/20091021190220/88/2/I/V03/0
> > Oct 21 19:08:27 notifyme[3769] INFO:    60408 20091021190225.064 NEXRAD2 88003  
> > L2-BZIP2/KFTG/20091021190220/88/3/I/V03/0
> > Oct 21 19:08:27 notifyme[3769] INFO:    61916 20091021190229.154 NEXRAD2 88004  
> > L2-BZIP2/KFTG/20091021190220/88/4/I/V03/0
> >  ...
> > Oct 21 19:08:28 notifyme[3769] INFO:    18450 20091021190615.685 NEXRAD2 88067  
> > L2-BZIP2/KFTG/20091021190220/88/67/I/V03/0
> > Oct 21 19:08:28 notifyme[3769] INFO:    16791 20091021190622.752 NEXRAD2 88068  
> > L2-BZIP2/KFTG/20091021190220/88/68/I/V03/0
> > Oct 21 19:08:28 notifyme[3769] INFO:    16010 20091021190626.810 NEXRAD2 88069  
> > L2-BZIP2/KFTG/20091021190220/88/69/I/V03/0
> > Oct 21 19:08:28 notifyme[3769] INFO:    16232 20091021190636.732 NEXRAD2 88070  
> > L2-BZIP2/KFTG/20091021190220/88/70/E/V03/0
> >
> > Notice that the last piece contains both the end of product indicator
> > and the exptected number of pieces:
> >
> > .../KFTG/20091021190220/88/70/E/V03/0
> >                            ^  ^______ end piece indicator
> >                            |_________ piece number/exptected number of pieces
> >
> > The product ID for the last piece shows that we should expect 70 pieces
> > for this particular product.  If the pieces were being written to
> > individual files, then it is easy enough to sort the files and cat them
> > into a single output file.
> >
> > I suggest that the assembly process/"decoder" be kicked off upon
> > receipt of the end of product piece.  It would then sort the pieces it
> > has received and create an output file and delete individual pieces_if_
> > all of the pieces are accounted for.  If one or more piece is missing,
> > it could reschedule itself in 'n' minutes (1 minute?) to check again.
> > If a rescheduling approach is taken, then the assembler/"decoder"
> > should pass itself a count of how many times it has been rescheduled,
> > and give up after some reasonable number (like 5?) of attempts.  On the
> > final pass, it should create an output file that is as complete as it
> > is likely to be, and provide some indication that the file is not
> > complete (log file output, or a special suffix, etc.).
> >
> > The above scheme is not foolproof: what happens if the missing piece(s)
> > are received after the assembly process has run the maximun number of
> > times?  I guess that the pieces could be resorted and then concatenated
> > into an output file that is then complete, but this would mean that one
> > would need to keep the pieces on disk for some undetermined amount of
> > time and kick off checking upon receipt of each piece.  It is my belief
> > that checking upon receipt of each piece would not work as the system
> > would get bogged down.  I say this because I tried logging the arrival
> > of each Nexrad Level III product, and the system rapidly became unusable
> > because of I/O load.
> >
> > Cheers,
> >
> > Tom
> > --
> > +----------------------------------------------------------------------------+
> > * Tom Yoksas                                            UCAR Unidata Program *
> > * (303) 497-8642 (last resort)                                 P.O. Box 3000 *
> > * address@hidden                                  Boulder, CO 80307 *
> > * Unidata WWW Service                            http://www.unidata.ucar.edu/*
> > +----------------------------------------------------------------------------+

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================