Hi Jeff,
re:
I would like to meet to discuss the writing of a decoder to insure proper
order and complete receipt of the full scan of level II radar files.
Anytime thurs or fri works for me.
I have an Africa Initiative meeting from 11:30 - ?? tomorrow (Thursday),
but any time after that or on Friday is OK with me.
It seems to me that the pieces of a Level II volume scan can be written
to disk and then assembled after all have been received. The tricky
part is that one needs to receive the final piece to know how many
pieces there are/should be and then know how long to wait before
deciding that all of the pieces have not been received before exiting.
Here is a snippit of a 'notifyme' listing for KFTG today (it just so
happens that the sequence of pieces for this volume scan are in
order):
Oct 21 19:08:26 notifyme[3769] INFO: 12632 20091021190220.884 NEXRAD2 88001
L2-BZIP2/KFTG/20091021190220/88/1/S/V03/0
Oct 21 19:08:27 notifyme[3769] INFO: 41835 20091021190222.965 NEXRAD2 88002
L2-BZIP2/KFTG/20091021190220/88/2/I/V03/0
Oct 21 19:08:27 notifyme[3769] INFO: 60408 20091021190225.064 NEXRAD2 88003
L2-BZIP2/KFTG/20091021190220/88/3/I/V03/0
Oct 21 19:08:27 notifyme[3769] INFO: 61916 20091021190229.154 NEXRAD2 88004
L2-BZIP2/KFTG/20091021190220/88/4/I/V03/0
...
Oct 21 19:08:28 notifyme[3769] INFO: 18450 20091021190615.685 NEXRAD2 88067
L2-BZIP2/KFTG/20091021190220/88/67/I/V03/0
Oct 21 19:08:28 notifyme[3769] INFO: 16791 20091021190622.752 NEXRAD2 88068
L2-BZIP2/KFTG/20091021190220/88/68/I/V03/0
Oct 21 19:08:28 notifyme[3769] INFO: 16010 20091021190626.810 NEXRAD2 88069
L2-BZIP2/KFTG/20091021190220/88/69/I/V03/0
Oct 21 19:08:28 notifyme[3769] INFO: 16232 20091021190636.732 NEXRAD2 88070
L2-BZIP2/KFTG/20091021190220/88/70/E/V03/0
Notice that the last piece contains both the end of product indicator
and the exptected number of pieces:
.../KFTG/20091021190220/88/70/E/V03/0
^ ^______ end piece indicator
|_________ piece number/exptected number of pieces
The product ID for the last piece shows that we should expect 70 pieces
for this particular product. If the pieces were being written to
individual files, then it is easy enough to sort the files and cat them
into a single output file.
I suggest that the assembly process/"decoder" be kicked off upon
receipt of the end of product piece. It would then sort the pieces it
has received and create an output file and delete individual pieces_if_
all of the pieces are accounted for. If one or more piece is missing,
it could reschedule itself in 'n' minutes (1 minute?) to check again.
If a rescheduling approach is taken, then the assembler/"decoder"
should pass itself a count of how many times it has been rescheduled,
and give up after some reasonable number (like 5?) of attempts. On the
final pass, it should create an output file that is as complete as it
is likely to be, and provide some indication that the file is not
complete (log file output, or a special suffix, etc.).
The above scheme is not foolproof: what happens if the missing piece(s)
are received after the assembly process has run the maximun number of
times? I guess that the pieces could be resorted and then concatenated
into an output file that is then complete, but this would mean that one
would need to keep the pieces on disk for some undetermined amount of
time and kick off checking upon receipt of each piece. It is my belief
that checking upon receipt of each piece would not work as the system
would get bogged down. I say this because I tried logging the arrival
of each Nexrad Level III product, and the system rapidly became unusable
because of I/O load.
Cheers,
Tom
--
+----------------------------------------------------------------------------+
* Tom Yoksas UCAR Unidata Program *
* (303) 497-8642 (last resort) P.O. Box 3000 *
* address@hidden Boulder, CO 80307 *
* Unidata WWW Service http://www.unidata.ucar.edu/*
+----------------------------------------------------------------------------+