[Datastream #ZYI-136966]: LDM nexrad2 feed

Hi Patrick,

I apologize for the slow reply...

re:
> I have noticed that the nexrad2 files were are getting from the LDM may
> not be complete before receiving the next file.

Correct.  There is no guarantee that all of the pieces of a volume scan
from one station will be sent before pieces from a different volume/station
are sent.

re:
> If I understand
> correctly...there are packets of data going through the IDD and there is
> a flag with the last packet to indicate to the LDM that the data is
> complete for that file.  So can a newer file be complete before an older
> file is?

Mostly correct.  Each piece of a NEXRAD2 volume has a sequence number in
its Product ID that indicates where the piece fits in the entire product.
The last piece's Product ID has an extra piece of information that indicates
that it is the last piece.

re:
> Below is a listing of my /ldm/nexrad2/KHTX directory to show you what I
> am talking about.  The file ".KHTX20100716212555 " is incomplete but
> there are two complete files that are newer (i.e., more recent timestamp
> in filename).  The incomplete file is eventually completed, but I just
> do not understand why it is completed after a newer file.

The pieces of a volume are not guaranteed to be in sequential order, nor
are they guaranteed to not be mixed with pieces from other volumes.

re:
> Based on file
> size alone, it almost seems like the newer file "KHTX20100716213036" is
> not complete.  Could the data packets/flags be getting mixed up for the
> two files?

This would depend on how you are packaging the pieces into a volume.

re:
> We use the dcnexr2 converter to do the uncompression of the
> data in the nexrad2 feed.  Could this decoder be our hold up?

I don't think that the culprit is GEMPAK's dcnexr2.  Rather, the
challenge is to assemble the pieces into whole volumes.

We did a lot of work here in the program center to develop a procedure
to reconstruct Level II volumes from pieces being received via the LDM/IDD.
The procedure we developed:

- writes the individual pieces to a site/volume-specific directory

- examines the contents of the site/volume-specific directory to
  see if all of the pieces have been received when the end piece
  is received.

  If all have been received, the volume is reconstructed from the pieces
  and the pieces and the temporary directory holding the pieces
  are deleted.

  If not all pieces have been received, the process that stitches the
  pieces back together sleeps for 60 seconds and then checks to
  see if all pieces have been received.

Our observation is that virtually all pieces for all volumes are being
received, and our procedure for reassembling the volumes is robust, but
a bit more complicated than the simple approach developed for GEMPAK
way back when.

We can provide our LDM pattern-action file actions and Perl stitching
script to you if you are interested.  Please let us know...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ZYI-136966
Department: Support IDD
Priority: Normal
Status: Closed