Gerry,
Good question. But not as simple as "/E" to determine- which has
its own flaws if you received the parts out of order from multiple
upstream hosts.
First off, every file inserted in CONDUIT has a .status file associated
with it,
which is inserted as the very last product of the file, however, due to
the
possibility of using multiple feed requests, you can't guarantee that it
will be the last
fLDM product received.
You have the following options:
1) On receipt of the 126 hour status product, you can use the sequence
number at the
end of the product (eg the 6 digit number following the !.*! pattern, or
(......)$ ) to tell you how many pieces there should be (since the
counter starts at 0, the number of
the status product will also be the number of grib messages to be
received).
You can then wait a short amount of time until either-
a) all pieces have been received
b) you haven't gotten any more pieces for a given timeout
The check in "b" would prevent waiting for ever for something that never
arrives.
I'd suggest 5 minutes as a max, but probably as little as 30 seconds
would suffice.
2) Use the inserted size included within the .status file to check for
the number of bytes files for the aggregate 126 hour file, and wait
until
either:
a) the file size is >= to the inserted size
b) no new data has been added for a given timeout
In application, I have to file the individual pieces to separate files
for performing the
reception stats as in:
http://motherlode.ucar.edu/cgi-bin/ldm/conduit_reception_new.csh?/data/nccf/com/gfs/prod/gfs.2007082912
however, most sites would probably find it better off to file by
forecast hour as in (2).
A last option, if you were keying on decoded output rather than FILE's
grib parts, would be to check the decoded file for completion/inactivity
(eg just "b" in
number 2 without "a".
Attached is a example csh script that takes 3 arguments (directory,
filename, and size_in_bytes extracted from the .status file inserted
string). The script will sleep a specified interval and recheck. It will
proceed if either:
1) The specified file is greater than or equal to the size in bytes
2) It has looped 10 times (eg 5 minutes at 30 seconds) without
the file being touched (eg the file is no longer actively being
updated)
3) It has looped 10 times without the file existing
Just in case you find it useful.....
Steve Chiswell
Unidata User SUpport
On Wed, 2007-08-29 at 13:47 -0500, Gerry Creager wrote:
> Chiz,
>
> Donna and I are scheming about using a pqact trigger to start
> preprocessing when GFS 0.5 deg hour 126 comes in. Is there a flag when
> the last of the data is received we can trigger on? Is it something
> similar to the end of a volume scan in Level II data
>
> Thanks, gerry
--
Steve Chiswell <address@hidden>
Unidata
Attachment:
donetest.csh
Description: C-Shell script