20050504: LDM: PIPE-ing to scripts

Scott,

>Date: Wed, 04 May 2005 10:59:16 -0800
>From: "Scott Swank" <address@hidden>
>Organization: NOAA/NWS
>To: Steve Emmerson <address@hidden>
>Subject: Re: 20050504: LDM: PIPE-ing to scripts
>Keywords: 200504291922.j3TJMbKx001883

The above message contained the following:

> Thanks for all your help! It seems to be working fine now.  I missed the 
> part about putting the cat > $1 at the beginning of my script, but it is 
> performing fine.

Glad to hear it.

> Two questions:
> 1) This works fine for text products, but how do I redirect stdin for 
> binary products?

The PIPE action causes the same redirection of the standard-input stream 
regardless of whether or not the data-product in question contains
non-textual data.  It is the decoders responsibility to know if its
input will be textual or not.

There are LDM decoders that have handled terabytes of binary data just 
fine.

> 2) We will eventually be handling upwards of 75,000 products daily on 
> this queue internally.  Is it possible for the stdin redirects to get 
> confused and output the wrong data to the wrong files or to mix and mesh 
> data as more and more is placed in the queue, or does stdin track 
> seperately for each PIPE call?  If I read your email correctly, the PIPE 
> calls should all execute in sequence thus making this question 
> irrelevant, but I just want to make sure.

The pqact(1) utility was explicitly designed to handle just this
situation -- and does so daily with terabytes of data.  Just ensure that
your selection criteria (i.e., feedtype and pattern) match only the
data-products of interest for that particular decoder and that all
decoders play nice with each other (e.g., they all decode into different
files).

> Thanks,
> Scott

Regards,
Steve Emmerson