20050504: LDM: PIPE-ing to scripts

Scott,

>Date: Wed, 04 May 2005 13:04:37 -0800
>From: "Scott Swank" <address@hidden>
>Organization: NOAA/NWS
>To: Steve Emmerson <address@hidden>
>Subject: Re: 20050504: LDM: PIPE-ing to scripts
>Keywords: 200504291922.j3TJMbKx001883

The above message contained the following:

> Depending on the datatype and the decoder, I will know whether it's 
> binary data or text data.  Won't cat > $1 fail on binary data?  If not, 
> great, but if so, how do you recommend I redirect stdin to output the file?

The cat(1) utility works just fine on non-textual data.  You can verify
this yourself via the command

    cat $HOME/bin/rpc.ldmd | wc

Any decoder you write, however, shouldn't just use the cat(1) utility
because that's just a less efficient way of performing the FILE action.
Typically, decoders invoked via PIPE actions transform the data-product
into something that is more easily accessible by a subsequent analysis
and display package (e.g., GEMPAK, McIDAS).

> > There are LDM decoders that have handled terabytes of binary data just 
> > fine.
> How are they getting the data out of stdin for manipulation?

The compiled decoders that handle non-textual data-products typically
read the data from the standard-input stream using either the fread() or
read() functions.

One can also write a perl(1) script that does something similar.

I take it you're not familiar with UNIX programs that read non-textual
data from standard-input.  You might want to consult with a UNIX person
at your site.

> Thanks,
> Scott

Regards,
Steve Emmerson