20050623:LDM data storage

Hi Lou,

Thought I would let everyone chime in....:)

Take and use as you wish...

Looks kinda moist in Asheville for next week, let's hope for no more
flooding!

Cheers,


Jeff
---------------------------------------------------------------------
Jeff Weber                                    address@hidden        :
Unidata Program Center                        PH:303-497-8676        :
University Corp for Atmospheric Research      3300 Mitchell Ln       :
http://www.unidata.ucar.edu/staff/jweber      Boulder,Co 80307-3000  :
---------------------------------------------------------------------

On Thu, 2005-06-23 at 13:27, Steve Emmerson wrote:

> Jeff,
>
> The following pqact(1) configuration-file entry:
>
>       NEXRD2 ^L2-(BZIP2|ZLIB)/((....)/(........)(......).*)
>              FILE    /home/ldm/nexradII/tmp/\3\4_\5.bz2
>
> would append data-products that are ZLIB-compressed to files with a
> BZIP-2 extension (.bz2).  This is probably not a good idea.
>
> The ".*" at the end is unnecessary -- as is the extra set of
> parentheses
>
> I'd be inclined to use something like this:
>
>       NEXRAD2 L2-BZIP2/(....)/(........)(......)
>               FILE    /home/ldm/nexradII/tmp/\1-\2T\3.bz2
>
>       NEXRAD2 L2-ZLIB/(....)/(........)(......)
>               FILE    /home/ldm/nexradII/tmp/\1-\2T\3.gz
>
> "NEXRAD2" is the preferred abreviation.
>
> The substring "\2T\3" creates timestamps like "20050623T171519" --
> which is an ISO standard (so we might as well get used to it :-).
>
> Is the timestamp in the product-identifier for all NEXRAD2 data-products
> in a volume-scan the same?  If not, are they the same for a given
> elevation?
>
> I'm conncerned about how many data-products will go into one file.
>
> Regards,
> Steve Emmerson
>

-----------------------------------------------------------------------
The Level2 data file is a single file per time (broken up into
all those radial/level chunks). There would be somewhere between
35 and 65 16Kbyte chunks in a file (generally files
are between 600K and 1.5M).

There shouldn't be any ZLIB strings in the data stream as that was
a legacy pre-craft compression. The Level2 interface control document
specifies bzip compression- so I don't know why that should be
part of the pattern (except it was in one of the old CRAFT/OU docs.

Chiz

> ---------- Forwarded message ----------
> Date: Thu, 23 Jun 2005 14:49:01 -0400
> From: Lou Vasquez <address@hidden>
> To: Jeff Weber <address@hidden>
> Subject: LDM data storage
>
> Hi,
>
> I was looking into storing some of the LDM data because NCDC may have a
> planned power outage that exceeds their UPS backup.  I've spoken with
> Alan Hall of NCDC and we were looking at simply adding an entry to the
> pqact.conf of the form.
>
> NEXRD2 ^L2-(BZIP2|ZLIB)/((....)/(........)(......).*)
>        FILE    /home/ldm/nexradII/tmp/\3\4_\5.bz2
>
> I just wanted to run it by you and see if you had any suggestions as to
> whether this is the best way to handle this.
>
> Thanks,
> Lou