[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDD #MIY-175774]: "WARN" Log Message in Concatenate NEXRAD2 Script

This archive contains answers to questions sent to Unidata support through mid-2025. Note that the archive is no longer being updated. We provide the archive for reference; many of the answers presented here remain technically correct, even if somewhat outdated. For the most up-to-date information on the use of NSF Unidata software and data services, please consult the Software Documentation first.

  • Subject: [IDD #MIY-175774]: "WARN" Log Message in Concatenate NEXRAD2 Script
  • Date: Thu, 28 Jan 2016 14:17:15 -0700
Ziv,

The reason bzip2 is saying that the file is corrupted is that the data product 
coming across from LDM isn't actually a bzip2-ed file, but rather it's a block 
of data that has had bzip2 compression applied, plus a 4 byte head prepended 
that gives the length of the compressed block:

[4-byte length][BZipped Level 2 data]

Additionally, the first chunk of every file also has a 24-byte volume header 
*before* the compression stuff, so like this:

[24-byte volume header][4-byte length][Bzipped Level 2 data]

(More details here if *really* interested: 
http://www.roc.noaa.gov/wsr88d/PublicDocs/ICDs/2620010E.pdf)

It may be possible to strip off those headers and hand the data to bzip2, but 
I've never done it--we're beyond my bash abilities here.

If you can install Python 3.5 (which you could install as a user, no need for a 
system install, from here: http://conda.pydata.org/miniconda.html), you could 
use the Python script I have for doing this very task here: 
https://github.com/Unidata/ldm-alchemy (l2assemble.py)

Ryan

> > > I am playing around with a bash decompression script to EXECUTE on the
> > > files as they are written but before they are concatenated.
> > >
> > > However, I am getting the following bzip2 error from test runs on files
> > in
> > > the .tmp folder:
> > >
> > > bzip2: Can't guess original name for KPOE_20160122_202252 -- using
> > > KPOE_20160122_202252.out
> > >
> > > bzip2: Compressed file ends unexpectedly;
> > > perhaps it is corrupted?  *Possible* reason follows.
> > > bzip2: No such file or directory
> > > Input file = KPOE_20160122_202252, output file = KPOE_20160122_202252.out
> > >
> > > It is possible that the compressed file(s) have become corrupted.

Ticket Details
===================
Ticket ID: MIY-175774
Department: Support IDD
Priority: Normal
Status: Open