Re: 20060626: trouble converting GRIB2 products with cnvgrib

On Tue, 27 Jun 2006, Steve Chiswell wrote:

> Robb & John,
>
> The message below indicated that the user had downloaded files from
> motherlode. The pattern/action that I use to file the conduit
> products on motherlode in pqact.conduit ensures that the inventory
> product that begins with .status is not processed by the FILE action.
> These files are available for users to ftp from the unidata account
> on motherlode- so it is not these files the user obtained.
>
> However, I found that there is a separate pqact.threddsconduit file
> where the products are being files separately for thredds. All of these
> actions are not using the ^/afs beginning, so the actions
> are filing the ascii text inventory that is .status./afs.etc...


hi,

i changed all the actions in pqact.threddsconduit file on motherlode to
have the pattern start with ^/afs.*  this should fix the problem. thanks
Chiz for catching this.

robb...

>
> The actual problem is that the user's program is too brittle to
> properly look for a GRIB product in a file with other
> stuff (such as would be the case with WMO headers etc in a file),
> but it would prevent confusion if the thredds pattern/action
> just put the grib products in the file and not the inventory.
>
> Chiz
>
>
> On Mon, 2006-06-26 at 14:36, Tom Yoksas wrote:
> > >From:  Rorik Peterson <address@hidden>
> > >Organization:  UAF
> > >Keywords:  200606262015.k5QKFfSZ017377 LDM GRIB2 CONDUIT status message
> >
> > Hi Rorik,
> >
> > re:
> > >GRIB2/cnvgrib gurus and other savy problem solvers,
> > >
> > >I'm trying to convert GRIB2 to GRIB1 files on a 64-bit Athlon linux
> > >machine using 'cnvgrib'.  I've tried 2 different types of files and
> > >succeeded with NAM Alaska grid 242, and failed with 0.5-degree GFS.  I
> > >get them both via the LDM/IDD.  With the GFS files, I actually succeed
> > >converting the first 270 or so messages; I've tried several different
> > >GFS files, and it usually stops around here.  There should be several
> > >thousand records according to 'degrib'.  I used a debugger to see if I
> > >could find out anything, although I don't know much about GRIB2.  This
> > >is what I see happening:
> > >The program is generating an index of all records in the GRIB2 file.  It
> > >does this by looking for characters 'GRIB...[12]' at the beginning of
> > >each record, reads forward to where the next record should be, and looks
> > >again.  After about 270 or so records, the next record starts with
> > >'/afs/.nwstg.nws.noaa.gov/ftp/SL.us008001/ST.opnt/MT.gfs-CY' instead of
> > >'GRIB...[12]'.
> > >This can't be a coincedence, being part of the product name I get from
> > >the IDD.
> >
> > This seems to indicate that you are filing the ASCII status file
> > sent in the CONDUIT stream along with the binary GRIB2 messages into
> > a single output file.  You can learn more about the catalog file
> > in the Unidata web page on CONDUIT data:
> >
> > Unidata HomePage
> > http://www.unidata.ucar.edu/
> >
> >   Data
> >   http://www.unidata.ucar.edu/data/
> >
> >     CONDUIT
> >     http://www.unidata.ucar.edu/data/
> >
> >       What is CONDUIT?
> >       http://www.unidata.ucar.edu/data/conduit/index.html
> >
> >         CONDUIT
> >         http://www.unidata.ucar.edu/data/conduit/ldm_idd/index.html
> >
> > Here is the relevant comment in the last URL:
> >
> >   Since the data is injected as individual products, a status file
> >   consisting of the injection time, original file size and data file
> >   name is inserted at the end of the product sequence, with the name:
> >   .status.ncep_file_name end_sequence  The status file will contain a
> >   catalog of each individual LDM product identifier within the original
> >   data file, as well as the insertion status code returned
> >   by the LDM.
> >
> > Please check your ~ldm/etc/pqact.conf action(s) for CONDUIT data
> > and make sure that you process the status file differently than
> > the GRIB/GRIB2 messages.
> >
> > >I don't know if this is suppose to be appearing within the
> > >GRIB2 file or not? (seems like it shouldn't, a waste of space, but I
> > >don't know the GRIB2 spec that well.)
> >
> > It isn't.
> >
> > >At this point, the program stops
> > >generating an index, and only processes these 270 or so records.
> >
> > This is consistent with my theory that the status file is being written
> > in the single output file.
> >
> > >Being curious, I crudely scanned the entire 1.2GB file and matched for
> > >the 13 character '/afs/.nwstg.n' pattern, and it only begins to appear
> > >at the approximate point in the GRIB2 file that record 270 is, and
> > >occurs several thousand times after that.  Again, not a coincedence, I
> > >figure (although I haven't done the math for a 13 byte sequence in 1.2GB).
> > >
> > >At this point, I'm stuck.  Is the problem with 'cnvgrib', the GRIB2
> > >files, my LDM, or perhaps something else?
> >
> > The problem is in the ~ldm/etc/pqact.conf action you are using to FILE
> > the GRIB/GRIB2 products.
> >
> > >I reproduced this problem with a GRIB2 file I downloaded straight from
> > >Unidata's motherlode http portal, suspecting problems with my LDM.
> >
> > This must have been one of the status files.
> >
> > Cheers,
> >
> > Tom
> > --
> > +----------------------------------------------------------------------------+
> > * Tom Yoksas                                            UCAR Unidata 
> > Program *
> > * (303) 497-8642 (last resort)                                 P.O. Box 
> > 3000 *
> > * address@hidden                                  Boulder, CO 80307 *
> > * Unidata WWW Service                            
> > http://www.unidata.ucar.edu/*
> > +----------------------------------------------------------------------------+
> >
> > ===============================================================================
> > To unsubscribe ldm-users, visit:
> > http://www.unidata.ucar.edu/mailing-list-delete-form.html
> > ===============================================================================
>

===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================