[LDM #ZNI-140417]: HRRR LDM Issue

Hi Brad,

re:
> We are attempting to ingest the new HRRR model data from noaaport onto our 
> LDM queue
> using pqing:
> 
> (Excerpt from ldmd.conf)
> exec    "pqing -l /home/engineer/ldm/logs/pqing-6002.log -f WMO -P 6002 
> noaa-web-s"
> 
> The HRRR bulletins are not being saved into the queue.  Checking the pqing 
> log file, we
> are seeing the following errors for every HRRR bulletin.
> 
> We are not seeing this issue with any other bulletins.
> 
> Oct 01 16:55:08 pqing[3836] NOTE: YZCA00 KWBY 011600: no end of product
> Oct 01 16:55:08 pqing[3836] ERROR: Not a WMO GRIB format message.     729631  
>        252  YZCA00 KWBY 011600
> Oct 01 16:55:09 pqing[3836] NOTE: YZCA01 KWBY 011600: no end of product
> Oct 01 16:55:09 pqing[3836] ERROR: Not a WMO GRIB format message.     699891  
>        253  YZCA01 KWBY 011600
> Oct 01 16:55:10 pqing[3836] NOTE: YACA98 KWBY 011600: no end of product
> Oct 01 16:55:10 pqing[3836] ERROR: Not a WMO GRIB format message.    1208773  
>        254  YACA98 KWBY 011600
> Oct 01 16:55:11 pqing[3836] NOTE: YXCA86 KWBY 011600: no end of product
> Oct 01 16:55:11 pqing[3836] ERROR: Not a WMO GRIB format message.    1523738  
>        255  YXCA86 KWBY 011600
> 
> Can you help us with this?

The issue is that 'pqing' was written before the appearance of GRIB2,
and its GRIB processing logic does not understand GRIB2 messages.  In a
response to another user who was (still is?) trying to do much the same
thing as you, our LDM developer commented:

 "Enhancing pqing(1) to support the ingestion of GRIB2 messages should be
  straightforward though non-trivial. It would probably take a week. I can
  add it to the issue-list  on the LDM's GitHub repository but there's no
  telling when I would get around to it."

Question:

- have you investigated using 'pqinsert' for GRIB2 products?

Comment:

Bundled with recent versions of the LDM is an ancillary package
named 'gribinsert'

'gribinsert' was developed for use in creating the IDD CONDUIT data  stream.
As it is currently constituted, 'gribinsert' can be used to:

- read a monolithic output file created by a model

- extract individual GRIB2 messages

- crack the individual GRIB2 messages to extract information that
  is then used to create a useful LDM/IDD Product ID

- insert the individual GRIB2 messages into an LDM queue

- create a manifest of all of the GRIB2 messages that were inserted
  into the LDM queue AND those that were not because they already
  exist in the queue (by virtue of their MD5 signatures matching
  a product already in the queue)

- send the manifest in the same datatream that the GRIB2 messages
  were sent to

Given that 'gribinsert' does a lot of what you are using 'pqing' for,
it (or 'pqinsert') might be the easiest/quickest thing for you to use
and/or adapt for inserting ingested GRIB2 messages from NOAAPort into
LDM queues.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: ZNI-140417
Department: Support LDM
Priority: Normal
Status: Closed