[LDM #MKT-798449]: GOES 16/17 for Gempak

Hi Clint,

Sorry for the slow reply.  I saw your inquiry when it first came in, and I
intended to respond ASAP, but one thing after another came up to delay my
being able to get to a reply...

re:
> I'm trying to get the GOES 16/17 imagery for our Gempak systems, but am having
> troubles with products not being filed by our LDM.  I have the pqact.conf 
> entry
> from GitHub (and checked that whitespace is tabs) and restarted the LDM - no 
> new
> directories were created and, obviously, no new datafiles were created.

Since Michael posted the examples on GitHub before the NIMAGE feed revamp was
completed, it is most likely that the examples that he posted were/are not
quite correct.

I put together a web page to highlight information on the revamped NIMAGE
feed:

Unidata HomePage
http://www.unidata.ucar.edu
  Data -> Satellite Data
  https://www.unidata.ucar.edu/data/index.html#satellite
    NIMAGE (FT21, IMAGE)
    https://www.unidata.ucar.edu/data/nimage.html

The page contains example LDM/IDD Product IDs and links to an
example pattern-action file actions.

re:
> I watched the feed to make sure data were flowing, but I noticed the product
> was coming as *.nc, not *.nc4 as specified in the GitHub pattern.
> 
> [ldm@edex etc]$ ldmadmin watch -f NIMAGE
> 20200108T134415.386355Z                pqutil[8021]         
> pqutil.c:display_watch() INFO     3679904 20200108134405.291441 NIMAGE 000  
> /data/ldm/pub/native/satellite/GOES/GOES16/Products/CloudAndMoistureImagery/CONUS/Channel13/20200108/OR_ABI-L2-CMIPC-M6C13_G16_s20200081341170_e20200081341170_c20200081341170.nc
> 20200108T134416.396210Z                pqutil[8021]         
> pqutil.c:display_watch() INFO     2963847 20200108134405.781390 NIMAGE 000  
> /data/ldm/pub/native/satellite/GOES/GOES16/Products/CloudAndMoistureImagery/CONUS/Channel10/20200108/OR_ABI-L2-CMIPC-M6C10_G16_s20200081341170_e20200081341170_c20200081341170.nc
> 20200108T134417.412983Z                pqutil[8021]         
> pqutil.c:display_watch() INFO     3751964 20200108134405.902455 NIMAGE 000  
> /data/ldm/pub/native/satellite/GOES/GOES16/Products/CloudAndMoistureImagery/CONUS/Channel11/20200108/OR_ABI-L2-CMIPC-M6C11_G16_s20200081341170_e20200081341170_c20200081341170.nc
> 20200108T134418.422687Z                pqutil[8021]         
> pqutil.c:display_watch() INFO     2435742 20200108134406.293146 NIMAGE 000  
> /data/ldm/pub/native/satellite/GOES/GOES16/Products/CloudAndMoistureImagery/CONUS/Channel09/20200108/OR_ABI-L2-CMIPC-M6C09_G16_s20200081341170_e20200081341170_c20200081341170.nc

re:
> I removed the trailing "4" from the pattern in pqact.conf and restarted the 
> LDM.  Still no GOES16/17.

Again, the example actions that Michael posted on GitHub are evidently 
incorrect.
Try using the example actions linked from the web page I referenced above as 
the basis
for the actions you use.

re:
> Now, I'm not regular expression expert, but I don't see how the pattern gets 
> past
> the "Products/CloudAndMoistureImagery" part of the product name so that the 
> region
> (CONUS, here), channel, date, and time components end up in the right 
> parenthesized
> groups to properly structure the full path and filename. So, I thought maybe 
> I'm
> just seeing the tiled imagery (but I thought that was on the SATELLITE 
> feedtype, not
> NIMAGE) and the nc4 is an important component of the stitched-together 
> imagery.

Michael added support for GOES-16/17 imagery that has been reconstituted into
full scenes by stitching together the tiles that are delivered in NOAAPort.
I revamped the NIMAGE feed to include those reconstituted images, and I
used a naming convention for the files that follows the example defined
in the PUG (GOES-R Product User's Guide).  The GOES-16/17 imagery that
is available in the SATELLITE feed is NOT supported by either GEMPAK or
AWIPS.

re:
> I then figured I'd start up a notifyme to look for *.nc4 products. It's
> been running for some time now (~0.5 hour) and I haven't seen anything.
> I can do the same, looking for *.nc and get what I see with ldmadmin watch.

What you are seeing in your 'ldmadmin watch -f NIMAGE' output is correct,
and the actions you use to FILE the products should match it.

re:
> Trying to follow the posted instructions, but not getting anywhere ...

I have to apologize for the incorrect information in the GitHub examples
that Michael posted before the NIMAGE feed was revamped.  That they
should be removed or, at least, updated is a given.  The problem is that
we don't have a replacement for the absence created by Michael's tragic
loss, so things like this have fallen through the cracks.

NB:

There are additional pieces of information that you should/must be
made aware of while trying to get the GOES-16/17 imagery working
in GEMPAK.  Here are two that we know about:

1) fully qualified pathnames of data files need to be less than or
   equal to 108 characters in length in GEMPAK

   Why?  Good question why the GEMPAK developers imposed such a
   restriction on supported names!!

   What this implies is that you can not simply FILE the products
   using the full Product ID for the GOES-16/17 images in NIMAGE.
   You will have to choose a directory structure that results in
   shorter pathnames to meet the GEMPAK restriction.

2) GEMPAK users have reported that the support of the GOES-16/17
   imagery is not perfect

   Four things have come to light via user reports:

   - the display of images colors/gray values is off

     There is a GEMPAK table that sets the min/max pixel values for an image:

     $GEMPAK/tables/sat/imgtyp.tbl

     The max values for GOES16 and GOES17 non-visible imagery need to be 
adjusted.
     I tested with a Channel 4 image and had to set the max value to 128. I did
     this because the NetCDF file says that the data range is 0-4095. However to
     map to the color tables, the data is scaled to 0-255 by shifting to the 
right
     4 bits. This should work, but I found that the data range for the image I
     tested is actually 0-2082. When this range is shifted 4 bits, the range 
becomes
     0-(about)130. Modifying imgtyp.tbl with the new max value makes the colors 
look
     correct.

   - GOES-16/17 GLM data does not display in GEMPAK

     Michael did NOT add support for GLM data (available in the SATELLITE
     feed) display to GEMPAK.

     GLM value added L2 images/grids that are created by TTU are being 
distributed
     in the IDD NIMAGE feed.  Given that their internal structure is like that 
in
     the L2 products that the NWS is distributing in NOAAPort, these may
     work in GEMPAK, but as far as I know, display of these value added
     GLM products has never tested here in the UPC, so we do not have
     any information on what changes neede to be made in datatype.tbl.

   - spurious lines experienced in contour plots that are overlain on
     top of the GOES-16/17 imagery that is being distributed in
     NOAAPort

     Here is what Scott Jacobs had to say about this problem and his
     workaround:

     "The problem is when a contour line breaks to allow a label to be
     plotted. I can see what is happening, but when I fix the contour
     lines, the map is drawn incorrectly.

     However, there is a way to avoid the problem. In the LINE parameter,
     there is a setting that allows a line to get a label with breaking
     the line.

     LINE = color / line type / line width / labeling

     If the labeling value is a positive number N, then every Nth contour
     gets labeled. The default is 1 so every contour gets a label. If the
     value is 0, then no labels are drawn. If the value is a negative number,
     -N, then every Nth contour gets labeled, but there is no line break.

     The user could use this feature to avoid the spurious lines until I can
     work on this some more. This is all done by the user at runtime, so no
     code needs to be changed, right now."
     
   - there is an issue in overlying wind barbs on top of GOES-16/17 imagery
     in GEMPAK

     Here is the report from a user:

     "I am trying to overlay wind barbs (any level) on the GOES16 and GOES17
     satellite data in gempak.  As you can see from my attached image it is
     clearly off. There was a similar problem which was resolved when overlaying
     contours.  The fix for that was to use the line option in GDPLOT and make
     the 4th spot in as a "-1"  (eg; line= 6/1/2/-1).  I looked but didn't see
     anything like that with the Wind parameter. So hopefully someone knows of
     a way to correct this.  The streamline do plot correctly but the barbs do
     not."

     We do know know of a workaround for this.  Hopefully, a GEMPAK expert
     can help out in figuring this out.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: MKT-798449
Department: Support GEMPAK
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.