[LDM #PHE-874806]: LDM and filing of GOES-17 NIMAGE data

Hi Mike,

Since the answers to your questions are really GEMPAK driven, I moved
this inquiry from our LDM to GEMPAK support area.

re:
> I'm new to GOES-17 data, but not new to LDM/GEMPAK, etc.  I'm in discovery
> mode, just trying to get some basic data filing established before I take
> more steps into image processing.
> 
> System: vulcan.met.sjsu.edu
> LDM: version: 6.13.11; built: Jul 12 2019 12:07:02
> GEMPAK 7.5.1

Very good.

re:
> I have some simple pattern matching attempts in a pqact.images file.  This
> very simple match and action works and will file data:
> 
> NIMAGE  ^/data/ldm/pub/native/satellite/GOES/(.*)
> FILE    -close  -overwrite images/sat/\1
> 
> Data looks like this:
> [ldm@vulcan 20190723]$ pwd
> /usr/local/ldm/var/data/images/sat/GOES17/Products/CloudAndMoistureImagery/CONUS/Channel14/20190723
> [ldm@vulcan 20190723]$ ll
> total 10608
> -rw-r--r-- 1 ldm metrole 3620194 Jul 23 16:41
> OR_ABI-L2-CMIPC-M6C14_G17_s20192041626190_e20192041626190_c20192041626190.nc
> -rw-r--r-- 1 ldm metrole 3619580 Jul 23 16:49
> OR_ABI-L2-CMIPC-M6C14_G17_s20192041631190_e20192041631190_c20192041631190.nc
> -rw-r--r-- 1 ldm metrole 3617371 Jul 23 16:53
> OR_ABI-L2-CMIPC-M6C14_G17_s20192041636190_e20192041636190_c20192041636190.nc
> 
> But with this pathing I can't plot files with NMAP2.  I can get to the
> point where the data shows in the data selector, see image included, but no
> actual data.  Perhaps the path is too long or something.

The path length is definitely too long for GEMPAK due to its 108 character limit
on the fully qualified names that it can use.  The 108th character in the first
fully qualified pathname is the 'L' in the file name:

OR_ABI-L2-CMIPC-M6C14_G17_s20192041626190_e20192041626190_c20192041626190.nc

re:
> Really though I want to be able to use a more specific match and filing
> entries according to this page:
> 
> https://www.unidata.ucar.edu/blogs/news/entry/gempak-7-5-1
> 
> which I have entered as:
> 
/data/ldm/pub/native/satellite/GOES/GOES16/Products/CloudAndMoistureImagery/PuertoRico/Channel08/20190723/OR_ABI-L2-CMIPPR-M6C08_G16_s20192041810420_e20192041810420_c20192041810420.nc
 

> NIMAGE 
> ^/data/ldm/pub/native/satellite/GOES/(.*)/([^/]*)/([^/]*)/([0-9]{8})/GOES(16|17)_([^/]*)_([0-9]{8})_([0-9]{4})(..)_([^/]*).nc
> FILE    -close  -overwrite images/sat/\1/\2/\3/\3_\7_\8
> 
> This files nothing, but seems like it should.  And no errors in ldmd.log.
> Any ideas?

I think that Michael wrote this page before the final format of the NIMAGE 
Product IDs
for GOES-16/17 ABI L2 imagery was finalized. 

re:
> I also try to file according to this page
> https://www.unidata.ucar.edu/data/pqact.conf_nimage.html#L2_ACTION
> 
> NIMAGE        
> ^(.*)/(GOES1.)/(Products)/(.*)/((CONUS|FullDisk|Mesoscale-1|Mesoscale-2|PuertoRico|Alaska|Hawaii).*)/(20......)/(OR_...-L2.*.nc)
> PIPE  -close
> util/L2ProdFile.sh \1 \2 \3 \4 \5 \7 \8 logs/L2ProdFile.log
> 
> Here I get an ERROR Couldn't pipe product to decoder "-close
> util/L2ProdFile.sh

The reason for this is a little bit subtle.  LDM versions that contain a
registry (~ldm/etc/registry.xml) define the current working directory for
'pqact' invocations via the <datadir-path> setting in the <pqact> section.
Since you installed the LDM on vulcan using the default setting for
<datadir-path> in its registry, what is trying to be executed is:

/usr/local/ldm/var/data/util/L2ProdFile.sh

Undoubtedly, you do not have a ~ldm/var/data/util directory, so the script
L2ProdFile.sh is not being found.

What to do?

There are two options:

1) change the <datadir-path> value in the LDM registry to default to
   what older LDMs assumed

   Older LDMs assumed that the current working directory for 'pqact'
   invocations was the HOME directory of the LDM, ~ldm.

   If you want to setup your new LDM in the same way that you were used
   to setting up the LDMs you were familiar with, you would need to modify
   a couple of entries in the LDM registry.  This may be the way you will
   want to go given your familiarity with pattern-action file actions provided
   with Unidata GEMPAK.  We can discuss this at length if you decide to
   go this route.

2) change all relative path references to absolute path references
   in all LDM pattern-action file actions

   For the action in question, this would look like:

NIMAGE 
^(.*)/(GOES1.)/(Products)/(.*)/((CONUS|FullDisk|Mesoscale-1|Mesoscale-2|PuertoRico|Alaska|Hawaii).*)/(20......)/(OR_...-L2.*.nc)
   PIPE -close
   /usr/local/ldm/util/L2ProdFile.sh \1 \2 \3 \4 \5 \7 \8 
/usr/local/ldm/var/logs/L2ProdFile.log

   NB:

   - this assumes that you copied L2ProdFile.sh to the ~ldm/util directory

     You next comment says that you do.

   - you made sure that ~ldm/util/L2ProdFile.sh is executable

   - you want to write the L2ProdFile.log log files to the ~ldm/var/logs
     directory

re:
> but I do have the L2ProdFile.sh file in the util folder.

Yup, this was what I was assuming.

re:
> Here is the NMAP image:
> 
> [image: image.png]
> 
> My ultimate goal is not to plot images in NMAP, I'm just using as a quick
> tool to see if I have something.

I understand.  In order to get this to work, you will need to play by
GEMPAK rules, and one of those rules is that data file pathnames can not
exceed 108 characters in length (weird, but true).

re:
> My goal really is to replace some of our
> outdated automated image creation for research, instruction and the web
> page.  I think I may just need to be pointed in the right direction here.

A couple of things have come to light wrt to GEMPAK being able to display
the GOES-16/17 imagery that is now flowing in the IDD NIMAGE feed:

- one needs to adjust the maximum pixel value for these images in
  the GEMPAK imgtyp.tbl file.  The maximum pixel value needs to be
  set to 128.

  Here is the explanation that has been provided by a community GEMPAK
  expert on why this is needed:

  "There is a table that sets the min/max pixel values for an image.
   The table is $GEMPAK/tables/sat/imgtyp.tbl. The max values for GOES16
   and GOES17 non-visible imagery need to be adjusted. I tested with a
   Channel 4 image and had to set the max value to 128. I did this because
   the NetCDF file says that the data range is 0-4095. However to map to the
   color tables, the data is scaled to 0-255 by shifting to the right 4 bits.
   This should work, but I found that the data range for the image I tested
   is actually 0-2082. When this range is shifted 4 bits, the range becomes
   0-(about)130. Modifying imgtyp.tbl with the new max value makes the colors
   look correct."

- if you want to overlay contour plots on top of GOES-16/17 ABI L2 imagery
  that is in the IDD NIMAGE feed, you will need to include a LINE=
  specification, and in that specification you will need to set the
  'labeling' value to a negative value.  If you don't the contour lines
  will "zing" off the screen in various places.

  Here is the explanation that has been provided by a community GEMPAK
  expert on why this is needed:

  "The problem is when a contour line breaks to allow a label to be plotted.
   I can see what is happening, but when I fix the contour lines, the map is
   drawn incorrectly. However, there is a way to avoid the problem. In the
   LINE parameter, there is a setting that allows a line to get a label with
   breaking the line. 

   LINE = color / line type / line width / labeling

   If the labeling value is a positive number N, then every Nth contour gets
   labeled. The default is 1 so every contour gets a label. If the value is 0,
   then no labels are drawn. If the value is a negative number, -N, then every
   Nth contour gets labeled, but there is no line break. The user could use this
   feature to avoid the spurious lines until I can work on this some more. This
   is all done by the user at runtime, so no code needs to be changed, right 
now."

re:
> Thanks for any specific or high level guidance.

I think that you may want to configure the LDM on vulcan so that it
looks like/works like LDMs you are (or were) familiar with.  If you
decide that this is what you want to do, I can give you a step-by-step
set of instructions for what to do.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: PHE-874806
Department: Support GEMPAK
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.