[LDM #PHE-874806]: LDM and filing of GOES-17 NIMAGE data

Hi Mike,

re:
> Thanks for the quality feedback.  Let me summarize by saying I was able to
> file the data with a variety of "pathing", so that is no longer the
> immediate issue.  I decided to just file directly for now, as the
> LPProdFile.sh would take a fair amount of editing to work right.

I don't think that L2ProdFile.sh needs to be modified at all.

re:
> So I made up this pqact entry: (don't laugh!)
> 
> NIMAGE
> ^/data/ldm/pub/native/satellite/GOES/(GOES1.)/P......./(Cloud).*/(CONUS|FullDisk|Mesoscale-1|Mesoscale-2|PuertoRico|Alaska|Hawaii)/(Channel..).*(OR_...-L2..............................).*(.nc)
> FILE    -close  -overwrite images/sat/\1/\2/\3/\4/\5\6

Just as an FYI:

The extended regular expression you are using could be simplified.  The 
following
action should be equivalent:

NIMAGE (GOES1.)/Products/(Cloud).*/(.*)/(Channel..).*(OR_.{36}).*(.nc)
   FILE -close -overwrite images/sat/\1/\2/\3/\4/\5\6

I'm not sure which is easier to understand, so take it for what it is worth.

Aside comment: the reason that your expression and my simplified one should
work OK is that there are no other L2 products in the NIMAGE feed that
have both "Cloud" and "Channel" in their Product IDs.

There are several other L2 products that have "Cloud" in their Product IDs;
here is the full list:

CloudTopTemperature
CloudTopPressure
CloudTopPhase
CloudTopHeight
CloudParticleSize
CloudOpticalDepth
CloudMask
CloudAndMoistureImagery

re:
> Which creates this for example:
> [ldm@vulcan Channel14]$ pwd
> /usr/local/ldm/var/data/images/sat/GOES17/Cloud/CONUS/Channel14
> 
> [ldm@vulcan Channel14]$ ll
> total 17124
> -rw-r--r-- 1 ldm metrole 3506496 Jul 24 00:37
> OR_ABI-L2-CMIPC-M6C14_G17_s201920500311.nc
> -rw-r--r-- 1 ldm metrole 3505381 Jul 24 00:39
> OR_ABI-L2-CMIPC-M6C14_G17_s201920500361.nc
> -rw-r--r-- 1 ldm metrole 3504877 Jul 24 00:44
> OR_ABI-L2-CMIPC-M6C14_G17_s201920500411.nc
> 
> And the path length on this is 106 as it happens.

Looks good!

Aside: some sites choose to file their products on a file system
other than the one that the LDM is installed on.  The "classic"
way of doing this was to make a link in the LDM HOME directory
to a "data" directory somewhere else.  This, in turn, could
result in a shorter fully qualified pathname for the files.

For instance:

<as 'ldm'>
cd ~ldm/var
rmdir data
ln -s /data/ldm data

The files from your example would then live in:

/data/ldm/images/sat/GOES17/CONUS/Channel14, and fully qualified pathnames
would look like:

/data/ldm/images/sat/GOES17/CONUS/Channel14/OR_ABI-L2-CMIPC-M6C14_G17_s201920500311.nc

This particular pathname is 86 characters long.

re:
> So, I try to plot with NMAP and I still get nothing after hitting accept.

I would post a question to the gembud email list.  Why?  Pete Pokrandt of
UW/AOS was the first one to alert us to the need to FILE the NIMAGE
images so that fully qualified pathnames would be less than or equal to
108 characters.  Also, we have had feedback from another GEMPAK user who
is creating displays from the images being distributed in the revamped
NIMAGE feed, so we know that GEMPAK display of ABI L2 images is working
for others.

re:
> I did edit the GEMTBL/sat/imgtyp.tbl file as indicated, although that is a
> separate issue.

Correct, that was/is a separate issue.

re:
> Any ideas why I may still be having an issue?

No, sorry, I am not a GEMPAK expert.  I don't know what may be wrong
other than to say that it might somehow be related to a GEMPAK table
setting.

re:
> Since my goal is to automate map production, I suppose I could try gpmap
> directly, which is what I would need to use eventually to script/automate,
> true?

Yes, you would need to use something like gpmap to automate things.  It is
my novice opinion, however, that if the images work when using gpmap, they
should work in NMAP.  This assumes, of course, that the code being used is
the latest distribution in which Michael added support for the GOES-R/S
imagery that is being distributed through NOAAPort.

Aside:

Why do I keep referring to NOAAPort?  GEMPAK does _not_ work with the GOES-R/S
images that are distributed in the GRB and which we distribute in the SATELLITE
feed.  The files in the revamped NIMAGE feed and in SATELLITE are both in
netCDF4 files, but their internal structures are completely different.  Michael
only added support for the images that derive from NOAAPort (as tiles and then
we stitch together the tiles to make full scenes and then send the result in
the revamped NIMAGE feed).

re:
> Perhaps I should head down the python route for plotting the imagery as
> there seem to be a lot of tools/techniques in the works.  From a 30K foot
> view, would that be best in the long run?

That is completely up to you, of course.  One comment that I can make
is that plotting the images using Python takes a LOT more time than
it does in GEMPAK or McIDAS.  This comment is based on feedback from
folks at the College of DuPage who were using Python to create image
displays of GOES-R/S products and then switched (back) to using McIDAS
after I made an ADDE server for the images that derive from NOAAPort.

re:
> Thanks again for the great help and suggestions, it is very much
> appreciated.

No worries.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: PHE-874806
Department: Support GEMPAK
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.