[McIDAS #NWD-275345]: v2017 and L2 Derived Products

Hi Mike,

re:
> I'm beginning to get my hands dirty with the L2 Derived products coming
> over NOAAPort, and wanted to know how much support there is for them in
> McIDAS at this point.

All I have done is verify that a NOAAPort-delivered L2 product named
in a way that the core McIDAS-X ABIN servers expect cab be served by
core McIDAS-X.

re:
> I know it's come up between us once or twice, but
> it's been a while and wasn't sure if anything has changed.

Nothing has changed.  I let the SSEC MUG folks know about being able to
serve the NOAAPort L2 products using the ABIN servers, and they have promised
to make available enhancements for those products as soon as they are created
from examples in AWIPS.

re:
> Below is what I've come across so far.
> 
> Some good news is I can at least display images.  Per your suggestion in a
> previous email, I add the datasets as the ABIN format, and name the netCDF
> files using the internal 'dataset_name' variable.  After that, IMGDISP
> reads and displays that data.

Right.  I guessed that the ABIN servers would work on the NOAAPort
L2 products after reviewing the structure of the L2 netCDF-4 files.
I ran simple tests by hand that convinced me that I was on the right
track, and then I wrote a Bourne shell script that is run out of
cron once-per-minute that creates hard links to the L2 files that
Ryan's ldm-alchemy Python script creates.  The script I wrote also
extracts the images' coverages and uses those in the creation of
a meaningful storage hierarchy on disk.  This hierarchy can be seen
in the TDS listing on two of our motherlode class data serving
machines, lead.unidata.ucar.edu and atm.ucar.edu (the TDS is _almost_
up on atm.ucar.edu):

thredds-test.unidata.ucar.edu (lead.unidata.ucar.edu)
http://thredds-test.unidata.ucar.edu/thredds/catalog/catalog.html

  Test Datasets
  http://thredds-test.unidata.ucar.edu/thredds/catalog/testDatasets.html

    GOES-16 NOAAPORT Products -> GOES-16 Products
    
http://thredds-test.unidata.ucar.edu/thredds/catalog/satellite/goes16/catalog.html

      GOES16
      
http://thredds-test.unidata.ucar.edu/thredds/catalog/satellite/goes16/GOES16/catalog.html

        Products
        
http://thredds-test.unidata.ucar.edu/thredds/catalog/satellite/goes16/GOES16/Products/catalog.html

The top nodes of the hierarchy created by Ryan's ldm-alchemy script are
directories with names like A99, B99, C99, etc.  The corresponding hierarchies
I create look like: AerosolDetection, AerosolOpticalDepth, etc.

re:
> Once displayed, I can also use IMGPLOT to retrieve values (examples
> below).  What makes me happy about this is I can see there is at least some
> support for knowing the actual units of the data and values (e.g. CAPE J/KG
> of 525.5).

The SSEC ABIN servers were written to support the L2 products that they
had access to from the GRB simulator program, so this is not surprising.

re:
> Here's that IMGPROBE output example:
> Image Name           Day      Nominal Time   Scan Time    Band
> ----------------      -------    ------------   ---------    ----
> NPGOESR/DSICN.288    7 Mar 18066   16:02:21      16:04:22       9
> 
> File     Nominal  Image     RAW         CAPE
> BRIT
> Lat/Lon      Line/Element  Line/Element                  J/KG
> 15:13:29/ 96:50:52    295/  138    5901/ 2761           6887      525.50 83

OK.

re:
> Unfortunately, whenever I add a color BAR, the values are always in
> brightness 0-255.  I suppose this is to be expected as I am not supplying
> an SU table.  But I'm not sure I can make my own SU tables for these
> products without knowing the correlation between brightness and
> expected/actual data values.  Do you happen to have any insight here?

BAR needs to be enhanced to support the L2 products.

re:
> While some products can be plotted straight away (e.g. TPW), there are
> others that contain sub-products to plot instead (e.g. stability indices).

Yup.

re:
> I can plot those by passing BAND=xx to IMGDISP, though I do need to do a
> little legwork to determine which band I'm looking for.  IMGLIST can tell
> me what bands are available, and either through trial & error, or by
> referencing the netCDF file directly, I can seem to get what I need.

I did a lot of the legwork for you by:

- defining ADDE datasets for the NOAAPort ABI images and L2 products

- adding those datasets to the ADDEIMAGE.CORE file in the Unidata
  McIDAS-X v2017x distributions

If you use the MCGUI delivered with Unidata McIDAS, then you can see
which products have multiple "sub products" (different bands) pretty
easily.  As soon as you find a dataset that has multiple bands, you
can use IMGLIST to list what is in those bands.  For instance:

DATALOC ADD NPGOESR atm.ucar.edu

- either look through ADDEIMAGE.CORE or use the MCGUI to see that the
  following datasets contain multiple bands:

RTGOESR/ARSL<coverage> -- RTGOESR/ARSLCN, RTGOESR/ARSLFD, RTGOESR/ARSLMS
RTGOESR/FSL<coverages> -- NPGOESR/FHSCN, NPGOESR/FHSFD, NPGOESR/FHSMS
NPGOESR/INDX<coverages>-- NPGOESR/INDXCN, NPGOESR/INDXFD, NPGOESR/INDXMS


IMGLIST RTGOESR/ARSLCN FORM=ALL
etc.

re:
> My question here is, will these bands always remain consistent?

I expect them to, but I can't guarantee that SSEC won't change the
band numbers at some point in the future (identifying a particular
product as a particular band is done in the ABIN server source
code).

re:
> In other
> words, will DSI BAND=9 always result in CAPE?  I'm not sure how these are
> determined under the hood.

I expect that they will stay the same, but I can't guarantee it.  They
are determined/set in the ABIN server source code.

re:
> One last thing, when using IMGPROBE I've occasionally gotten a "Frame /
> Image mismatch" error.  It's rare, it's tough to recreate and I'm not sure
> what conditions result in that, but I'm also not sure what that error
> actually means.  Just thought it might be note-worthy.

This can happen when the dataset composition changes after an image has
been loaded using IMGDISP.  By dataset composition, I mean the set of
images that are in the dataset.

re:
> Sorry if I rambled on a bit; hope all is going well for you on your end,

No worries.  As far as how things are going, I am still trying to trace down
what to do to the MERCator navigation code to support the mesoscale images
that you had let me know about while still supporting the Puerto Rico
image sectors.  I expect that I will have to go back to my derivation of
the navigation transforms used in the SCMI server code (scmiutil.c) to
see where the resolution ratio (requested resolution / source resolution)
needs to be incorporated.  I have set aside time to do this today...

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: NWD-275345
Department: Support McIDAS
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.