Re: [awips2-users] EDEX Not Ingesting GOES-17 Meso from NOAAPORT

On Wed, Dec 30, 2020 at 12:23 PM Tiffany Meyer <address@hidden> wrote:

Hi Ryan,

I was able to replicate the issue of not displaying, but was able to get around it by closing CAVE, remove ~/caveData, start up CAVE again and then try loading.

It's probably an issue with what's cached, but by removing caveData it clears out the cache.

Let me know if that resolves the issue. Thanks,
~Tiffany
On Wed, Dec 30, 2020 at 12:28 PM Ryan Hickman <address@hidden> wrote:
Tiffany,

It actually appears that while the TIU* files are being ingested by the GOES-R ingester, they're not able to be displayed in AWIPS.

INFO 2020-12-30 18:24:39,600 1263 [Ingest.GOESR-2] Ingest: EDEX: Ingest - goes-r:: /awips2/data_store/goesr/noaaport/TIUR07_KNES_301823_PAA_925331.gini processed in: 0.0460 (sec) Latency: 0.2260 (sec)
INFO 2020-12-30 18:24:40,583 1274 [Ingest.GOESR-2] Ingest: EDEX: Ingest - goes-r:: /awips2/data_store/goesr/noaaport/TIUR08_KNES_301823_PAA_925339.gini processed in: 0.0470 (sec) Latency: 0.2080 (sec)
INFO 2020-12-30 18:24:40,674 1279 [Ingest.GOESR-2] Ingest: EDEX: Ingest - goes-r:: /awips2/data_store/goesr/noaaport/TIUR11_KNES_301823_PAA_925341.gini processed in: 0.0420 (sec) Latency: 0.2830 (sec)

[awips@edex-new satellite]$ ls
AKREGI ECONUS EFD EMESO-1 EMESO-2 HIREGI PRREGI WCONUS WFD WMESO-1 WMESO-2
[awips@edex-new satellite]$ ls -al WMESO-1/CH-01-0.47um/
total 57372
drwxr-xr-x 2 awips fxalpha 4096 Dec 30 18:25 .
drwxr-xr-x 18 awips fxalpha 4096 Dec 30 17:48 ..
-rw-r--r-- 1 awips fxalpha 20016296 Dec 30 18:00 satellite-2020-12-30-17.h5
-rw-r--r-- 1 awips fxalpha 38714523 Dec 30 18:25 satellite-2020-12-30-18.h5

Is there potentially another change I might need to make?
On Wed, Dec 30, 2020 at 11:21 AM Tiffany Meyer <address@hidden> wrote:
Hi Ryan,

Glad you got this issue resolved. There are two ways to get/ingest the GOES 16/17 ABI data.

1. Use the NOAAPort feed which get stored as the .gini files (The top option below)
2. Use the NIMAGE data Unidata is sending out which are the netcdf files stitched together, with the WMO header stripped so the data can also be read into other software (the bottom option below)
#
# GOES 16/17 ABI
# NOAAPort
NOTHER  ^(TI....) (KNES) (......) (...)
        FILE    -close -edex
        /awips2/data_store/GOES/GINI/\1_\2_\3_\4_(seq).gini
#
# GOES 16/17 ABI
# Unidata-stitched
NIMAGE  ^/data/ldm/pub/native/satellite/GOES/([^/]*)/Products/CloudAndMoistureImagery/([^/]*)/([^/]*)/([0-9]{8})/([^/]*).nc
        FILE    -close -edex
        /awips2/data_store/GOES/CMI/\5.nc4
In-house we are ingesting the NIMAGE netcdf files, but either way works.

Thanks,
~Tiffany

On Wed, Dec 30, 2020 at 11:49 AM Ryan Hickman <address@hidden> wrote:
I noticed my goof. You need to edit `distribution/goesr.xml` not satellite.xml.

"<regex>^TI[SR]</regex>" becomes "<regex>^TI[SRU]</regex>" and all is well.

INFO 2020-12-30 17:49:16,588 2820 [Ingest.GOESR-1] Ingest: EDEX: Ingest - goes-r:: /awips2/data_store/goesr/noaaport/TIUR13_KNES_301748_PAA_921505.gini processed in: 0.0600 (sec) Latency: 0.0830 (sec)

Consider this resolved.

On Wed, Dec 30, 2020 at 10:17 AM Ryan Hickman <address@hidden> wrote:
I am unable to get EDEX to ingest the GOES-17 Mesoscale sectors from NOAAPORT. This started with a report of that data missing and I was able to find entries in the "unrecognized files" log.

WARN 2020-12-30 17:01:41,160 6124 [external.dropbox-2] RouteFailedLog: TRDA01 KWBC 301600
WARN 2020-12-30 17:01:41,161 6125 [external.dropbox-5] RouteFailedLog: THBA01 KWBC 301500
WARN 2020-12-30 17:01:41,162 6126 [external.dropbox-4] RouteFailedLog: TRDA01 KWBC 301700
WARN 2020-12-30 17:01:41,166 6127 [external.dropbox-5] RouteFailedLog: TRAA01 KWBC 301600
WARN 2020-12-30 17:01:41,167 6128 [external.dropbox-4] RouteFailedLog: TRBA01 KWBC 301500
WARN 2020-12-30 17:01:41,171 6129 [external.dropbox-5] RouteFailedLog: TRDA01 KWBC 301600
WARN 2020-12-30 17:01:41,171 6130 [external.dropbox-4] RouteFailedLog: THDA01 KWBC 301600
WARN 2020-12-30 17:01:41,174 6131 [external.dropbox-2] RouteFailedLog: THDA01 KWBC 301600
WARN 2020-12-30 17:01:42,308 6132 [external.dropbox-5] RouteFailedLog: TRBA01 KWBC 301500
WARN 2020-12-30 17:01:42,314 6133 [external.dropbox-5] RouteFailedLog: THDA01 KWBC 301600

The pqact.conf entry is the default one that was included with EDEX.
#
# GOES 16/17 ABI
#
NOTHER ^(TI....) (KNES) (......) (...)
FILE -close -edex
/awips2/data_store/goesr/noaaport/\1_\2_\3_\4_(seq).gini

Since the `distribution/satellite.gini.xml` includes "<regex>.*.gini.*</regex>" you'd figure it would get ingested. Regardless, I added "<regex>TIU... KNES</regex>" and did an `edex restart ingest` which got the data flowing to the right ingester.

INFO 2020-12-30 17:09:30,981 9392 [Ingest.Gini-1] Ingest: EDEX: Ingest - satellite:: /awips2/data_store/goesr/noaaport/TIUR06_KNES_301708_PAA_917280.gini processed in: 0.0040 (sec) Latency: 0.0240 (sec)
INFO 2020-12-30 17:09:33,202 9397 [Ingest.Gini-1] Ingest: EDEX: Ingest - satellite:: /awips2/data_store/goesr/noaaport/TIUR08_KNES_301708_PAA_917282.gini processed in: 0.0030 (sec) Latency: 0.1670 (sec)
INFO 2020-12-30 17:09:33,211 9400 [Ingest.Gini-1] Ingest: EDEX: Ingest - satellite:: /awips2/data_store/goesr/noaaport/TIUR04_KNES_301708_PAA_917283.gini processed in: 0.0040 (sec) Latency: 0.1760 (sec)

However the data is still not displayable in AWIPS CAVE ("No Data Available") and the log above seems to indicate it's being processed as a generic "Gini-1" instead of "GOESR-2" like the rest of the GOES-17 products.

Has anyone else had similar issues with the NOAAPORT GOES-17 Mesoscale data? Is there another configuration file I might need to change to get it to recognize these files as GOES-17 HDFs?

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.

awips2-users mailing list
address@hidden
For list information, to unsubscribe, or change your membership options, visit: https://www.unidata.ucar.edu/mailing_lists/

--

Tiffany Meyer
AWIPS Lead Software Engineer IV
UCAR - Unidata
--

Tiffany Meyer
AWIPS Lead Software Engineer IV
UCAR - Unidata