[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[ldm-users] INC0316836 has been Resolved ( GOES Imagery)

     INC0316836 has been Resolved ( GOES Imagery)

Some details of the ticket are included below:

End User:Jeff Kelley
Short description: GOES Imagery
Customer Impact Statement:missing tiles in GOES imagery
State:Resolved
Priority:High
Category:General
Sub-Category:Other
Assigned to:Chun Lam
Assignment group:NCO OMB Tech Control
Description:Received email notification from Jeff Kelley: Hi NCO. Have you been in contact with NESDIS at all about the missing tiles in GOES-16? Every image that comes in has missing tiles, whether in AWIPS, the FAA weather system, or even the College of Dupage website.

https://weather.cod.edu/satrad/?parms=continental-conus-13-24-1-100-1&checked=map&colorbar=undefined

I haven't seen any ADA or ADM messages either.

Jeff
ZID CWSU
Close notes:[NCO-MCL] Guest reported the following:

Additional comments•03-26-2022 23:43:34just now
reply from: address@hidden

This issue has been solved on our end. It was corrected several weeks later.
My bad for not mailing back and requesting that the ticket be closed. My apolgies, and I deem this issue to be solved.

Gilbert Sebenste
AllisonHouse, LLC

The following comments were given about the Incident:

2022-03-26 19:44:51 EDT - Chun LamAdditional comments
Closed Code: Closed/Resolved by Caller
Closed Notes: [NCO-MCL] Guest reported the following:

Additional comments•03-26-2022 23:43:34just now
reply from: address@hidden

This issue has been solved on our end. It was corrected several weeks later.
My bad for not mailing back and requesting that the ticket be closed. My apolgies, and I deem this issue to be solved.

Gilbert Sebenste
AllisonHouse, LLC
2022-03-26 19:43:34 EDT - Guest Additional comments
reply from: address@hidden

This issue has been solved on our end. It was corrected several weeks later.
My bad for not mailing back and requesting that the ticket be closed. My apolgies, and I deem this issue to be solved.

Gilbert Sebenste
AllisonHouse, LLC
2022-03-26 19:34:27 EDT - Chun LamAdditional comments
[NCO-MCL] NCO is requesting via email an update from Jeff Kelley to see if the issue still exists.
2021-12-16 10:52:44 EST - Jeff KelleyAdditional comments
reply from: address@hidden

Seeing the same thing at the CWSU in Indianapolis, in AWIPS and the FAA
weather system.

Jeff
2021-12-16 10:40:49 EST - Guest Additional comments
reply from: address@hidden

NCO,

You can see this issue by running "notifyme -vl- -h localhost -f NOTHER -p TIRE02". That will look for all GOES-East CONUS Band 02 tiles, they'll have an identifier at the end of PAx where x will be an increasing letter representing each tile. There should be 15 tiles for that product, and quite a few aren't being seen.
2021-12-16 10:22:47 EST - Guest Additional comments
reply from: address@hidden

This issue has started again as of 14:30Z this morning.
2021-12-13 20:14:24 EST - Guest Additional comments
reply from: address@hidden

Someone from the College of DuPage reached out to me with an interesting observation from the University of Oklahoma...

We had an outage on our NOAAport receive system this morning, which caused our systems to fail over to our partner organization's feeds. On investigation, we found that our receiver, ingest software, and LDM software were all working fine. We saw packets coming into the Linux-based ingest server, but nothing was being ingested. Then, it was noticed that the source IP of the NOAAport multicast traffic had changed, for at least some of the packets. Whereas in the past, it had been 10.0.9.51 , we are now seeing packets from 10.0.9.52 . Our ingest server was not accepting those because it had no special routing rule for that IP. We added such a rule (and did *not* delete the existing one for the .51 IP) and our NOAAport ingest began working as before.

I can confirm the source IP for me is 10.0.9.52 as well, but unlike him that IP range was already open on my system and that traffic was making its way in. I'm not sure if this is related or not, but I wanted to pass along the intel in case it is.
2021-12-13 19:54:36 EST - Jeff KelleyAdditional comments
reply from: address@hidden

I just got off the phone with NCF. They are aware that data dropouts are
occurring (with all data types, not just satellite) after switching between
systems earlier today. The tech mentioned that it might be a firewall
issue.

Jeff
2021-12-13 19:53:01 EST - Shellene CopeAdditional comments
Gilbert Sebenste emailed: 1) I have been using the GOES Restitch for a couple of months now and it has worked perfectly until today. Today I am getting partial images. Upon looking at the log files I am finding that it is "Timing Out" waiting for the tile to arrive. I am not seeing any large amounts of dropped packets. I am not sure what could be causing my sudden issue. Any insight as to what I might check?



Thanks, Jim ( partial log below)





[TIRE 132051] 2021-12-13 20:54:52,378 [process_items]: OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:54:53,177 [process_items]: OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:54:54,033 [process_items]: OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:00,091 [process_items]: OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc: 11 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:00,753 [process_items]: OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:01,554 [process_items]: OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc: 12 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:06,066 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 6 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:06,350 [process_items]: OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc: 13 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:07,089 [process_items]: OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:07,592 [process_items]: OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 10 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:08,514 [process_items]: OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 10 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:13,149 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:16,471 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:22,188 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9 of 15 tiles received.

[TIRE 132051] 2021-12-13 20:55:34,664 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 10 of 15 tiles received.

[TIRE 132046] 2021-12-13 20:55:44,834 [process_items]: Finishing due to timeout.

[TIRE 132046] 2021-12-13 20:55:44,944 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel07/20211213/OR_ABI-L2-CMIPC-M6C07_G16_s20213472046170_e20213472046170_c20213472046170.nc finished.

[TIRE 132046] 2021-12-13 20:55:44,947 [process_items]: Finishing due to timeout.

[TIRE 132046] 2021-12-13 20:55:45,051 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel12/20211213/OR_ABI-L2-CMIPC-M6C12_G16_s20213472046170_e20213472046170_c20213472046170.nc finished.

[TIRE 132051] 2021-12-13 20:55:45,698 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 11 of 15 tiles received.

[TIRE 132046] 2021-12-13 20:55:47,772 [process_items]: Finishing due to timeout.

[TIRE 132046] 2021-12-13 20:55:47,875 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel10/20211213/OR_ABI-L2-CMIPC-M6C10_G16_s20213472046170_e20213472046170_c20213472046170.nc finished.

[TIRE 132046] 2021-12-13 20:55:48,075 [process_items]: Finishing due to timeout.

[TIRE 132046] 2021-12-13 20:55:48,181 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel11/20211213/OR_ABI-L2-CMIPC-M6C11_G16_s20213472046170_e20213472046170_c20213472046170.nc finished.

[TIRE 132046] 2021-12-13 20:55:49,455 [process_items]: Finishing due to timeout.

[TIRE 132046] 2021-12-13 20:55:49,559 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel09/20211213/OR_ABI-L2-CMIPC-M6C09_G16_s20213472046170_e20213472046170_c20213472046170.nc finished.

[TIRE 132046] 2021-12-13 20:55:49,560 [process_items]: Finishing due to timeout.



[TIRE 132046] 2021-12-13 20:55:49,664 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel08/20211213/OR_ABI-L2-CMIPC-M6C08_G16_s20213472046170_e20213472046170_c20213472046170.nc finished.

[TIRE 132051] 2021-12-13 20:55:49,891 [process_items]: OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 12 of 15 tiles received.

[TIRE 132046] 2021-12-13 20:55:51,447 [process_items]: Finishing due to timeout.

[TIRE 132046] 2021-12-13 20:55:51,556 [store_done]: GOES16/Products/CloudAndMoistureImagery/CONUS/Channel13/20211213/OR_ABI-L2-CMIPC-M6C13_G16_s20213472046170_e20213472046170_c20213472046170.nc finished.

2) Hi Jim,

I was just about to write my own email about this. It seems to me that not all GOES-16 tiles are making it to NOAAPort. I was about to ask if anyone else is seeing this, and I guess you are.

I noticed this began around 14Z today. GOES-16 CONUS & Full-Disk are impacted, Mesoscale and GOES-17 seem okay. I haven't seen any announcements about data flow issues. Using notifyme I saw that Unidata was missing the same tiles I was.

Does anyone else have any insight?

-Mike

3) Hello NCO,

Some GOES-16 tiles on NOAAport/SBN are missing. This started just after 14Z,
when you awitched to the BMGS. See the thread below from a UNIDATA user log.
Multiple people are seeing this.

Gilbert Sebenste
Meteorological Consultant
AllisonHouse, LLC

(SEC)
2021-12-13 19:48:34 EST - Shellene CopeAdditional comments
Jeff Kelley has emailed the following reply: There has been no improvement in this issue. Have you heard anything from NESDIS? It's strange that the imagery on the STAR and NASA websites appear fine.

https://weather.msfc.nasa.gov/cgi-bin/basicLooper.pl?category=goeseastabiconus&regex=10p35um&title=GOES-East%20ABI%20CONUS%20-%20%2010.35%20um%20-%20Band%2013&time_drop=show
https://www.star.nesdis.noaa.gov/GOES/conus_band.php?sat=G16&band=13&length=24

AWIPS and the FAA weather system continue to exhibit the same issue as this link: https://weather.cod.edu/satrad/

(SEC)

The following work notes were given about the Incident:

2022-03-26 19:34:27 EDT - Chun LamWork notes
[NCO-MCL] NCO is requesting via email an update from Jeff Kelley to see if the issue still exists.
2021-12-16 11:20:51 EST - Zheng TzengWork notes
reply from: address@hidden

Seeing the same thing at the CWSU in Indianapolis, in AWIPS and the FAA
weather system.

Jeff
2021-12-16 10:37:58 EST - Zheng TzengWork notes
reply from: address@hidden

This issue has started again as of 14:30Z this morning.
2021-12-13 20:39:36 EST - Curtis SimsWork notes
@0114z - OMB received an update from address@hidden
reply from: address@hidden

Someone from the College of DuPage reached out to me with an interesting observation from the University of Oklahoma...

We had an outage on our NOAAport receive system this morning, which caused our systems to fail over to our partner organization's feeds. On investigation, we found that our receiver, ingest software, and LDM software were all working fine. We saw packets coming into the Linux-based ingest server, but nothing was being ingested. Then, it was noticed that the source IP of the NOAAport multicast traffic had changed, for at least some of the packets. Whereas in the past, it had been 10.0.9.51 , we are now seeing packets from 10.0.9.52 . Our ingest server was not accepting those because it had no special routing rule for that IP. We added such a rule (and did *not* delete the existing one for the .51 IP) and our NOAAport ingest began working as before.

I can confirm the source IP for me is 10.0.9.52 as well, but unlike him that IP range was already open on my system and that traffic was making its way in. I'm not sure if this is related or not, but I wanted to pass along the intel in case it is.
(CS)
2021-12-13 20:10:02 EST - Curtis SimsWork notes
@0054z - OMB received an update from Jeff Kelley
reply from: address@hidden

I just got off the phone with NCF. They are aware that data dropouts are
occurring (with all data types, not just satellite) after switching between
systems earlier today. The tech mentioned that it might be a firewall
issue.

Jeff
(CS)
 
Ref:MSG6116092
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


ldm-users mailing list
address@hidden
For list information or to unsubscribe,  visit: 
https://www.unidata.ucar.edu/mailing_lists/