Thanks, Pete. With the latest version of the LDM, it's screaming for me to fix it. :-)
Gilbert On Oct 23, 2020, at 12:52 AM, Pete Pokrandt <address@hidden> wrote:
Yes, it is definitely an upstream issue. The root conduit.ncep.noaa.gov server is not running ntp or any other time sync protocol, so the clock is drifting. Check the conduit lags for unidata.ucar.edu servers, they are the same. Tom knows about this, and has
emailed the NOAA folks multiple times. They told him a fix would be coming in October.. we'll see. We've known about the clock drift for several months now.
Pete
-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - address@hidden
From: Gilbert Sebenste <address@hidden>
Sent: Friday, October 23, 2020 12:26 AM
To: Pete Pokrandt <address@hidden>
Subject: Re: Delays in CONDUIT feed
We're still getting warnings here by the tons that your feed is still 10 minutes behind at 12:20 AM. So, I did a notifyme. Your feed is lagging about 10 minutes, but we are getting the data pretty much immediately when you do pass it on. So...upstream
issue?
The reason why we are noticing such things now is LDM 6.13.12. You should be seeing similar entries in your ldmd.log file.
Gilbert
On Oct 22, 2020, at 10:31 PM, Pete Pokrandt <address@hidden> wrote:
Fingers crossed - it looks like they may have resolved the issue. We'll see if the 00 UTC runs stay at an acceptible lag. Looked like the NAM was ok. GFS starting up now.
Pete
-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - address@hidden
From: Gilbert Sebenste <address@hidden>
Sent: Thursday, October 22, 2020 3:35 PM
To: Pete Pokrandt <address@hidden>
Cc: Ryan Hickman <address@hidden>
Subject: Re: Delays in CONDUIT feed
OK, thanks, Pete. That's when our logs started showing the issues as well, on Tuesday. Best on your network engineers getting that diagnosed and repaired ASAP!
Gilbert
On Oct 22, 2020, at 2:19 PM, Pete Pokrandt <address@hidden> wrote:
Something weird is going on. Starting with the 12 UTC run on Tuesday, our connection to conduit.ncep.noaa.gov changed somehow, and our latencies went way up.
Also just heard from a tech in SSEC who said they also noticed network issues starting Tuesday. Apparently, our campus network folks are investigating..
I switched my primary CONDUIT feed from conduit.unidata.ucar.edu to idd.unidata.ucar.edu yesterday, and that got my latencies down, but apparently the outbound stuff is still affected.
At any rate, thanks for letting me know.
Pete
-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - address@hidden
From: Gilbert Sebenste <address@hidden>
Sent: Thursday, October 22, 2020 2:06 PM
To: Pete Pokrandt <address@hidden>
Cc: Ryan Hickman <address@hidden>
Subject: Delays in CONDUIT feed
Hi Pete,
Over the last 4 days, our ldmd.log has been ranging from 2 GB to 5 GB. And, almost all of it are "product too old" warning messages from your CONDUIT feed. Here's a sample:
20201022T075547.607942Z idd.aos.wisc.edu[19429] down6.c:vetProduct:226 WARN Ignoring too-old product: 29705 20201022074631.942530 CONDUIT 062 data/nccf/com/rap/prod/rap.20201022/rap.t07z.awp252pgrbf10.grib2
!grib$
After looking in the logs, it looks like there is a 5 to 10 minute delay of your CONDUIT feed. It also means that our backup site is getting it before you are. And, these
20201022T185728.914660Z idd.aos.wisc.edu[19429] down6.c:vetProduct:226 WARN Ignoring too-old product: 19302 20201022184813.924306 CONDUIT 055 data/nccf/com/rap/prod/rap.20201022/rap.t18z.awp252pgrbf19.grib2
!grib2/ncep/RUC2/#000/202010221800F019/OMEG/325 hPa PRES! 000055
20201022T185728.915451Z idd.aos.wisc.edu[19429] down6.c:vetProduct:226 WARN Ignoring too-old product: 19720 20201022184813.954789 CONDUIT 075 data/nccf/com/rap/prod/rap.20201022/rap.t18z.awp252pgrbf19.grib2
!grib2/ncep/RUC2/#000/202010221800F019/OMEG/425 hPa PRES! 000075
20201022T185728.916499Z idd.aos.wisc.edu[19429] down6.c:vetProduct:226 WARN Ignoring too-old product: 36727 20201022184813.981765 CONDUIT 095 data/nccf/com/rap/prod/rap.20201022/rap.t18z.awp252pgrbf19.grib2
!grib2/ncep/RUC2/#000/202010221800F019/RELH/525 hPa PRES! 000095
20201022T185728.917871Z idd.aos.wisc.edu[19429] down6.c:vetProduct:226
Every single model appears to be delayed 5 to 15 minutes. Now, this may be because you are getting the feed that late, there are network issues at WISC or AllisonHouse, or it could be 2020 telling us to go die in a fire (I wouldn't be surprised. ;-) ).
I don't know. It does mean, however, that our backup site is getting it considerably faster than you, and is rejecting your products as being too old (already in the queue). This is not happening to any other of your feeds.
I just wanted to give you and Ryan a heads up!
Gilbert
--
----
Gilbert Sebenste
Consulting Meteorologist
AllisonHouse, LLC
|