Re: Latency/Bandwidth issues from idd.unidata ?

Hi Pete,

Your `nex2gini` process would be sensitive to latencies as timestamped files fall outside the RADDUR setting for your composite. Do you have it something tight like 15 minutes? A look on disk `ls -l` of your raw nexrad3 products may show you such things as you compare VCP times and the local file write times.

daryl

________________________________________
From: Pete Pokrandt <address@hidden>
Sent: Monday, March 7, 2022 12:20 PM
To: Herzmann, Daryl E [AGRON]; Unidata IDD Support
Subject: Re: Latency/Bandwidth issues from idd.unidata ?

I was about to send this to ldm-users but I wonder if it's related to this.

I had been feeding NEXRAD3 data from idd.ssec.wisc.edu and am seeing rouge NEXRAD3 products coming from that same rouge source (wxengine4.fox.com)

I just changed to idd.unidata.ucar.edu and I am still seeing NEXRAD3 products with a source of wxengine4.fox.com.

I wonder if these products may be the source of the flashing/dropouts on my radar composites like this

https://tempest.aos.wisc.edu/radar/ne3comphtml5.html

Or if maybe there is just some other weirdness causing that, that Daryl described below..

Here is a little of a recent rtstats report from idd-agg on the NEXRAD3 feed, showing data from

20220307160059 20220307160059             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu            1         1803 0.069468       0.07    0@0059              6.13.16
20220307160354 20220307160354             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           11       290983 0.303538       0.50    1@0249              6.13.16
20220307160522 20220307160522             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           13       304981 0.055078       0.43    1@0249              6.13.16
20220307160925 20220307160925             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           34       927356 0.051925       0.70    1@0849              6.13.16
20220307161154 20220307161155             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           38       989290 0.744785       0.70    1@0849              6.13.16
20220307161223 20220307161223             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           50      1969558 0.874931       0.71    1@0849              6.13.16
20220307161456 20220307161456             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           51      1977395 0.053628       0.70    1@0849              6.13.16
20220307161546 20220307161546             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           52      2001967 0.201653       0.69    1@0849              6.13.16
20220307161746 20220307161752             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           53      2013890 6.141386       0.79    6@1746              6.13.16
20220307162157 20220307162158             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           62      2603056 0.639316       0.74    6@1746              6.13.16
20220307162328 20220307162328             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           63      2606698 0.065504       0.73    6@1746              6.13.16
20220307162435 20220307162436             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           65      2747416 0.519428       0.72    6@1746              6.13.16
20220307162531 20220307162532             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           69      2866788 0.475435       0.70    6@1746              6.13.16
20220307162858 20220307162859             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           70      2868086 0.056212       0.69    6@1746              6.13.16
20220307163122 20220307163123             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           79      3239226 1.249094       0.75    6@1746              6.13.16
20220307163655 20220307163655             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           81      3255629 0.128856       0.73    6@1746              6.13.16
20220307163834 20220307163834             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           82      3257000 0.049984       0.72    6@1746              6.13.16
20220307164207 20220307164207             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           92      3345866 0.1997       0.66    6@1746              6.13.16
20220307164311 20220307164311             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          120      4131507 0.075646       0.72    6@1746              6.13.16
20220307164417 20220307164417             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          121      4131772 0.179297       0.71    6@1746              6.13.16
20220307164650 20220307164650             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          122      4134247 0.066279       0.71    6@1746              6.13.16
20220307164748 20220307164748             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          129      4434613 0.56368       0.70    6@1746              6.13.16
20220307164842 20220307164842             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          130      4440431 0.438608       0.70    6@1746              6.13.16
20220307164932 20220307164932             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          131      4454224 0.057495       0.69    6@1746              6.13.16
20220307165111 20220307165112             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          147      4986022 0.585622       0.66    6@1746              6.13.16
20220307165346 20220307165347             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          178      5634921 0.582222       0.64    6@1746              6.13.16
20220307165500 20220307165500             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          179      5647071 0.056245       0.64    6@1746              6.13.16
20220307165708 20220307165708             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          187      5874550 0.089858       0.62    6@1746              6.13.16
20220307165720 20220307165720             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          189      5893299 0.354626       0.62    6@1746              6.13.16
20220307165925 20220307165925             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          191      5909191 0.099638       0.61    6@1746              6.13.16
20220307170202 20220307170202             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu            7        63944 0.196477       0.17    0@0202              6.13.16
20220307170315 20220307170316             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu            9       168845 0.108202       0.15    0@0202              6.13.16
20220307170455 20220307170455             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           30       705076 0.064073       0.19    0@0417              6.13.16
20220307170627 20220307170627             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu           96      1735479 0.662655       0.39    1@0523              6.13.16
20220307170751 20220307170752             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          123      2424505 0.707923       0.40    1@0523              6.13.16
20220307171119 20220307171120             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          124      2477852 0.2725       0.40    1@0523              6.13.16
20220307171208 20220307171208             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          126      2484022 0.051567       0.40    1@0523              6.13.16
20220307171333 20220307171333             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          127      2497485 0.057292       0.39    1@0523              6.13.16
20220307171527 20220307171527             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          137      2582921 0.053404       0.40    1@0523              6.13.16
20220307171805 20220307171805             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          138      2610779 0.087739       0.40    1@0523              6.13.16
20220307171946 20220307171946             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          139      2613373 0.192138       0.39    1@0523              6.13.16
20220307172209 20220307172209             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          169      3063900 0.449082       0.42    1@2155              6.13.16
20220307172329 20220307172329             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          170      3152340 0.232085       0.42    1@2155              6.13.16
20220307172730 20220307172730             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          171      3152867 0.05771       0.42    1@2155              6.13.16
20220307173146 20220307173147             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          194      4154218 0.417551       0.42    1@2155              6.13.16
20220307173419 20220307173419             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          195      4167324 0.061655       0.42    1@2155              6.13.16
20220307173530 20220307173531             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          220      4848342 0.317733       0.41    1@2155              6.13.16
20220307173622 20220307173622             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          221      4886782 0.112724       0.41    1@2155              6.13.16
20220307174350 20220307174350             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          249      5812629 0.551216       0.44    2@4233              6.13.16
20220307174438 20220307174438             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          306      6671605 0.072321       0.42    2@4233              6.13.16
20220307175225 20220307175226             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          324      6914743 0.369451       0.42    2@4233              6.13.16
20220307175335 20220307175335             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          325      6924609 0.170021       0.42    2@4233              6.13.16
20220307175526 20220307175526             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          355      7618467 0.053879       0.40    2@4233              6.13.16
20220307175626 20220307175626             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          356      7624472 0.061621       0.40    2@4233              6.13.16
20220307175711 20220307175712             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          357      7646834 0.589739       0.40    2@4233              6.13.16
20220307175841 20220307175842             idd-agg.aos.wisc.edu NEXRAD3 wxengine4.fox.com_v_idd.ssec.wisc.edu          358      7733677 0.413886       0.40    2@4233              6.13.16

<http://www.weather.com/tv/shows/wx-geeks/video/the-incredible-shrinking-cold-pool>-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - address@hidden

________________________________
From: Herzmann, Daryl E [AGRON] <address@hidden>
Sent: Monday, March 7, 2022 12:10 PM
To: Unidata IDD Support <address@hidden>
Cc: Pete Pokrandt <address@hidden>
Subject: Latency/Bandwidth issues from idd.unidata ?

Hi Unidata,

Ufff, I've been having quite the time recently and don't wish to send you off on goose chases, but over the past number of days, maybe all the way since the NEXRAD3 super-res addition, I have been seeing puzzling latency from various Unidata hosts to me here at ISU over the IDD. As an example raw bandwidth test:

$ wget https://motherlode.ucar.edu/decoded/gempak/model/gfs/2022030700_gfs215.gem

2.36MB/s first test.
25.6MB/s second test a few minutes later.
52.8MB/s max on third test, but bouncing around a bunch

I am trying this from different data centers at ISU to see about isolating against local congestion.

So I am reviewing rtstats metrics and for example, for me:

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+idd-dc.agron.iastate.edu

You can see a lot of "chop" between 0 and 300 seconds. I looked at Pete's and observe similar chop.

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?IDS|DDPLUS+idd1.aos.wisc.edu

with these strange "blips" up to 50 seconds for me. Looking at other feeds, there's been some odd things too:

https://rtstats.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?NEXRAD2+idd1.aos.wisc.edu

It feels like there are gremlins in the system. Over the past few week, I have multiple times found LDM 6.13.16 seemingly stuck, not moving much of any data, whilst upstream `notifyme` shows data flowing, but even sometimes not showing data flowing.

Golly, this email is horrible for you folks to offer support for. I guess an overall question.

Are any known troubles (bandwidth, latency) afoot with {idd,iddb}.unidata.ucar.edu afoot? I keep switching between idd, iddb, and idd.aos(Pete, cc'd), attempting to find stable flows. I sometimes find the need to re-create the queue to make things happy.

My local monitoring on my LDM queue age does show dipping below 3600 seconds these days, so I will effort getting that expanded, but figured I would ping you folks to see if you know what is going on?

thanks
daryl