LDM: rtstats(1) limitation on number of reporting "bins"

Warren & Gini,

The rtstats(1) utility in version 6.0 of the LDM does, indeed, have a
limitation on the number of "bins" on which it can report.  That limit
is 96.

That limit was increased to 4000 in version 6.1.0 of the LDM (which
was released on 2004-08-25).  Subsequent versions also have the higher
limit.  (We bumped the limit up because encountered the same monitoring
problem here).

What do you want to do?  Options include the following:

    Replacing the LDM-s with a later version.

    Replacing just the rtstats(1) utility with a later version.

Either of the above would have to be done for every LDM system that
collected NEXRAD2 data from more than 96 upstream LDM-s.

All versions of the LDM are "backward" compatible: any LDM will always
work with an older version.

Regards,
Steve Emmerson
LDM Developer

--- Begin Message ---

• Subject: Re: rtstats
• Date: Wed, 15 Jun 2005 13:03:28 -0500

Steve,
Our RadarII developers are running into limitation with rtstats

and believe that rtstats can only keep track of 100 sources of data per feedtype (CRAFT) and that it drops sources that have not reported over an extended period of time.

We need to keep track of at least 300 sources and need to be able to configure tracking of sources that may have extended periods of outage.

How can we do this?
Thanks
gini galvin
NWS/OCIO/TOC/Telecommunications Software Branch
Radar II  Administrator
240 393 3348 c
301 713 0882 x 176 work

David S Alden wrote:

Hi Warren,
    Gini sent me a message Monday indicating she would check with her
supervisor about resolving the 'rtstats' source tracking problems but
I have not heard about any progress she might have made up to this
point.  I suspect she is still working the problem in the absence of
an update on her progress.
    As you may know, the only information I have on the functionality
of the LDM 'rtstats' utility is the 'man' page provided with the LDM
distribution.  This documentation does not go into detail related to
the inner workings and/or ideosyncracies it may have so we are learning
by experience how it works.  There appears to be at present two problems
that affect its use in our Level II Radar Monitor project.  The first
is an apparent limitation it has for keeping track of only about 100
sources of data ingesting into any particular feedtype (ours is CRAFT)
and the second is its apparent inability to report on sources that have
been out of service for extended periods of time.  The man page does
not indicate any limitations with regard to either of these
considerations or any others for that matter.  Upon reflection,
however, one can easily see how in the general case, 'rtstats' would
drop a source for which nothing had been received for some internally
predetermined period of time because there are many applications of
LDM for which the sources of data are not as stable as the AWIPS
radars (stable in terms of number not necessarily operational
consistency).  I would propose that 'rtstats' provide a configurable
source retention option that prevents it from dropping sources before
they are restored to service.  If, in fact, it has an internal
limitation of 100 sources it can keep track of, that this also be
configurable if it is not already.  Of course, it is possible to code
around such limitations but, in the absence of specific documentation,
we could not determine the need for this beforehand.  Any help you can
get from Steve Emerson would be appreciated.

>Date: Wed, 15 Jun 2005 07:38:51 -0400
>From: "Warren.Blanchard" <address@hidden>
>Organization: DOC/NOAA/NWS - National Weather Service
>To: David S Alden <address@hidden>
>CC: Allan Darling <address@hidden>,
>   Jami Casamento <address@hidden>,
>   "address@hidden" <address@hidden>
>Subject: Level II radar monitor
>
>Dave,
>I was following your e-mails relative to the rtstats limitations. If not
>already resolved, we could consult with Steve Emerson (Unidata
>LDM developer) on if and how to correct that problem.
>I also noticed, when checking the tgweb Level II page that on Wed.
>morning only 130 sites were being monitored. I believe the number
>of sites should always be 133 unless more are added to the system.
>It looks as though three sites (KHDX, KOKX and KVNX) were
>not included. I guessed this was because each of these had been
>down for an extended period of time, 11 to 18 hours. I would think
>all 133 should be included even if some are down for long periods.
>Attached is the page at that time, and the TOC e-mail notice of
>outages/latencies.
>Hope this helps...
>Warren
>
>Subject: NWS Region WSR-88D Level II Data Outage/Data Latency Update
>Date: Wed, 15 Jun 2005 10:18:05 GMT
>From: address@hidden
>To: address@hidden
>CC: address@hidden
>
>NWS Region WSR-88D Level II Data Outage/Data Latency Update FOR WEDNESDAY
>15JUN05 AT 6AM EST LOCAL:
>
>KHDX = 11 HRS 53 MINS
>KIND =  9 HRS 22 MINS
>KOKX = 17 HRS 30 MINS
>KMOB =  1 HRS 28 MINS
>KVNX = 18 HRS 23 MINS

        
Warren B

--- End Message ---

--- Begin Message ---

• Subject: ldm rtstats usage
• Date: Wed, 15 Jun 2005 13:49:33 -0400

Hi Steve,
In the concluding phases of our NEXRAD Level II project, the
NWS Telecommunications Operations Center is tasked with developing
a level II monitoring and statistics capability. It is expected to be similar
to that developed by Carl Sinclair at OU for IRADS which you are
probably familiar with. It is based entirely on the use of rtstats for the
statistics gathering and monitoring.
While not directly involved in the development, I am still monitoring the
activities as the project manager, and from experience I know you are the
"answer man"!
The monitor tool is trying to track all 133 sites currently sending via LDM.
All 133 being received into a single LDM system as "CRAFT" feed type.
"rtstats" does not seem to be able to keep track of more than 99-100
sources. In alphabetical order listing, see attached, after the first 99 (knqa)
only 1 product is recorded even though 200+ are actually received. Is this
a known limitation, or perhaps a configurable parameter?
I know that IRADS only receives from one region (28 to 40 sites each) into
a single system so for them a single "rtstats" instance would probably never
experience this problem.
Thanks for the help.
Warren


kabr 1118417377 20050610152935 20050610152935 389 2803416 0 0.51 1@1824
kabx 1118414510 20050610144102 20050610144102 320 2871377 0 0.28 1@1105
kakq 1118417377 20050610152949 20050610152935 231 2626657 -14 0.00 0@0000
kama 1118417377 20050610152933 20050610152933 474 5636931 0 0.37 2@2835
kamx 1118417377 20050610152933 20050610152934 446 16960636 1 1.05 3@1203
kapx 1118417377 20050610152934 20050610152935 227 2613826 1 0.84 3@0842
karx 1118417377 20050610152935 20050610152936 477 6475905 1 1.03 4@0029
katx 1118417377 20050610152934 20050610152935 229 2606013 1 0.36 1@1633
kbbx 1118417377 20050610152926 20050610152927 88 1133976 1 0.86 2@2405
kbgm 1118417377 20050610152928 20050610152928 229 2697553 0 0.38 2@2040
kbhx 1118417377 20050610152936 20050610152936 230 1787468 0 0.29 2@2010
kbis 1118417377 20050610152933 20050610152934 473 3803288 1 0.38 1@2917
kblx 1118417377 20050610152933 20050610152934 457 5620270 1 0.42 2@2421
kbmx 1118417377 20050610152844 20050610152933 367 4572327 49 48.44 49@0254
kbox 1118417377 20050610152933 20050610152933 231 1963780 0 0.23 2@1007
kbro 1118417377 20050610152930 20050610152930 230 2530469 0 0.28 1@0841
kbuf 1118417377 20050610152934 20050610152935 461 5893924 1 0.40 2@0155
kbyx 1118417377 20050610152929 20050610152930 454 16008197 1 0.75 3@2128
kcae 1118417377 20050610152935 20050610152936 473 6283307 1 0.33 2@1250
kcbw 1118417377 20050610152934 20050610152934 229 1998392 0 0.18 1@2456
kcbx 1118417377 20050610152930 20050610152930 380 4422051 0 0.49 1@1809
kccx 1118417377 20050610152933 20050610152934 473 4487559 1 0.36 3@1751
kcle 1118417377 20050610152933 20050610152934 374 4343552 1 0.32 2@1329
kclx 1118417377 20050610152932 20050610152932 389 5654239 0 0.43 2@1233
kcrp 1118417377 20050610152920 20050610152921 88 1388793 1 0.51 1@1400
kcxx 1118417377 20050610152932 20050610152932 230 2654422 0 0.58 3@2559
kcys 1118417377 20050610152931 20050610152932 389 4952747 1 2.98 87@0419
kdax 1118417377 20050610152913 20050610152913 87 1455933 0 0.57 2@2400
kddc 1118417377 20050610152931 20050610152933 478 9550394 2 1.16 4@0117
kdfx 1118417377 20050610152933 20050610152933 227 2227353 0 0.55 1@1458
kdgx 1118417377 20050610152934 20050610152935 229 2856748 1 0.36 1@1520
kdix 1118417377 20050610152929 20050610152929 229 1904124 0 0.18 1@0652
kdlh 1118417377 20050610152931 20050610152932 228 4650768 1 1.14 6@0137
kdmx 1118417377 20050610152927 20050610152929 228 3283007 2 1.07 3@2327
kdtx 1118417377 20050610152934 20050610152934 89 1100773 0 0.83 2@0732
kdvn 1118417377 20050610152918 20050610152936 227 2624433 18 17.77 19@0326
kdyx 1118417377 20050610152930 20050610152931 226 3060375 1 0.83 2@2354
keax 1118417377 20050610152933 20050610152934 374 5372968 1 0.73 3@0802
kemx 1118417377 20050610152928 20050610152928 88 815425 0 0.37 1@2330
kenx 1118417377 20050610152929 20050610152929 225 2425583 0 0.32 3@0009
kepz 1118417377 20050610152922 20050610152922 88 950718 0 0.38 1@2347
kesx 1118417377 20050610152930 20050610152930 88 926206 0 0.59 1@2831
kevx 1118417377 20050610152912 20050610152935 468 11197697 23 23.22 25@1600
kewx 1118417377 20050610152919 20050610152919 88 1479603 0 0.34 2@0437
kfcx 1118417377 20050610152935 20050610152936 387 5122102 1 0.25 1@1243
kfdr 1118417377 20050610152933 20050610152933 229 2451717 0 0.67 2@0733
kfdx 1118417377 20050610152931 20050610152932 230 2406846 1 0.74 2@0131
kffc 1118417377 20050610152935 20050610152936 229 3340904 1 0.38 2@0009
kfsd 1118417377 20050610152941 20050610152932 227 2353728 -9 0.00 0@0000
kfsx 1118417377 20050610152918 20050610152918 87 573740 0 0.31 2@0036
kftg 1118417377 20050610152932 20050610152933 228 2662373 1 0.67 2@1025
kfws 1118415669 20050610150009 20050610150014 2 11793 5 5.04 5@0009
kggw 1118417377 20050610152934 20050610152934 230 2746126 0 0.58 3@2134
kgjx 1118417377 20050610152933 20050610152933 230 3052960 0 1.18 3@2645
kgld 1118417377 20050610152932 20050610152932 461 6640813 0 1.08 5@0734
kgrb 1118417377 20050610152931 20050610152934 229 2544193 3 0.77 3@0614
kgrr 1118417377 20050610152931 20050610152931 231 2576132 0 0.90 3@0749
kgsp 1118417377 20050610152932 20050610152933 470 5718963 1 0.29 1@2224
kgwx 1118416279 20050610150957 20050610151021 1 901 24 23.42 23@0957
kgyx 1118417377 20050610152935 20050610152935 195 2162841 0 0.24 2@2632
khgx 1118417377 20050610152920 20050610152921 83 1148925 1 1.33 2@1358
khnx 1118417377 20050610152918 20050610152919 87 1208289 1 0.52 2@0212
khtx 1118417377 20050610152934 20050610152934 216 2442471 0 0.27 1@1953
kict 1118417377 20050610152935 20050610152936 462 10026884 1 1.49 4@1829
kicx 1118417377 20050610152930 20050610152930 88 337242 0 0.12 1@0852
kiln 1118417377 20050610152934 20050610152934 456 8497689 0 0.30 2@2551
kilx 1118417377 20050610152932 20050610152933 459 8135813 1 1.16 4@1243
kind 1118417377 20050610152929 20050610152930 227 3346858 1 0.87 3@0818
kinx 1118417377 20050610152932 20050610152933 229 3175009 1 0.43 2@0254
kiwa 1118417377 20050610152916 20050610152917 87 1255836 1 0.46 4@1806
kiwx 1118417377 20050610152932 20050610152933 387 3911318 1 0.74 6@2512
kjax 1118417377 20050610152934 20050610152935 440 10370513 1 0.45 1@2717
kjgx 1118417377 20050610152930 20050610152931 226 3313248 1 0.87 2@1621
kjkl 1118417377 20050610152934 20050610152935 112 1458740 1 1.22 4@2844
klbb 1118417377 20050610152926 20050610152934 457 6606562 8 8.39 10@0310
klch 1118417377 20050610152935 20050610152935 230 2839120 0 0.34 1@0135
klix 1118417377 20050610152935 20050610152935 229 3678377 0 0.49 2@1235
klnx 1118417377 20050610152931 20050610152931 230 1831297 0 0.52 2@0821
klot 1118417377 20050610152933 20050610152934 388 4618363 1 0.69 3@2914
klrx 1118417377 20050610152934 20050610152934 229 1916352 0 0.35 1@1103
klsx 1118417377 20050610152933 20050610152935 388 3788121 2 0.66 3@0035
kltx 1118417377 20050610152933 20050610152934 229 2084027 1 0.21 1@2624
klvx 1118417377 20050610152935 20050610152936 230 2973743 1 0.77 2@0147
klwx 1118417377 20050610152936 20050610152936 225 2381201 0 0.15 1@0008
klzk 1118417377 20050610152928 20050610152929 229 2783496 1 0.34 1@2128
kmaf 1118417377 20050610152933 20050610152934 475 11138048 1 0.62 2@1037
kmax 1118417377 20050610152919 20050610152920 88 601003 1 0.48 2@2358
kmhx 1118417377 20050610152928 20050610152928 226 1976298 0 0.17 1@2534
kmkx 1118417377 20050610152935 20050610152936 472 6244855 1 1.01 5@0049
kmlb 1118417377 20050610152934 20050610152935 386 8837246 1 0.49 2@2232
kmob 1118417377 20050610152935 20050610152936 477 8133587 1 0.40 1@0649
kmpx 1118412070 20050610140106 20050610140108 9 197078 2 1.39 2@0009
kmqt 1118417377 20050610152934 20050610152934 473 5049214 0 1.08 4@0527
kmrx 1118417377 20050610152930 20050610152931 229 2512985 1 0.32 1@1917
kmsx 1118408105 20050610125406 20050610125407 419 3674311 1 0.49 2@2848
kmtx 1118417377 20050610152932 20050610152932 227 1311913 0 0.18 1@2358
kmux 1118417377 20050610152932 20050610152932 89 714288 0 0.34 1@1408
kmvx 1118417377 20050610152941 20050610152934 231 2665748 -7 0.00 0@0000
knkx 1118417377 20050610152920 20050610152920 88 708091 0 0.26 0@1423
knqa 1118417133 20050610152531 20050610152532 1 7466 1 0.34 0@2531
koax 1118417316 20050610152834 20050610152835 1 9610 1 0.94 1@2834
kohx 1118413290 20050610142128 20050610142129 1 36344 1 0.96 1@2128
kotx 1118416950 20050610152228 20050610152228 1 5446 0 0.34 0@2228
kpah 1118416889 20050610152127 20050610152128 1 19837 1 0.76 1@2127
kpbz 1118417316 20050610152833 20050610152833 1 25363 0 0.69 1@2833
kpdt 1118415547 20050610145904 20050610145905 1 28839 1 0.94 1@5904
kpux 1118417377 20050610152935 20050610152937 1 12167 2 1.32 1@2935
krax 1118417316 20050610152835 20050610152835 1 11230 0 0.44 0@2835
krgx 1118417316 20050610152834 20050610152834 1 4527 0 0.04 0@2834
kriw 1118416889 20050610152127 20050610152127 1 5862 0 0.24 0@2127
krlx 1118417011 20050610152329 20050610152330 1 29410 1 0.82 1@2329
krtx 1118417133 20050610152530 20050610152531 1 18973 1 0.52 1@2530
ksfx 1118417255 20050610152734 20050610152734 1 17145 0 0.42 0@2734
ksgf 1118417377 20050610152936 20050610152936 1 9634 0 0.48 0@2936
kshv 1118417255 20050610152734 20050610152734 1 7217 0 0.15 0@2734
ksjt 1118417255 20050610152732 20050610152732 1 12402 0 0.33 0@2732
ksox 1118387121 20050610070519 20050610070520 1 20795 1 0.90 1@0519
ksrx 1118416828 20050610152027 20050610152027 1 4831 0 0.12 0@2027
ktbw 1118417255 20050610152732 20050610152732 1 35774 0 0.64 1@2732
ktfx 1118417133 20050610152532 20050610152532 1 5165 0 0.20 0@2532
ktlh 1118416950 20050610152154 20050610152229 1 11064 35 34.92 35@2154
ktlx 1118417377 20050610152935 20050610152935 1 4382 0 0.14 0@2935
ktwx 1118416584 20050610151621 20050610151622 1 15910 1 1.21 1@1621
ktyx 1118417316 20050610152832 20050610152833 1 20397 1 1.12 1@2832
kudx 1118415608 20050610150003 20050610150003 1 9820 0 0.63 1@0003
kuex 1118417255 20050610152731 20050610152733 1 15399 2 2.50 3@2731
kvnx 1118417377 20050610152936 20050610152936 1 10107 0 0.47 0@2936
kvtx 1118416950 20050610152228 20050610152229 1 4931 1 0.13 0@2228
kvwx 1118417011 20050610152329 20050610152330 1 5746 1 0.38 0@2329
kyux 1118415120 20050610145158 20050610145159 1 22388 1 0.57 1@5158

begin:vcard 
n:Blanchard;Warren
tel;fax:301.713.9395
tel;work:301.713.1570 x130
x-mozilla-html:FALSE
adr:;;;;;;
version:2.1
email;internet:address@hidden
fn:Warren Blanchard
end:vcard

--- End Message ---