[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
20001218: LDM: Meaning of nfree in pqmon and if values <10 indicate a need to change something
- Subject: 20001218: LDM: Meaning of nfree in pqmon and if values <10 indicate a need to change something
- Date: Mon, 18 Dec 2000 10:45:23 -0700
>From: Gregory Grosshans <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200012181729.eBIHTno07428
>Can you tell me what nfree and maxfree are and if the LDM configuration
>or queue size should be changed when nfree is less than 10 most of the
>time? I've searched for pqmon on the web based email archive and didn't
>see any discussion in regards to this issue.
Gregg,
NFREE is the number of free space regions. It is good for this number to
be as small as possible since that means that the available space is in
contiguous areas. MAXFREE is the greatest value that nfree ever reaches.
When the LDM has to delete products in order to make room for a new
product, it creates regions of free space. Deletion occurs until
there is a region that is large enough to hold the incoming data.
Fewer NFREE regions means that searching for available regions is
faster. This searching is order LOG(NFREE), so MAXFREE is something
to watch, but can be pretty large before slow down would occur.
I have heavilly loaded systems with MAXFREE ~5000. Our fast systems
usually only show MAXFREE ~100. If you had a heavily IO loaded system,
then you might see this number get big.
>
>In particular, a system, ldmalice, is ingesting NOAAPORT NWSTG and
>GOES-EAST channels, as well as a UNISYS radar feed containing single
>site radar data and national mosaics. The queue size is 750 MB and the
>version of LDM is 5.1.2. The output of pqmon over the past 12 hours
>results in nfree usually hovering around 1-3 and one time it was as high
>as 21. Maxfree was 26 and increased to 28 during the last few hours.
These numbers are very good.
>
>On another system, ldmpooh, LDM 5.1.2 system with a 750 MB queue,
>ingesting NOAAPORT NWSTG and the UNISYS radar data the pqmon output over
>the past 12 hours results in nfree in the 10-30s, with a high of 130 and
>a value of 1 being logged less than 10 times. Maxfree started out at 91
>and increased to 135.
Probably not much to wory about. NFREE generally will grow if a lot
of data comes in very fast and a lot of deletion must occur. The algorithm
used allows the data to decrement the free regions as it will fit. When
given the debug flag, pqmon will show the size of each of the free regions.
If you are trying to insert a 26MB GOESE 1km visible, and deletion is removing
lots of 1K nwstg products, the number of regions will grow some, but Russ
Rew has spent considerable time ensuring that this is much faster
than the previous pqexpire which ran every 5 minutes to try to
keep up with the amount of free space needed.
>
>Below I've included several lines of output from pqmon from both
>systems.
>
>Do the low values of nfree on ldmalice indicate something should be
>changed in the LDM configuration?
>
>Thanks,
>Gregg
>
>output from pqmon on ldmalice: Time is UTC.
>Dec 18 16:00:17 pqmon: 41261 2 141842 715164048 71473
>28 111631 25365360 6534
>Dec 18 16:01:17 pqmon: 41611 2 141492 721272800 71473
>28 111631 25365360 6594
>Dec 18 16:02:17 pqmon: 41911 1 141193 726136640 71473
>28 111631 23865536 6654
>Dec 18 16:03:17 pqmon: 42155 1 140949 728213880 71473
>28 111631 21788296 6714
>Dec 18 16:04:17 pqmon: 42404 1 140700 729933704 71473
>28 111631 20068472 6774
>Dec 18 16:05:17 pqmon: 42628 1 140476 732010896 71473
>28 111631 17991280 6834
>Dec 18 16:06:17 pqmon: 42986 1 140118 734549720 71473
>28 111631 15452456 6894
>Dec 18 16:07:17 pqmon: 43378 1 139726 737208824 71473
>28 111631 12793352 6954
>Dec 18 16:08:17 pqmon: 43688 1 139416 739417344 71473
>28 111631 10584832 7014
>Dec 18 16:09:17 pqmon: 44127 1 138977 742004192 71473
>28 111631 7997984 7074
>Dec 18 16:10:17 pqmon: 44581 1 138523 748933552 71473
>28 111631 1068624 7135
>Dec 18 16:11:17 pqmon: 43252 2 139851 749978064 71473
>28 111631 23888 7064
>Dec 18 16:12:17 pqmon: 43429 1 139675 749966144 71473
>28 111631 36032 7101
>Dec 18 16:13:17 pqmon: 43643 1 139461 749995512 71473
>28 111631 6664 7135
>Dec 18 16:14:17 pqmon: 43581 1 139523 749995768 71473
>28 111631 6408 7165
>Dec 18 16:15:17 pqmon: 43593 1 139511 749967592 71473
>28 111631 34584 7183
>Dec 18 16:16:17 pqmon: 43744 1 139360 749996048 71473
>28 111631 6128 7185
>Dec 18 16:17:17 pqmon: 43660 2 139443 749998024 71473
>28 111631 3920 7182
>Dec 18 16:18:18 pqmon: 43918 1 139186 749988240 71473
>28 111631 13936 7219
>Dec 18 16:19:18 pqmon: 44087 1 139017 749999720 71473
>28 111631 2456 7255
>Dec 18 16:20:18 pqmon: 44376 2 138727 749989048 71473
>28 111631 12832 7287
>Dec 18 16:21:18 pqmon: 44558 1 138546 749998672 71473
>28 111631 3504 7322
>Dec 18 16:22:18 pqmon: 44765 1 138339 749974176 71473
>28 111631 28000 7373
>Dec 18 16:23:18 pqmon: 44952 2 138151 749979328 71473
>28 111631 21976 7423
>Dec 18 16:24:18 pqmon: 43581 11 139513 724076144 71473
>28 111631 5612992 7242
>Dec 18 16:25:18 pqmon: 43979 8 139118 731879848 71473
>28 111631 5612992 7302
>Dec 18 16:26:18 pqmon: 44301 8 138796 735153616 71473
>28 111631 5612992 7362
>Dec 18 16:27:18 pqmon: 44714 6 138385 738839000 71473
>28 111631 5612992 7422
>Dec 18 16:28:18 pqmon: 45244 2 137859 741934136 71473
>28 111631 5612992 7482
>Dec 18 16:29:18 pqmon: 45728 1 137376 745185200 71473
>28 111631 4816976 7542
>Dec 18 16:30:18 pqmon: 46171 1 136933 747740328 71473
>28 111631 2261848 7602
>Dec 18 16:31:18 pqmon: 46307 7 136791 748690120 71473
>28 111631 496304 7626
>Dec 18 16:32:18 pqmon: 46364 2 136739 749794232 71473
>28 111631 207440 7648
>Dec 18 16:33:18 pqmon: 46248 1 136856 749995440 71473
>28 111631 6736 7650
>Dec 18 16:34:18 pqmon: 46184 1 136920 749986936 71473
>28 111631 15240 7677
>Dec 18 16:35:18 pqmon: 46063 1 137041 749999400 71473
>28 111631 2776 7698
>Dec 18 16:36:18 pqmon: 45760 2 137343 749999328 71473
>28 111631 1976 7673
>Dec 18 16:37:18 pqmon: 45409 1 137695 750001904 71473
>28 111631 272 7652
>Dec 18 16:38:18 pqmon: 45491 2 137612 749665640 71473
>28 111631 336352 7656
>Dec 18 16:39:18 pqmon: 45822 1 137282 749615240 71473
>28 111631 386936 7689
>Dec 18 16:40:18 pqmon: 43478 16 139611 702894136 71473
>28 111631 11237896 7223
>Dec 18 16:41:18 pqmon: 43831 7 139267 714306664 71473
>28 111631 11237896 7283
>Dec 18 16:42:18 pqmon: 44270 4 138831 719661056 71473
>28 111631 11237896 7343
>Dec 18 16:43:19 pqmon: 44704 3 138398 723602152 71473
>28 111631 11237896 7403
>Dec 18 16:44:19 pqmon: 45159 3 137943 727980744 71473
>28 111631 11237896 7463
>Dec 18 16:45:19 pqmon: 45573 2 137530 731232536 71473
>28 111631 11237896 7523
>Dec 18 16:46:19 pqmon: 46100 2 137003 734536184 71473
>28 111631 11237896 7583
>Dec 18 16:47:19 pqmon: 46734 1 136370 739679328 71473
>28 111631 10322848 7643
>Dec 18 16:48:19 pqmon: 47296 1 135808 744846840 71473
>28 111631 5155336 7703
>Dec 18 16:49:19 pqmon: 47736 3 135366 749072368 71473
>28 111631 683632 7756
>Dec 18 16:50:19 pqmon: 47864 2 135239 749872784 71473
>28 111631 128792 7789
>Dec 18 16:51:19 pqmon: 47930 1 135174 746948680 71473
>28 111631 3053496 7826
>Dec 18 16:52:19 pqmon: 48124 1 134980 748769176 71473
>28 111631 1233000 7886
>Dec 18 16:53:19 pqmon: 48365 1 134739 749994984 71473
>28 111631 7192 7940
>Dec 18 16:54:19 pqmon: 44524 2 138579 740785528 71473
>28 111631 8453360 7547
>Dec 18 16:55:19 pqmon: 44812 1 138292 747609672 71473
>28 111631 2392504 7607
>Dec 18 16:56:19 pqmon: 45234 1 137870 749986808 71473
>28 111631 15368 7665
>Dec 18 16:57:19 pqmon: 45504 1 137600 749991440 71473
>28 111631 10736 7699
>Dec 18 16:58:19 pqmon: 45618 1 137486 749997992 71473
>28 111631 4184 7733
>Dec 18 16:59:19 pqmon: 45787 2 137316 750000048 71473
>28 111631 1232 7768
>Dec 18 17:00:19 pqmon: 45918 1 137186 749954936 71473
>28 111631 47240 7809
>Dec 18 17:01:19 pqmon: 46036 1 137068 749944928 71473
>28 111631 57248 7809
>
>
>output from pqmon on ldmpooh. Time is in UTC.
>Dec 18 16:00:52 pqmon: 73157 33 109915 748922760 89724
>135 93374 361944 12019
>Dec 18 16:01:52 pqmon: 73323 14 109768 749979104 89724
>135 93374 18928 12040
>Dec 18 16:02:52 pqmon: 73460 13 109632 749952944 89724
>135 93374 30144 12083
>Dec 18 16:03:52 pqmon: 73457 29 109619 749979008 89724
>135 93374 12544 12102
>Dec 18 16:04:52 pqmon: 73285 39 109781 749978000 89724
>135 93374 2984 12115
>Dec 18 16:05:52 pqmon: 73104 37 109964 749983816 89724
>135 93374 6008 12096
>Dec 18 16:06:52 pqmon: 73095 35 109975 749995936 89724
>135 93374 568 12066
>Dec 18 16:07:52 pqmon: 73136 35 109934 749994368 89724
>135 93374 1312 12064
>Dec 18 16:08:52 pqmon: 73240 42 109823 749969832 89724
>135 93374 5232 12043
>Dec 18 16:09:52 pqmon: 73224 62 109819 749959032 89724
>135 93374 11520 12019
>Dec 18 16:10:52 pqmon: 73202 63 109840 749766608 89724
>135 93374 209224 12004
>Dec 18 16:11:52 pqmon: 73312 67 109726 749969464 89724
>135 93374 7824 11988
>Dec 18 16:12:52 pqmon: 73345 74 109686 749956808 89724
>135 93374 16920 11963
>Dec 18 16:13:52 pqmon: 73404 69 109632 749972016 89724
>135 93374 1768 11958
>Dec 18 16:14:52 pqmon: 73182 73 109850 749960600 89724
>135 93374 5840 11914
>Dec 18 16:15:52 pqmon: 73029 80 109996 749952632 89724
>135 93374 5040 11868
>Dec 18 16:16:52 pqmon: 73013 74 110018 749969040 89724
>135 93374 2808 11848
>Dec 18 16:17:52 pqmon: 73040 43 110022 749975776 89724
>135 93374 15992 11850
>Dec 18 16:18:52 pqmon: 73092 12 110001 749979936 89724
>135 93374 15056 11831
>Dec 18 16:19:52 pqmon: 72944 12 110149 749585776 89724
>135 93374 338416 11818
>Dec 18 16:20:52 pqmon: 72989 20 110096 749963376 89724
>135 93374 12984 11826
>Dec 18 16:21:52 pqmon: 73022 22 110061 749987824 89724
>135 93374 2896 11826
>Dec 18 16:22:52 pqmon: 73111 22 109972 749963192 89724
>135 93374 27400 11846
>Dec 18 16:23:52 pqmon: 73011 23 110071 749974632 89724
>135 93374 8152 11865
>Dec 18 16:24:52 pqmon: 72732 25 110348 749966384 89724
>135 93374 20872 11849
>Dec 18 16:25:52 pqmon: 72686 29 110390 749941216 89724
>135 93374 13776 11815
>Dec 18 16:26:52 pqmon: 72675 16 110414 749997512 89724
>135 93374 2360 11798
>Dec 18 16:27:52 pqmon: 72682 15 110408 749983296 89724
>135 93374 7504 11796
>Dec 18 16:28:52 pqmon: 72493 17 110595 749974512 89724
>135 93374 14528 11788
>Dec 18 16:29:52 pqmon: 72239 42 110824 749945920 89724
>135 93374 6504 11748
>Dec 18 16:30:52 pqmon: 72145 31 110929 749806232 89724
>135 93374 99944 11762
>Dec 18 16:31:52 pqmon: 72055 14 111036 749989688 89724
>135 93374 9272 11751
>Dec 18 16:32:52 pqmon: 71929 16 111160 749989216 89724
>135 93374 8816 11711
>Dec 18 16:33:52 pqmon: 71862 26 111217 749918408 89724
>135 93374 53832 11681
>Dec 18 16:34:52 pqmon: 71742 22 111341 749978728 89724
>135 93374 16488 11675
>Dec 18 16:35:52 pqmon: 71653 30 111422 749983712 89724
>135 93374 6200 11650
>Dec 18 16:36:52 pqmon: 71652 45 111408 749957928 89724
>135 93374 7968 11616
>Dec 18 16:37:52 pqmon: 71803 21 111281 749877600 89724
>135 93374 121352 11626
>Dec 18 16:38:52 pqmon: 71496 25 111584 749135720 89724
>135 93374 363560 11563
>Dec 18 16:39:52 pqmon: 71494 32 111579 749974168 89724
>135 93374 11552 11576
>Dec 18 16:40:52 pqmon: 71522 26 111557 749988464 89724
>135 93374 5744 11558
>Dec 18 16:41:52 pqmon: 71673 10 111422 749934912 89724
>135 93374 55792 11537
>Dec 18 16:42:52 pqmon: 71720 28 111357 749917136 89724
>135 93374 43328 11528
>Dec 18 16:43:52 pqmon: 71690 18 111397 749968048 89724
>135 93374 13368 11521
>Dec 18 16:44:52 pqmon: 71578 15 111512 749965464 89724
>135 93374 16360 11504
>Dec 18 16:45:52 pqmon: 71463 15 111627 749983112 89724
>135 93374 6952 11505
>Dec 18 16:46:52 pqmon: 71374 15 111716 749990552 89724
>135 93374 6792 11487
>Dec 18 16:47:52 pqmon: 71352 28 111725 749977256 89724
>135 93374 5808 11496
>Dec 18 16:48:52 pqmon: 71422 20 111663 749981448 89724
>135 93374 9048 11478
>Dec 18 16:49:52 pqmon: 71244 30 111831 749925456 89724
>135 93374 12472 11462
>Dec 18 16:50:52 pqmon: 71185 31 111889 749948744 89724
>135 93374 17920 11489
>Dec 18 16:51:52 pqmon: 71205 29 111871 749961536 89724
>135 93374 27320 11494
>Dec 18 16:52:52 pqmon: 71303 15 111787 749980696 89724
>135 93374 9848 11502
>Dec 18 16:53:52 pqmon: 71069 18 112018 749988240 89724
>135 93374 2144 11484
>Dec 18 16:54:52 pqmon: 70970 27 112108 749952664 89724
>135 93374 8792 11495
>Dec 18 16:55:52 pqmon: 71148 16 111941 749957600 89724
>135 93374 15584 11478
>Dec 18 16:56:52 pqmon: 71166 14 111925 749991120 89724
>135 93374 2424 11462
>Dec 18 16:57:52 pqmon: 71212 15 111878 749979016 89724
>135 93374 12256 11447
>Dec 18 16:58:52 pqmon: 71349 2 111754 749989744 89724
>135 93374 12240 11452
>Dec 18 16:59:52 pqmon: 71278 31 111796 749937672 89724
>135 93374 19576 11441
>Dec 18 17:00:52 pqmon: 71254 26 111825 749969384 89724
>135 93374 7688 11439
>Dec 18 17:01:52 pqmon: 71346 17 111742 749975720 89724
>135 93374 13976 11432
>
>
I have several queues running at 2GB (and we have tested 64 bit versions
with extended queues). The new LDM design ensures that the searching for
space does not rapidly become too exhaustive with large queues. Your 750MB
queue seems to be functioning very well. You can use the AGE field to
tune the size of your queue. If you only want 1-2 hours worth of data in the
queue, then you can increase or decrease the queue size until AGE shows that
the oldest product is staying within your expected range.
Steve Chiswell