* 2.6.0-test8/test9 io scheduler needs tuning?
@ 2003-10-27 21:52 Michael Frank
2003-10-27 22:55 ` cliff white
0 siblings, 1 reply; 19+ messages in thread
From: Michael Frank @ 2003-10-27 21:52 UTC (permalink / raw)
To: linux-kernel
To my surprise 2.6 - which used to do better then 2.4 - does no longer
handle these test that well.
Generally, IDE IO throughput is _very_ uneven and IO _stops_ at times with the
system cpu load very high (and the disk LED off).
IMHO the CPU scheduling is OK but the IO scheduling acts up here.
The test system is a 2.4GHz P4 with 512M RAM and a 55MB/s udma IDE harddisk.
The tests load the system to loadavg > 30. IO should be about 20MB/s on avg.
Enclosed are vmstat -1 logs for 2.6-test9-Vanilla, followed by 2.6-test8-Vanilla
(-mm1 behaves similar), 2.4.22-Vanilla and 2.4.21+swsusp all compiled wo preempt.
IO on 2.6 stops now for seconds at a time. -test8 is worse than -test9
2.6-test9-Vanilla:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
11 10 2 0 17164 60408 96292 0 0 0 5756 1147 30803 21 79 0
13 7 1 0 22968 60600 83876 0 0 0 30100 1079 6475 19 81 0
16 6 4 0 4728 59900 106028 0 0 4 29556 1122 71394 23 77 0
12 7 3 0 27128 60056 70588 0 0 68 6344 1212 111635 19 81 0
13 8 2 0 45176 60200 51656 0 0 0 22868 1238 20471 13 87 0
12 9 2 0 18544 60324 76668 0 0 0 14720 1267 36875 22 78 0
15 5 1 0 4784 60352 95196 0 0 184 37448 1191 144856 32 68 0
3 15 1 0 14768 59980 91368 0 0 180 12376 1112 133689 41 59 0
4 15 1 0 40816 60112 59320 0 0 164 188 1092 75854 31 69 0
6 14 1 0 4976 60164 102640 0 0 16 56904 1122 85364 27 73 0
11 8 1 0 25648 60188 79316 0 0 4 3784 1100 75808 26 74 0
9 8 3 0 8896 60280 103316 0 0 0 24772 1154 4031 23 77 0
5 14 2 0 23744 60408 75220 0 0 232 108 1157 86855 38 62 0
10 8 1 0 12032 60028 91796 0 0 8 27048 1272 79355 24 76 0
10 10 2 0 21120 60292 80144 0 0 0 49700 1092 3015 10 90 0
13 8 2 0 34064 60488 66528 0 0 0 172 1061 3871 10 90 0
14 7 2 0 40200 60540 79824 0 0 8 36864 1121 3138 12 88 0
20 2 2 0 42304 60544 78452 0 0 4 504 1103 281659 23 77 0
12 8 1 0 59024 60656 61688 0 0 28 26080 1156 158476 22 78 0
14 4 1 0 59536 60924 54128 0 0 0 9132 1251 3056 9 91 0
15 2 3 0 46528 61108 65628 0 0 0 23092 1360 58596 11 89 0
8 10 4 0 4288 61244 103836 0 0 56 25904 1135 33984 21 79 0
14 3 2 0 48640 61340 67512 0 0 56 18156 1252 53401 24 76 0
13 1 2 0 30336 61572 80856 0 0 4 15192 1365 1880 20 80 0
10 7 1 0 13376 61752 93496 0 0 176 2320 1225 21294 30 70 0
12 3 2 0 22272 62012 81292 0 0 112 17808 1238 38432 19 81 0
10 3 2 0 12736 62288 91356 0 0 192 21840 1121 2531 21 79 0
16 2 3 0 26752 62228 67660 0 0 56 32500 1745 2100 21 79 0
15 8 2 0 4208 62292 74004 0 0 20 108 1152 4642 37 63 0
17 7 4 0 25200 62444 68544 0 0 60 19184 1384 2337 20 80 0
22 4 4 0 45224 62540 59164 0 0 64 24852 1309 451 14 86 0
22 4 4 0 27240 62684 77048 0 0 0 7480 1280 323 15 85 0
24 3 3 0 26528 62960 66448 0 0 0 24788 1240 837 9 91 0
23 3 2 0 26656 63208 62952 0 0 4 204 1068 616 8 92 0
24 3 2 0 20704 63500 69628 0 0 40 0 1029 440 7 93 0
24 3 2 0 13400 63796 76536 0 0 4 0 1020 426 8 92 0
25 3 2 0 13400 64032 76340 0 0 0 0 1019 371 10 90 0
26 3 2 0 23688 64048 81792 0 0 124 31828 1098 2308 22 78 0
18 5 2 0 30664 64356 69520 0 0 304 2828 1099 4965 32 68 0
22 0 2 0 26120 64424 68936 0 0 40 31328 1076 1737 29 71 0
19 2 3 0 21960 64580 71500 0 0 36 7136 1167 1952 23 77 0
21 2 2 0 15496 64772 87160 0 0 136 30928 1143 2391 19 81 0
23 1 3 0 21720 64904 96284 0 0 304 19200 1118 3474 19 81 0
15 5 3 0 18008 65232 91716 0 0 164 17032 1153 1545 19 81 0
19 0 3 0 18376 64508 80988 0 0 68 0 1145 2052 11 89 0
19 2 3 0 6088 64696 95896 0 0 140 21112 1241 3761 11 89 0
15 7 2 0 16408 64184 89024 0 0 600 28172 1141 1273 25 75 0
15 10 0 0 54024 64420 50096 0 0 1544 0 1182 5076 35 65 0
24 1 2 0 8520 64660 92148 0 0 268 32496 1136 695 14 86 0
11 9 1 0 47560 64908 54868 0 0 436 864 1199 7414 49 51 0
18 0 2 0 14424 65260 86804 0 0 160 46684 1179 2291 16 84 0
15 8 4 0 16728 65552 86248 0 0 288 7612 1159 6312 27 73 0
21 1 4 0 14488 65868 93388 0 0 296 4 1188 11041 20 80 0
18 7 4 0 41752 66116 52124 0 0 196 880 1115 80040 16 84 0
2.6-test8-Vanilla:
This one is terrible. The disk led goes off for many seconds.
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
29 1 1 0 313908 9272 89024 0 0 0 0 2572 17213 82 18 0
29 2 1 0 313924 9272 89024 0 0 0 0 2216 90773 64 36 0
31 2 1 0 313660 9272 89024 0 0 0 0 1203 286591 29 71 0
30 2 1 0 313276 9280 89024 0 0 0 44 1294 283827 33 67 0
27 3 1 0 313756 9280 89024 0 0 0 0 2311 64008 78 22 0
29 3 1 0 314316 9280 89028 0 0 0 0 2513 2516 82 18 0
23 2 2 0 317620 9304 84124 0 0 28 6348 1865 86787 55 45 0
19 0 0 0 318148 9448 81264 0 0 140 540 2235 2050 78 22 0
14 3 2 0 317884 9524 81288 0 0 100 36 2460 2556 77 23 0
19 0 1 0 317476 9564 81300 0 0 32 160 2310 2426 76 24 0
24 0 1 0 307748 9584 90536 0 0 4 0 2314 2270 71 29 0
26 0 1 0 314036 9596 84132 0 0 0 0 2208 2606 68 32 0
24 0 2 0 304356 9616 93268 0 0 36 0 2486 2650 80 20 0
18 0 2 0 284724 9676 111564 0 0 24 12048 2296 2230 69 31 0
18 3 2 0 277100 9724 118888 0 0 0 21120 2690 2464 74 26 0
22 1 4 0 295068 9756 100984 0 0 0 9828 2584 2549 76 24 0
20 0 4 0 288220 9796 107940 0 0 4 288 2104 1927 60 40 0
22 0 4 0 287236 9852 108188 0 0 8 0 2293 2094 70 30 0
20 0 5 0 291212 9884 104208 0 0 12 0 2317 2212 79 21 0
22 0 5 0 295756 9904 99764 0 0 0 0 2395 2178 79 21 0
21 0 5 0 294740 9932 100796 0 0 4 0 2398 2441 79 21 0
23 0 5 0 287444 9952 107812 0 0 0 10620 2456 1828 83 17 0
21 0 5 0 296580 9956 98816 0 0 0 0 2413 1852 81 19 0
24 2 5 0 296564 9956 98816 0 0 0 0 2107 1114 83 17 0
26 0 5 0 296596 9956 98816 0 0 0 0 1920 1232 73 27 0
29 0 5 0 296380 9956 98820 0 0 4 0 2192 1731 77 23 0
28 0 5 0 296324 9968 98820 0 0 0 5956 2561 2448 81 19 0
32 0 5 0 296300 9968 98808 0 0 0 92 2019 1247 79 21 0
32 0 5 0 296308 9968 98808 0 0 0 0 2053 1444 83 17 0
32 0 5 0 296316 9968 98808 0 0 0 0 2070 1390 82 18 0
31 0 5 0 296324 9968 98808 0 0 0 0 2267 1900 81 19 0
32 0 5 0 296316 9976 98808 0 0 0 112 2292 1971 85 15 0
31 0 5 0 296324 9976 98808 0 0 0 0 2127 1392 84 16 0
32 0 5 0 296308 9976 98808 0 0 0 0 2284 1902 84 16 0
31 0 5 0 296308 9976 98808 0 0 0 0 2112 1617 81 19 0
31 0 5 0 296308 9976 98808 0 0 0 0 2160 1645 81 19 0
32 0 5 0 296236 9976 98808 0 0 0 0 2698 2965 81 19 0
32 0 5 0 296380 9976 98808 0 0 0 0 2151 1548 82 18 0
31 0 5 0 296364 9976 98808 0 0 0 0 2068 1309 83 17 0
32 0 5 0 296364 9976 98808 0 0 0 0 2337 2073 79 21 0
38 0 5 0 296212 9976 98808 0 0 0 0 2294 2077 81 19 0
36 0 5 0 296220 9976 98808 0 0 0 0 2294 1933 84 16 0
35 0 5 0 296236 9976 98808 0 0 0 0 2577 2775 84 16 0
35 0 5 0 296140 9976 98808 0 0 0 0 2565 2671 77 23 0
37 0 5 0 296236 9984 98808 0 0 0 68 1844 908 85 15 0
38 0 5 0 296236 9984 98808 0 0 0 0 1892 943 84 16 0
29 5 5 0 293348 10004 98884 0 0 100 220 1950 1235 81 19 0
25 10 4 0 267476 10080 123724 0 0 464 24416 2020 1288 72 28 0
26 10 4 0 267996 10084 124488 0 0 768 0 2382 1824 79 21 0
25 10 4 0 267684 10124 124948 0 0 500 0 2053 1389 83 17 0
25 10 4 0 266908 10168 125012 0 0 108 0 2179 1654 85 15 0
23 10 4 0 266468 10236 125148 0 0 204 0 2182 1673 82 18 0
27 10 4 0 266428 10296 125248 0 0 160 0 2496 2356 85 15 0
25 11 5 0 266340 10308 125248 0 0 12 0 2781 3085 84 16 0
22 13 5 0 266348 10316 125252 0 0 12 0 2828 3210 86 14 0
21 13 5 0 266364 10320 125256 0 0 8 0 2777 3052 86 14 0
22 13 5 0 266300 10324 125268 0 0 16 0 2769 3124 85 15 0
23 13 5 0 266148 10324 125268 0 0 0 0 2789 3229 83 17 0
33 4 2 0 266692 10380 124560 0 0 16 4744 2560 2557 77 23 0
31 5 3 0 276908 10400 114340 0 0 12 56 2676 2859 82 18 0
33 5 3 0 268884 10408 122176 0 0 0 0 2855 3324 83 17 0
34 5 3 0 265684 10412 125284 0 0 0 0 2947 3480 84 16 0
If you compare this with 2.4.22, it's io load is comparatively even
and it never stops:
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
15 9 1 0 198856 19864 185180 0 0 244 18060 659 8954 34 66 0
11 13 1 0 178604 19896 179860 0 0 52 19120 692 87760 37 63 0
9 15 1 0 192816 19940 159160 0 0 52 20720 685 94859 31 69 0
8 15 1 0 201836 19948 150144 0 0 0 17604 650 113990 34 66 0
10 15 1 0 198572 20004 149444 0 0 72 21172 728 100160 30 70 0
12 11 1 0 129312 20156 176044 0 0 140 8940 584 54396 30 70 0
16 8 1 0 84604 20288 185412 0 0 32 25852 746 48955 33 67 0
18 8 1 0 86116 20384 183840 0 0 96 13404 662 51891 27 73 0
19 7 1 0 88336 20508 182492 0 0 48 14340 697 40311 19 81 0
17 9 1 0 82640 20608 183760 0 0 100 23088 735 62380 39 61 0
2 21 1 0 61484 20628 188908 0 0 44 18112 674 45239 36 64 0
2 21 1 0 37096 20684 196092 0 0 40 24252 729 572 25 75 0
2 21 1 0 47580 20684 182996 0 0 0 13464 653 363 43 57 0
3 21 1 0 59624 20712 171620 0 0 8 22988 646 448 39 61 0
2 21 1 0 50584 20744 178204 0 0 16 23008 805 657 31 69 0
2 21 1 0 59244 20744 169552 0 0 24 17672 670 408 44 56 0
14 2 1 0 36096 20836 175416 0 0 4 19148 730 1044 40 60 0
15 7 1 76 35792 19384 150752 0 0 52 14492 658 843 22 78 0
11 15 1 76 34736 19540 153128 0 0 60 17396 745 1346 23 77 0
2 21 1 76 27060 19548 159248 0 0 44 21008 703 543 36 64 0
2 21 1 76 38796 19548 147504 0 0 0 18528 712 495 43 57 0
2 23 1 76 39448 19592 148160 0 0 56 23328 725 625 33 67 0
22 8 1 76 30200 19616 157304 0 0 0 20876 680 499 29 71 0
12 3 1 76 4420 19712 151292 0 0 64 23968 675 2676 24 76 0
15 5 1 76 4632 19764 116028 0 0 60 13824 695 1015 27 73 0
24 3 1 244 4356 19432 128800 0 0 24 23752 636 1174 28 72 0
23 3 1 244 29684 19532 113968 0 0 40 26124 724 18608 25 75 0
21 2 1 244 17316 19592 124904 0 0 88 18596 660 51583 38 62 0
17 9 1 244 20728 19660 107408 0 0 52 22244 710 55023 27 73 0
21 5 1 244 24172 19748 115872 0 0 108 22824 680 44490 25 75 0
23 4 1 244 4460 19832 113824 0 0 240 15136 691 44779 25 75 0
14 12 1 244 5588 19884 131408 0 0 72 20208 719 69408 25 75 0
23 4 1 244 4404 19964 133028 0 0 128 18564 659 51254 24 76 0
22 5 1 244 9084 20068 129856 0 0 208 16824 631 22191 28 72 0
7 17 1 244 35000 20100 104024 0 0 28 18816 708 106858 38 62 0
Also 2.4.21, it's behavior is similar to 2.4.22.
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
3 20 2 22480 5472 2032 82500 8 0 128 14716 3279 4137 71 26 3
8 19 1 22480 5068 2044 76988 56 0 96 15016 2704 104072 56 25 19
2 24 1 22480 4976 2044 77008 8 0 28 12836 3521 167313 52 48 0
3 24 1 22480 4788 2044 77036 72 0 100 16684 2886 155685 60 40 0
5 23 2 22480 12092 1876 83660 96 4 304 12556 2953 68285 39 61 0
4 27 2 22480 10796 1892 79136 56 0 144 8544 2373 139592 50 50 0
6 30 1 22480 20548 1912 72924 76 0 124 17428 2858 69541 39 61 0
6 26 1 22480 9284 1952 89440 36 0 152 17128 2334 65785 40 60 0
7 26 1 22480 9056 1952 89552 24 0 136 10440 2475 72782 42 58 0
6 26 2 22480 8428 1956 89732 128 0 312 22380 2704 79724 50 50 0
6 25 1 22340 29768 2016 67656 104 0 216 17048 2198 71301 42 58 0
6 31 1 21552 39852 2060 57028 124 0 288 16376 2845 39256 51 49 0
6 29 1 21552 39708 2064 57084 20 0 80 17832 2381 5321 50 50 0
7 21 1 21464 27672 2192 74152 184 0 356 6808 2006 5181 32 68 0
5 23 1 21400 27032 2196 62272 180 0 284 14552 2903 7905 40 60 0
6 23 1 21340 26540 2196 62312 204 0 244 15556 2279 6140 53 47 0
1 26 1 21284 25952 2212 62416 124 0 244 19804 3322 8600 54 46 0
Regards
Michael
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-27 21:52 2.6.0-test8/test9 io scheduler needs tuning? Michael Frank
@ 2003-10-27 22:55 ` cliff white
2003-10-27 23:50 ` Nick Piggin
0 siblings, 1 reply; 19+ messages in thread
From: cliff white @ 2003-10-27 22:55 UTC (permalink / raw)
To: Michael Frank; +Cc: linux-kernel
On Tue, 28 Oct 2003 05:52:45 +0800
Michael Frank <mhf@linuxmail.org> wrote:
> To my surprise 2.6 - which used to do better then 2.4 - does no longer
> handle these test that well.
>
> Generally, IDE IO throughput is _very_ uneven and IO _stops_ at times with the
> system cpu load very high (and the disk LED off).
>
> IMHO the CPU scheduling is OK but the IO scheduling acts up here.
>
> The test system is a 2.4GHz P4 with 512M RAM and a 55MB/s udma IDE harddisk.
>
> The tests load the system to loadavg > 30. IO should be about 20MB/s on avg.
>
> Enclosed are vmstat -1 logs for 2.6-test9-Vanilla, followed by 2.6-test8-Vanilla
> (-mm1 behaves similar), 2.4.22-Vanilla and 2.4.21+swsusp all compiled wo preempt.
>
> IO on 2.6 stops now for seconds at a time. -test8 is worse than -test9
We see the same delta at OSDL. Try repeating your tests with 'elevator=deadline'
to confirm.
For example, on the 8-cpu platform:
STP id Kernel Name MaxJPM Change Options
281669 linux-2.6.0-test8 7014.42 0.0
281671 linux-2.6.0-test8 8294.94 +18.26% elevator=deadline
The -mm kernels don't show this big delta. We also do not see this delta on
smaller machines
cliffw
>
> 2.6-test9-Vanilla:
>
> procs memory swap io system cpu
> r b w swpd free buff cache si so bi bo in cs us sy id
> 11 10 2 0 17164 60408 96292 0 0 0 5756 1147 30803 21 79 0
> 13 7 1 0 22968 60600 83876 0 0 0 30100 1079 6475 19 81 0
> 16 6 4 0 4728 59900 106028 0 0 4 29556 1122 71394 23 77 0
> 12 7 3 0 27128 60056 70588 0 0 68 6344 1212 111635 19 81 0
> 13 8 2 0 45176 60200 51656 0 0 0 22868 1238 20471 13 87 0
> 12 9 2 0 18544 60324 76668 0 0 0 14720 1267 36875 22 78 0
> 15 5 1 0 4784 60352 95196 0 0 184 37448 1191 144856 32 68 0
> 3 15 1 0 14768 59980 91368 0 0 180 12376 1112 133689 41 59 0
> 4 15 1 0 40816 60112 59320 0 0 164 188 1092 75854 31 69 0
> 6 14 1 0 4976 60164 102640 0 0 16 56904 1122 85364 27 73 0
> 11 8 1 0 25648 60188 79316 0 0 4 3784 1100 75808 26 74 0
> 9 8 3 0 8896 60280 103316 0 0 0 24772 1154 4031 23 77 0
> 5 14 2 0 23744 60408 75220 0 0 232 108 1157 86855 38 62 0
> 10 8 1 0 12032 60028 91796 0 0 8 27048 1272 79355 24 76 0
> 10 10 2 0 21120 60292 80144 0 0 0 49700 1092 3015 10 90 0
> 13 8 2 0 34064 60488 66528 0 0 0 172 1061 3871 10 90 0
> 14 7 2 0 40200 60540 79824 0 0 8 36864 1121 3138 12 88 0
> 20 2 2 0 42304 60544 78452 0 0 4 504 1103 281659 23 77 0
> 12 8 1 0 59024 60656 61688 0 0 28 26080 1156 158476 22 78 0
> 14 4 1 0 59536 60924 54128 0 0 0 9132 1251 3056 9 91 0
> 15 2 3 0 46528 61108 65628 0 0 0 23092 1360 58596 11 89 0
> 8 10 4 0 4288 61244 103836 0 0 56 25904 1135 33984 21 79 0
> 14 3 2 0 48640 61340 67512 0 0 56 18156 1252 53401 24 76 0
> 13 1 2 0 30336 61572 80856 0 0 4 15192 1365 1880 20 80 0
> 10 7 1 0 13376 61752 93496 0 0 176 2320 1225 21294 30 70 0
> 12 3 2 0 22272 62012 81292 0 0 112 17808 1238 38432 19 81 0
> 10 3 2 0 12736 62288 91356 0 0 192 21840 1121 2531 21 79 0
> 16 2 3 0 26752 62228 67660 0 0 56 32500 1745 2100 21 79 0
> 15 8 2 0 4208 62292 74004 0 0 20 108 1152 4642 37 63 0
> 17 7 4 0 25200 62444 68544 0 0 60 19184 1384 2337 20 80 0
> 22 4 4 0 45224 62540 59164 0 0 64 24852 1309 451 14 86 0
> 22 4 4 0 27240 62684 77048 0 0 0 7480 1280 323 15 85 0
> 24 3 3 0 26528 62960 66448 0 0 0 24788 1240 837 9 91 0
> 23 3 2 0 26656 63208 62952 0 0 4 204 1068 616 8 92 0
> 24 3 2 0 20704 63500 69628 0 0 40 0 1029 440 7 93 0
> 24 3 2 0 13400 63796 76536 0 0 4 0 1020 426 8 92 0
> 25 3 2 0 13400 64032 76340 0 0 0 0 1019 371 10 90 0
> 26 3 2 0 23688 64048 81792 0 0 124 31828 1098 2308 22 78 0
> 18 5 2 0 30664 64356 69520 0 0 304 2828 1099 4965 32 68 0
> 22 0 2 0 26120 64424 68936 0 0 40 31328 1076 1737 29 71 0
> 19 2 3 0 21960 64580 71500 0 0 36 7136 1167 1952 23 77 0
> 21 2 2 0 15496 64772 87160 0 0 136 30928 1143 2391 19 81 0
> 23 1 3 0 21720 64904 96284 0 0 304 19200 1118 3474 19 81 0
> 15 5 3 0 18008 65232 91716 0 0 164 17032 1153 1545 19 81 0
> 19 0 3 0 18376 64508 80988 0 0 68 0 1145 2052 11 89 0
> 19 2 3 0 6088 64696 95896 0 0 140 21112 1241 3761 11 89 0
> 15 7 2 0 16408 64184 89024 0 0 600 28172 1141 1273 25 75 0
> 15 10 0 0 54024 64420 50096 0 0 1544 0 1182 5076 35 65 0
> 24 1 2 0 8520 64660 92148 0 0 268 32496 1136 695 14 86 0
> 11 9 1 0 47560 64908 54868 0 0 436 864 1199 7414 49 51 0
> 18 0 2 0 14424 65260 86804 0 0 160 46684 1179 2291 16 84 0
> 15 8 4 0 16728 65552 86248 0 0 288 7612 1159 6312 27 73 0
> 21 1 4 0 14488 65868 93388 0 0 296 4 1188 11041 20 80 0
> 18 7 4 0 41752 66116 52124 0 0 196 880 1115 80040 16 84 0
>
> 2.6-test8-Vanilla:
>
> This one is terrible. The disk led goes off for many seconds.
>
> procs memory swap io system cpu
> r b w swpd free buff cache si so bi bo in cs us sy id
> 29 1 1 0 313908 9272 89024 0 0 0 0 2572 17213 82 18 0
> 29 2 1 0 313924 9272 89024 0 0 0 0 2216 90773 64 36 0
> 31 2 1 0 313660 9272 89024 0 0 0 0 1203 286591 29 71 0
> 30 2 1 0 313276 9280 89024 0 0 0 44 1294 283827 33 67 0
> 27 3 1 0 313756 9280 89024 0 0 0 0 2311 64008 78 22 0
> 29 3 1 0 314316 9280 89028 0 0 0 0 2513 2516 82 18 0
> 23 2 2 0 317620 9304 84124 0 0 28 6348 1865 86787 55 45 0
> 19 0 0 0 318148 9448 81264 0 0 140 540 2235 2050 78 22 0
> 14 3 2 0 317884 9524 81288 0 0 100 36 2460 2556 77 23 0
> 19 0 1 0 317476 9564 81300 0 0 32 160 2310 2426 76 24 0
> 24 0 1 0 307748 9584 90536 0 0 4 0 2314 2270 71 29 0
> 26 0 1 0 314036 9596 84132 0 0 0 0 2208 2606 68 32 0
> 24 0 2 0 304356 9616 93268 0 0 36 0 2486 2650 80 20 0
> 18 0 2 0 284724 9676 111564 0 0 24 12048 2296 2230 69 31 0
> 18 3 2 0 277100 9724 118888 0 0 0 21120 2690 2464 74 26 0
> 22 1 4 0 295068 9756 100984 0 0 0 9828 2584 2549 76 24 0
> 20 0 4 0 288220 9796 107940 0 0 4 288 2104 1927 60 40 0
> 22 0 4 0 287236 9852 108188 0 0 8 0 2293 2094 70 30 0
> 20 0 5 0 291212 9884 104208 0 0 12 0 2317 2212 79 21 0
> 22 0 5 0 295756 9904 99764 0 0 0 0 2395 2178 79 21 0
> 21 0 5 0 294740 9932 100796 0 0 4 0 2398 2441 79 21 0
> 23 0 5 0 287444 9952 107812 0 0 0 10620 2456 1828 83 17 0
> 21 0 5 0 296580 9956 98816 0 0 0 0 2413 1852 81 19 0
> 24 2 5 0 296564 9956 98816 0 0 0 0 2107 1114 83 17 0
> 26 0 5 0 296596 9956 98816 0 0 0 0 1920 1232 73 27 0
> 29 0 5 0 296380 9956 98820 0 0 4 0 2192 1731 77 23 0
> 28 0 5 0 296324 9968 98820 0 0 0 5956 2561 2448 81 19 0
> 32 0 5 0 296300 9968 98808 0 0 0 92 2019 1247 79 21 0
> 32 0 5 0 296308 9968 98808 0 0 0 0 2053 1444 83 17 0
> 32 0 5 0 296316 9968 98808 0 0 0 0 2070 1390 82 18 0
> 31 0 5 0 296324 9968 98808 0 0 0 0 2267 1900 81 19 0
> 32 0 5 0 296316 9976 98808 0 0 0 112 2292 1971 85 15 0
> 31 0 5 0 296324 9976 98808 0 0 0 0 2127 1392 84 16 0
> 32 0 5 0 296308 9976 98808 0 0 0 0 2284 1902 84 16 0
> 31 0 5 0 296308 9976 98808 0 0 0 0 2112 1617 81 19 0
> 31 0 5 0 296308 9976 98808 0 0 0 0 2160 1645 81 19 0
> 32 0 5 0 296236 9976 98808 0 0 0 0 2698 2965 81 19 0
> 32 0 5 0 296380 9976 98808 0 0 0 0 2151 1548 82 18 0
> 31 0 5 0 296364 9976 98808 0 0 0 0 2068 1309 83 17 0
> 32 0 5 0 296364 9976 98808 0 0 0 0 2337 2073 79 21 0
> 38 0 5 0 296212 9976 98808 0 0 0 0 2294 2077 81 19 0
> 36 0 5 0 296220 9976 98808 0 0 0 0 2294 1933 84 16 0
> 35 0 5 0 296236 9976 98808 0 0 0 0 2577 2775 84 16 0
> 35 0 5 0 296140 9976 98808 0 0 0 0 2565 2671 77 23 0
> 37 0 5 0 296236 9984 98808 0 0 0 68 1844 908 85 15 0
> 38 0 5 0 296236 9984 98808 0 0 0 0 1892 943 84 16 0
> 29 5 5 0 293348 10004 98884 0 0 100 220 1950 1235 81 19 0
> 25 10 4 0 267476 10080 123724 0 0 464 24416 2020 1288 72 28 0
> 26 10 4 0 267996 10084 124488 0 0 768 0 2382 1824 79 21 0
> 25 10 4 0 267684 10124 124948 0 0 500 0 2053 1389 83 17 0
> 25 10 4 0 266908 10168 125012 0 0 108 0 2179 1654 85 15 0
> 23 10 4 0 266468 10236 125148 0 0 204 0 2182 1673 82 18 0
> 27 10 4 0 266428 10296 125248 0 0 160 0 2496 2356 85 15 0
> 25 11 5 0 266340 10308 125248 0 0 12 0 2781 3085 84 16 0
> 22 13 5 0 266348 10316 125252 0 0 12 0 2828 3210 86 14 0
> 21 13 5 0 266364 10320 125256 0 0 8 0 2777 3052 86 14 0
> 22 13 5 0 266300 10324 125268 0 0 16 0 2769 3124 85 15 0
> 23 13 5 0 266148 10324 125268 0 0 0 0 2789 3229 83 17 0
> 33 4 2 0 266692 10380 124560 0 0 16 4744 2560 2557 77 23 0
> 31 5 3 0 276908 10400 114340 0 0 12 56 2676 2859 82 18 0
> 33 5 3 0 268884 10408 122176 0 0 0 0 2855 3324 83 17 0
> 34 5 3 0 265684 10412 125284 0 0 0 0 2947 3480 84 16 0
>
> If you compare this with 2.4.22, it's io load is comparatively even
> and it never stops:
>
> procs memory swap io system cpu
> r b w swpd free buff cache si so bi bo in cs us sy id
> 15 9 1 0 198856 19864 185180 0 0 244 18060 659 8954 34 66 0
> 11 13 1 0 178604 19896 179860 0 0 52 19120 692 87760 37 63 0
> 9 15 1 0 192816 19940 159160 0 0 52 20720 685 94859 31 69 0
> 8 15 1 0 201836 19948 150144 0 0 0 17604 650 113990 34 66 0
> 10 15 1 0 198572 20004 149444 0 0 72 21172 728 100160 30 70 0
> 12 11 1 0 129312 20156 176044 0 0 140 8940 584 54396 30 70 0
> 16 8 1 0 84604 20288 185412 0 0 32 25852 746 48955 33 67 0
> 18 8 1 0 86116 20384 183840 0 0 96 13404 662 51891 27 73 0
> 19 7 1 0 88336 20508 182492 0 0 48 14340 697 40311 19 81 0
> 17 9 1 0 82640 20608 183760 0 0 100 23088 735 62380 39 61 0
> 2 21 1 0 61484 20628 188908 0 0 44 18112 674 45239 36 64 0
> 2 21 1 0 37096 20684 196092 0 0 40 24252 729 572 25 75 0
> 2 21 1 0 47580 20684 182996 0 0 0 13464 653 363 43 57 0
> 3 21 1 0 59624 20712 171620 0 0 8 22988 646 448 39 61 0
> 2 21 1 0 50584 20744 178204 0 0 16 23008 805 657 31 69 0
> 2 21 1 0 59244 20744 169552 0 0 24 17672 670 408 44 56 0
> 14 2 1 0 36096 20836 175416 0 0 4 19148 730 1044 40 60 0
> 15 7 1 76 35792 19384 150752 0 0 52 14492 658 843 22 78 0
> 11 15 1 76 34736 19540 153128 0 0 60 17396 745 1346 23 77 0
> 2 21 1 76 27060 19548 159248 0 0 44 21008 703 543 36 64 0
> 2 21 1 76 38796 19548 147504 0 0 0 18528 712 495 43 57 0
> 2 23 1 76 39448 19592 148160 0 0 56 23328 725 625 33 67 0
> 22 8 1 76 30200 19616 157304 0 0 0 20876 680 499 29 71 0
> 12 3 1 76 4420 19712 151292 0 0 64 23968 675 2676 24 76 0
> 15 5 1 76 4632 19764 116028 0 0 60 13824 695 1015 27 73 0
> 24 3 1 244 4356 19432 128800 0 0 24 23752 636 1174 28 72 0
> 23 3 1 244 29684 19532 113968 0 0 40 26124 724 18608 25 75 0
> 21 2 1 244 17316 19592 124904 0 0 88 18596 660 51583 38 62 0
> 17 9 1 244 20728 19660 107408 0 0 52 22244 710 55023 27 73 0
> 21 5 1 244 24172 19748 115872 0 0 108 22824 680 44490 25 75 0
> 23 4 1 244 4460 19832 113824 0 0 240 15136 691 44779 25 75 0
> 14 12 1 244 5588 19884 131408 0 0 72 20208 719 69408 25 75 0
> 23 4 1 244 4404 19964 133028 0 0 128 18564 659 51254 24 76 0
> 22 5 1 244 9084 20068 129856 0 0 208 16824 631 22191 28 72 0
> 7 17 1 244 35000 20100 104024 0 0 28 18816 708 106858 38 62 0
>
> Also 2.4.21, it's behavior is similar to 2.4.22.
>
> procs memory swap io system cpu
> r b w swpd free buff cache si so bi bo in cs us sy id
> 3 20 2 22480 5472 2032 82500 8 0 128 14716 3279 4137 71 26 3
> 8 19 1 22480 5068 2044 76988 56 0 96 15016 2704 104072 56 25 19
> 2 24 1 22480 4976 2044 77008 8 0 28 12836 3521 167313 52 48 0
> 3 24 1 22480 4788 2044 77036 72 0 100 16684 2886 155685 60 40 0
> 5 23 2 22480 12092 1876 83660 96 4 304 12556 2953 68285 39 61 0
> 4 27 2 22480 10796 1892 79136 56 0 144 8544 2373 139592 50 50 0
> 6 30 1 22480 20548 1912 72924 76 0 124 17428 2858 69541 39 61 0
> 6 26 1 22480 9284 1952 89440 36 0 152 17128 2334 65785 40 60 0
> 7 26 1 22480 9056 1952 89552 24 0 136 10440 2475 72782 42 58 0
> 6 26 2 22480 8428 1956 89732 128 0 312 22380 2704 79724 50 50 0
> 6 25 1 22340 29768 2016 67656 104 0 216 17048 2198 71301 42 58 0
> 6 31 1 21552 39852 2060 57028 124 0 288 16376 2845 39256 51 49 0
> 6 29 1 21552 39708 2064 57084 20 0 80 17832 2381 5321 50 50 0
> 7 21 1 21464 27672 2192 74152 184 0 356 6808 2006 5181 32 68 0
> 5 23 1 21400 27032 2196 62272 180 0 284 14552 2903 7905 40 60 0
> 6 23 1 21340 26540 2196 62312 204 0 244 15556 2279 6140 53 47 0
> 1 26 1 21284 25952 2212 62416 124 0 244 19804 3322 8600 54 46 0
>
> Regards
> Michael
>
>
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
>
--
The church is near, but the road is icy.
The bar is far, but i will walk carefully. - Russian proverb
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-27 22:55 ` cliff white
@ 2003-10-27 23:50 ` Nick Piggin
2003-10-28 1:37 ` Nigel Cunningham
` (2 more replies)
0 siblings, 3 replies; 19+ messages in thread
From: Nick Piggin @ 2003-10-27 23:50 UTC (permalink / raw)
To: cliff white; +Cc: Michael Frank, linux-kernel
[-- Attachment #1: Type: text/plain, Size: 1330 bytes --]
cliff white wrote:
>On Tue, 28 Oct 2003 05:52:45 +0800
>Michael Frank <mhf@linuxmail.org> wrote:
>
>
>>To my surprise 2.6 - which used to do better then 2.4 - does no longer
>>handle these test that well.
>>
>>Generally, IDE IO throughput is _very_ uneven and IO _stops_ at times with the
>>system cpu load very high (and the disk LED off).
>>
>>IMHO the CPU scheduling is OK but the IO scheduling acts up here.
>>
>>The test system is a 2.4GHz P4 with 512M RAM and a 55MB/s udma IDE harddisk.
>>
>>The tests load the system to loadavg > 30. IO should be about 20MB/s on avg.
>>
>>Enclosed are vmstat -1 logs for 2.6-test9-Vanilla, followed by 2.6-test8-Vanilla
>>(-mm1 behaves similar), 2.4.22-Vanilla and 2.4.21+swsusp all compiled wo preempt.
>>
>>IO on 2.6 stops now for seconds at a time. -test8 is worse than -test9
>>
>
>We see the same delta at OSDL. Try repeating your tests with 'elevator=deadline'
>to confirm.
>For example, on the 8-cpu platform:
>STP id Kernel Name MaxJPM Change Options
>281669 linux-2.6.0-test8 7014.42 0.0
>281671 linux-2.6.0-test8 8294.94 +18.26% elevator=deadline
>
>The -mm kernels don't show this big delta. We also do not see this delta on
>smaller machines
>
I'm working with Randy to fix this. Attached is what I have so far. See how
you go with it.
[-- Attachment #2: as-fix.patch --]
[-- Type: text/plain, Size: 3392 bytes --]
linux-2.6-npiggin/drivers/block/as-iosched.c | 30 ++++++---------------------
1 files changed, 7 insertions(+), 23 deletions(-)
diff -puN drivers/block/as-iosched.c~as-fix drivers/block/as-iosched.c
--- linux-2.6/drivers/block/as-iosched.c~as-fix 2003-10-27 14:53:47.000000000 +1100
+++ linux-2.6-npiggin/drivers/block/as-iosched.c 2003-10-27 14:54:04.000000000 +1100
@@ -99,7 +99,6 @@ struct as_data {
sector_t last_sector[2]; /* last REQ_SYNC & REQ_ASYNC sectors */
struct list_head *dispatch; /* driver dispatch queue */
struct list_head *hash; /* request hash */
- unsigned long new_success; /* anticipation success on new proc */
unsigned long current_batch_expires;
unsigned long last_check_fifo[2];
int changed_batch; /* 1: waiting for old batch to end */
@@ -585,18 +584,11 @@ static void as_antic_stop(struct as_data
int status = ad->antic_status;
if (status == ANTIC_WAIT_REQ || status == ANTIC_WAIT_NEXT) {
- struct as_io_context *aic;
-
if (status == ANTIC_WAIT_NEXT)
del_timer(&ad->antic_timer);
ad->antic_status = ANTIC_FINISHED;
/* see as_work_handler */
kblockd_schedule_work(&ad->antic_work);
-
- aic = ad->io_context->aic;
- if (aic->seek_samples == 0)
- /* new process */
- ad->new_success = (ad->new_success * 3) / 4 + 256;
}
}
@@ -612,14 +604,8 @@ static void as_antic_timeout(unsigned lo
spin_lock_irqsave(q->queue_lock, flags);
if (ad->antic_status == ANTIC_WAIT_REQ
|| ad->antic_status == ANTIC_WAIT_NEXT) {
- struct as_io_context *aic;
ad->antic_status = ANTIC_FINISHED;
kblockd_schedule_work(&ad->antic_work);
-
- aic = ad->io_context->aic;
- if (aic->seek_samples == 0)
- /* new process */
- ad->new_success = (ad->new_success * 3) / 4;
}
spin_unlock_irqrestore(q->queue_lock, flags);
}
@@ -708,11 +694,10 @@ static int as_can_break_anticipation(str
return 1;
}
- if (ad->new_success < 256 &&
- (aic->seek_samples == 0 || aic->ttime_samples == 0)) {
+ if (aic->seek_samples == 0 || aic->ttime_samples == 0) {
/*
- * Process has just started IO and we have a bad history of
- * success anticipating on new processes!
+ * Process has just started IO. Don't anticipate.
+ * TODO! Must fix this up.
*/
return 1;
}
@@ -1292,7 +1277,7 @@ fifo_expired:
ad->changed_batch = 0;
- arq->request->flags |= REQ_SOFTBARRIER;
+// arq->request->flags |= REQ_SOFTBARRIER;
}
/*
@@ -1391,6 +1376,7 @@ static void as_add_request(struct as_dat
} else {
as_add_aliased_request(ad, arq, alias);
+
/*
* have we been anticipating this request?
* or does it come from the same process as the one we are
@@ -1427,8 +1413,6 @@ static void as_requeue_request(request_q
/* Stop anticipating - let this request get through */
as_antic_stop(ad);
-
- return;
}
static void
@@ -1437,10 +1421,12 @@ as_insert_request(request_queue_t *q, st
struct as_data *ad = q->elevator.elevator_data;
struct as_rq *arq = RQ_DATA(rq);
+#if 0
/* barriers must flush the reorder queue */
if (unlikely(rq->flags & (REQ_SOFTBARRIER | REQ_HARDBARRIER)
&& where == ELEVATOR_INSERT_SORT))
where = ELEVATOR_INSERT_BACK;
+#endif
switch (where) {
case ELEVATOR_INSERT_BACK:
@@ -1823,8 +1809,6 @@ static int as_init(request_queue_t *q, e
if (ad->write_batch_count < 2)
ad->write_batch_count = 2;
- ad->new_success = 512;
-
return 0;
}
_
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-27 23:50 ` Nick Piggin
@ 2003-10-28 1:37 ` Nigel Cunningham
2003-10-28 1:48 ` Nick Piggin
` (2 more replies)
2003-10-28 4:13 ` Michael Frank
2003-10-28 17:26 ` cliff white
2 siblings, 3 replies; 19+ messages in thread
From: Nigel Cunningham @ 2003-10-28 1:37 UTC (permalink / raw)
To: Nick Piggin; +Cc: cliff white, Michael Frank, Linux Kernel Mailing List
I'll try it with my software suspend patch. Under 2.4, I get around 45
pages per jiffy written when suspending. Under 2.6, I'm currently
getting 2-4, so any improvement should be obvious!
Regards,
Nigel
On Tue, 2003-10-28 at 12:50, Nick Piggin wrote:
> cliff white wrote:
>
> >On Tue, 28 Oct 2003 05:52:45 +0800
> >Michael Frank <mhf@linuxmail.org> wrote:
> >
> >
> >>To my surprise 2.6 - which used to do better then 2.4 - does no longer
> >>handle these test that well.
> >>
> >>Generally, IDE IO throughput is _very_ uneven and IO _stops_ at times with the
> >>system cpu load very high (and the disk LED off).
> >>
> >>IMHO the CPU scheduling is OK but the IO scheduling acts up here.
> >>
> >>The test system is a 2.4GHz P4 with 512M RAM and a 55MB/s udma IDE harddisk.
> >>
> >>The tests load the system to loadavg > 30. IO should be about 20MB/s on avg.
> >>
> >>Enclosed are vmstat -1 logs for 2.6-test9-Vanilla, followed by 2.6-test8-Vanilla
> >>(-mm1 behaves similar), 2.4.22-Vanilla and 2.4.21+swsusp all compiled wo preempt.
> >>
> >>IO on 2.6 stops now for seconds at a time. -test8 is worse than -test9
> >>
> >
> >We see the same delta at OSDL. Try repeating your tests with 'elevator=deadline'
> >to confirm.
> >For example, on the 8-cpu platform:
> >STP id Kernel Name MaxJPM Change Options
> >281669 linux-2.6.0-test8 7014.42 0.0
> >281671 linux-2.6.0-test8 8294.94 +18.26% elevator=deadline
> >
> >The -mm kernels don't show this big delta. We also do not see this delta on
> >smaller machines
> >
>
> I'm working with Randy to fix this. Attached is what I have so far. See how
> you go with it.
>
>
> ______________________________________________________________________
>
> linux-2.6-npiggin/drivers/block/as-iosched.c | 30 ++++++---------------------
> 1 files changed, 7 insertions(+), 23 deletions(-)
>
> diff -puN drivers/block/as-iosched.c~as-fix drivers/block/as-iosched.c
> --- linux-2.6/drivers/block/as-iosched.c~as-fix 2003-10-27 14:53:47.000000000 +1100
> +++ linux-2.6-npiggin/drivers/block/as-iosched.c 2003-10-27 14:54:04.000000000 +1100
> @@ -99,7 +99,6 @@ struct as_data {
> sector_t last_sector[2]; /* last REQ_SYNC & REQ_ASYNC sectors */
> struct list_head *dispatch; /* driver dispatch queue */
> struct list_head *hash; /* request hash */
> - unsigned long new_success; /* anticipation success on new proc */
> unsigned long current_batch_expires;
> unsigned long last_check_fifo[2];
> int changed_batch; /* 1: waiting for old batch to end */
> @@ -585,18 +584,11 @@ static void as_antic_stop(struct as_data
> int status = ad->antic_status;
>
> if (status == ANTIC_WAIT_REQ || status == ANTIC_WAIT_NEXT) {
> - struct as_io_context *aic;
> -
> if (status == ANTIC_WAIT_NEXT)
> del_timer(&ad->antic_timer);
> ad->antic_status = ANTIC_FINISHED;
> /* see as_work_handler */
> kblockd_schedule_work(&ad->antic_work);
> -
> - aic = ad->io_context->aic;
> - if (aic->seek_samples == 0)
> - /* new process */
> - ad->new_success = (ad->new_success * 3) / 4 + 256;
> }
> }
>
> @@ -612,14 +604,8 @@ static void as_antic_timeout(unsigned lo
> spin_lock_irqsave(q->queue_lock, flags);
> if (ad->antic_status == ANTIC_WAIT_REQ
> || ad->antic_status == ANTIC_WAIT_NEXT) {
> - struct as_io_context *aic;
> ad->antic_status = ANTIC_FINISHED;
> kblockd_schedule_work(&ad->antic_work);
> -
> - aic = ad->io_context->aic;
> - if (aic->seek_samples == 0)
> - /* new process */
> - ad->new_success = (ad->new_success * 3) / 4;
> }
> spin_unlock_irqrestore(q->queue_lock, flags);
> }
> @@ -708,11 +694,10 @@ static int as_can_break_anticipation(str
> return 1;
> }
>
> - if (ad->new_success < 256 &&
> - (aic->seek_samples == 0 || aic->ttime_samples == 0)) {
> + if (aic->seek_samples == 0 || aic->ttime_samples == 0) {
> /*
> - * Process has just started IO and we have a bad history of
> - * success anticipating on new processes!
> + * Process has just started IO. Don't anticipate.
> + * TODO! Must fix this up.
> */
> return 1;
> }
> @@ -1292,7 +1277,7 @@ fifo_expired:
>
> ad->changed_batch = 0;
>
> - arq->request->flags |= REQ_SOFTBARRIER;
> +// arq->request->flags |= REQ_SOFTBARRIER;
> }
>
> /*
> @@ -1391,6 +1376,7 @@ static void as_add_request(struct as_dat
>
> } else {
> as_add_aliased_request(ad, arq, alias);
> +
> /*
> * have we been anticipating this request?
> * or does it come from the same process as the one we are
> @@ -1427,8 +1413,6 @@ static void as_requeue_request(request_q
>
> /* Stop anticipating - let this request get through */
> as_antic_stop(ad);
> -
> - return;
> }
>
> static void
> @@ -1437,10 +1421,12 @@ as_insert_request(request_queue_t *q, st
> struct as_data *ad = q->elevator.elevator_data;
> struct as_rq *arq = RQ_DATA(rq);
>
> +#if 0
> /* barriers must flush the reorder queue */
> if (unlikely(rq->flags & (REQ_SOFTBARRIER | REQ_HARDBARRIER)
> && where == ELEVATOR_INSERT_SORT))
> where = ELEVATOR_INSERT_BACK;
> +#endif
>
> switch (where) {
> case ELEVATOR_INSERT_BACK:
> @@ -1823,8 +1809,6 @@ static int as_init(request_queue_t *q, e
> if (ad->write_batch_count < 2)
> ad->write_batch_count = 2;
>
> - ad->new_success = 512;
> -
> return 0;
> }
>
>
> _
--
Nigel Cunningham
495 St Georges Road South, Hastings 4201, New Zealand
Evolution (n): A hypothetical process whereby infinitely improbable events occur
with alarming frequency, order arises from chaos, and no one is given credit.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 1:37 ` Nigel Cunningham
@ 2003-10-28 1:48 ` Nick Piggin
2003-10-28 2:58 ` Michael Frank
2003-10-28 2:57 ` Michael Frank
2003-10-28 3:31 ` Nigel Cunningham
2 siblings, 1 reply; 19+ messages in thread
From: Nick Piggin @ 2003-10-28 1:48 UTC (permalink / raw)
To: Nigel Cunningham; +Cc: cliff white, Michael Frank, Linux Kernel Mailing List
Nigel Cunningham wrote:
>I'll try it with my software suspend patch. Under 2.4, I get around 45
>pages per jiffy written when suspending. Under 2.6, I'm currently
>getting 2-4, so any improvement should be obvious!
>
It might speed it up a bit for you. Although maybe you are measuring
with 100 vs 1000 jiffies/s?
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 1:37 ` Nigel Cunningham
2003-10-28 1:48 ` Nick Piggin
@ 2003-10-28 2:57 ` Michael Frank
2003-10-28 3:31 ` Nigel Cunningham
2 siblings, 0 replies; 19+ messages in thread
From: Michael Frank @ 2003-10-28 2:57 UTC (permalink / raw)
To: Nigel Cunningham, Nick Piggin; +Cc: cliff white, Linux Kernel Mailing List
On Tuesday 28 October 2003 09:37, Nigel Cunningham wrote:
> I'll try it with my software suspend patch. Under 2.4, I get around 45
> pages per jiffy written when suspending. Under 2.6, I'm currently
> getting 2-4, so any improvement should be obvious!
Strange, I had no problems suspending/resuming with -alpha1 - it seems to do "normal (2.4)" speed ...
Will try again and let you know.
Regards
Michael
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 1:48 ` Nick Piggin
@ 2003-10-28 2:58 ` Michael Frank
0 siblings, 0 replies; 19+ messages in thread
From: Michael Frank @ 2003-10-28 2:58 UTC (permalink / raw)
To: Nick Piggin, Nigel Cunningham; +Cc: cliff white, Linux Kernel Mailing List
On Tuesday 28 October 2003 09:48, Nick Piggin wrote:
>
> Nigel Cunningham wrote:
>
> >I'll try it with my software suspend patch. Under 2.4, I get around 45
> >pages per jiffy written when suspending. Under 2.6, I'm currently
> >getting 2-4, so any improvement should be obvious!
> >
>
> It might speed it up a bit for you. Although maybe you are measuring
> with 100 vs 1000 jiffies/s?
Good point ;)
Regards
Michael
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 1:37 ` Nigel Cunningham
2003-10-28 1:48 ` Nick Piggin
2003-10-28 2:57 ` Michael Frank
@ 2003-10-28 3:31 ` Nigel Cunningham
2003-10-28 3:54 ` Nick Piggin
2 siblings, 1 reply; 19+ messages in thread
From: Nigel Cunningham @ 2003-10-28 3:31 UTC (permalink / raw)
To: Nick Piggin; +Cc: cliff white, Michael Frank, Linux Kernel Mailing List
As you rightly guessed, I was forgetting there are now 1000 jiffies per
second.
With your patch applied, I can achieve something close to 2.4
performance, but only if I set the limit on the number of pages to
submit at one time quite high. If I set it to 3000, I can get 20107 4K
pages written in 5267 jiffies (14MB/s) and can read them back at resume
time (so cache is not a factor) in 4620 jiffies (16MB/s). In 2.4, I
normally set the limit on async commits to 100, and achieve the same
performance. 100 here makes it very jerky and much slower.
Could there be some timeout value on BIOs that I might be able to
tweak/disable during suspend?
Regards,
Nigel
On Tue, 2003-10-28 at 14:37, Nigel Cunningham wrote:
> I'll try it with my software suspend patch. Under 2.4, I get around 45
> pages per jiffy written when suspending. Under 2.6, I'm currently
> getting 2-4, so any improvement should be obvious!
>
> Regards,
>
> Nigel
>
> On Tue, 2003-10-28 at 12:50, Nick Piggin wrote:
> > cliff white wrote:
> >
> > >On Tue, 28 Oct 2003 05:52:45 +0800
> > >Michael Frank <mhf@linuxmail.org> wrote:
> > >
> > >
> > >>To my surprise 2.6 - which used to do better then 2.4 - does no longer
> > >>handle these test that well.
> > >>
> > >>Generally, IDE IO throughput is _very_ uneven and IO _stops_ at times with the
> > >>system cpu load very high (and the disk LED off).
> > >>
> > >>IMHO the CPU scheduling is OK but the IO scheduling acts up here.
> > >>
> > >>The test system is a 2.4GHz P4 with 512M RAM and a 55MB/s udma IDE harddisk.
> > >>
> > >>The tests load the system to loadavg > 30. IO should be about 20MB/s on avg.
> > >>
> > >>Enclosed are vmstat -1 logs for 2.6-test9-Vanilla, followed by 2.6-test8-Vanilla
> > >>(-mm1 behaves similar), 2.4.22-Vanilla and 2.4.21+swsusp all compiled wo preempt.
> > >>
> > >>IO on 2.6 stops now for seconds at a time. -test8 is worse than -test9
> > >>
> > >
> > >We see the same delta at OSDL. Try repeating your tests with 'elevator=deadline'
> > >to confirm.
> > >For example, on the 8-cpu platform:
> > >STP id Kernel Name MaxJPM Change Options
> > >281669 linux-2.6.0-test8 7014.42 0.0
> > >281671 linux-2.6.0-test8 8294.94 +18.26% elevator=deadline
> > >
> > >The -mm kernels don't show this big delta. We also do not see this delta on
> > >smaller machines
> > >
> >
> > I'm working with Randy to fix this. Attached is what I have so far. See how
> > you go with it.
> >
> >
> > ______________________________________________________________________
> >
> > linux-2.6-npiggin/drivers/block/as-iosched.c | 30 ++++++---------------------
> > 1 files changed, 7 insertions(+), 23 deletions(-)
> >
> > diff -puN drivers/block/as-iosched.c~as-fix drivers/block/as-iosched.c
> > --- linux-2.6/drivers/block/as-iosched.c~as-fix 2003-10-27 14:53:47.000000000 +1100
> > +++ linux-2.6-npiggin/drivers/block/as-iosched.c 2003-10-27 14:54:04.000000000 +1100
> > @@ -99,7 +99,6 @@ struct as_data {
> > sector_t last_sector[2]; /* last REQ_SYNC & REQ_ASYNC sectors */
> > struct list_head *dispatch; /* driver dispatch queue */
> > struct list_head *hash; /* request hash */
> > - unsigned long new_success; /* anticipation success on new proc */
> > unsigned long current_batch_expires;
> > unsigned long last_check_fifo[2];
> > int changed_batch; /* 1: waiting for old batch to end */
> > @@ -585,18 +584,11 @@ static void as_antic_stop(struct as_data
> > int status = ad->antic_status;
> >
> > if (status == ANTIC_WAIT_REQ || status == ANTIC_WAIT_NEXT) {
> > - struct as_io_context *aic;
> > -
> > if (status == ANTIC_WAIT_NEXT)
> > del_timer(&ad->antic_timer);
> > ad->antic_status = ANTIC_FINISHED;
> > /* see as_work_handler */
> > kblockd_schedule_work(&ad->antic_work);
> > -
> > - aic = ad->io_context->aic;
> > - if (aic->seek_samples == 0)
> > - /* new process */
> > - ad->new_success = (ad->new_success * 3) / 4 + 256;
> > }
> > }
> >
> > @@ -612,14 +604,8 @@ static void as_antic_timeout(unsigned lo
> > spin_lock_irqsave(q->queue_lock, flags);
> > if (ad->antic_status == ANTIC_WAIT_REQ
> > || ad->antic_status == ANTIC_WAIT_NEXT) {
> > - struct as_io_context *aic;
> > ad->antic_status = ANTIC_FINISHED;
> > kblockd_schedule_work(&ad->antic_work);
> > -
> > - aic = ad->io_context->aic;
> > - if (aic->seek_samples == 0)
> > - /* new process */
> > - ad->new_success = (ad->new_success * 3) / 4;
> > }
> > spin_unlock_irqrestore(q->queue_lock, flags);
> > }
> > @@ -708,11 +694,10 @@ static int as_can_break_anticipation(str
> > return 1;
> > }
> >
> > - if (ad->new_success < 256 &&
> > - (aic->seek_samples == 0 || aic->ttime_samples == 0)) {
> > + if (aic->seek_samples == 0 || aic->ttime_samples == 0) {
> > /*
> > - * Process has just started IO and we have a bad history of
> > - * success anticipating on new processes!
> > + * Process has just started IO. Don't anticipate.
> > + * TODO! Must fix this up.
> > */
> > return 1;
> > }
> > @@ -1292,7 +1277,7 @@ fifo_expired:
> >
> > ad->changed_batch = 0;
> >
> > - arq->request->flags |= REQ_SOFTBARRIER;
> > +// arq->request->flags |= REQ_SOFTBARRIER;
> > }
> >
> > /*
> > @@ -1391,6 +1376,7 @@ static void as_add_request(struct as_dat
> >
> > } else {
> > as_add_aliased_request(ad, arq, alias);
> > +
> > /*
> > * have we been anticipating this request?
> > * or does it come from the same process as the one we are
> > @@ -1427,8 +1413,6 @@ static void as_requeue_request(request_q
> >
> > /* Stop anticipating - let this request get through */
> > as_antic_stop(ad);
> > -
> > - return;
> > }
> >
> > static void
> > @@ -1437,10 +1421,12 @@ as_insert_request(request_queue_t *q, st
> > struct as_data *ad = q->elevator.elevator_data;
> > struct as_rq *arq = RQ_DATA(rq);
> >
> > +#if 0
> > /* barriers must flush the reorder queue */
> > if (unlikely(rq->flags & (REQ_SOFTBARRIER | REQ_HARDBARRIER)
> > && where == ELEVATOR_INSERT_SORT))
> > where = ELEVATOR_INSERT_BACK;
> > +#endif
> >
> > switch (where) {
> > case ELEVATOR_INSERT_BACK:
> > @@ -1823,8 +1809,6 @@ static int as_init(request_queue_t *q, e
> > if (ad->write_batch_count < 2)
> > ad->write_batch_count = 2;
> >
> > - ad->new_success = 512;
> > -
> > return 0;
> > }
> >
> >
> > _
--
Nigel Cunningham
495 St Georges Road South, Hastings 4201, New Zealand
Evolution (n): A hypothetical process whereby infinitely improbable events occur
with alarming frequency, order arises from chaos, and no one is given credit.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 3:31 ` Nigel Cunningham
@ 2003-10-28 3:54 ` Nick Piggin
2003-10-28 7:48 ` Michael Frank
0 siblings, 1 reply; 19+ messages in thread
From: Nick Piggin @ 2003-10-28 3:54 UTC (permalink / raw)
To: Nigel Cunningham; +Cc: cliff white, Michael Frank, Linux Kernel Mailing List
Nigel Cunningham wrote:
>As you rightly guessed, I was forgetting there are now 1000 jiffies per
>second.
>
>With your patch applied, I can achieve something close to 2.4
>performance, but only if I set the limit on the number of pages to
>submit at one time quite high. If I set it to 3000, I can get 20107 4K
>pages written in 5267 jiffies (14MB/s) and can read them back at resume
>time (so cache is not a factor) in 4620 jiffies (16MB/s). In 2.4, I
>normally set the limit on async commits to 100, and achieve the same
>performance. 100 here makes it very jerky and much slower.
>
>Could there be some timeout value on BIOs that I might be able to
>tweak/disable during suspend?
>
Try setting /sys/block/xxx/queue/iosched/antic_expire to 0 on your
device under IO and see how it goes. That shouldn't be causing the
problem though, especially as you are mostly writing I think?
Otherwise might possibly be the queue plugging stuff, or maybe a
regression in the disk driver.
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-27 23:50 ` Nick Piggin
2003-10-28 1:37 ` Nigel Cunningham
@ 2003-10-28 4:13 ` Michael Frank
2003-10-28 4:30 ` Nick Piggin
2003-10-28 17:26 ` cliff white
2 siblings, 1 reply; 19+ messages in thread
From: Michael Frank @ 2003-10-28 4:13 UTC (permalink / raw)
To: Nick Piggin, cliff white; +Cc: linux-kernel
On Tuesday 28 October 2003 07:50, Nick Piggin wrote:
>
> cliff white wrote:
>
> >On Tue, 28 Oct 2003 05:52:45 +0800
> >Michael Frank <mhf@linuxmail.org> wrote:
> >
> >
> >>To my surprise 2.6 - which used to do better then 2.4 - does no longer
> >>handle these test that well.
> >>
> >>Generally, IDE IO throughput is _very_ uneven and IO _stops_ at times with the
> >>system cpu load very high (and the disk LED off).
> >>
> >>IMHO the CPU scheduling is OK but the IO scheduling acts up here.
> >>
> >>The test system is a 2.4GHz P4 with 512M RAM and a 55MB/s udma IDE harddisk.
> >>
> >>The tests load the system to loadavg > 30. IO should be about 20MB/s on avg.
> >>
> >>Enclosed are vmstat -1 logs for 2.6-test9-Vanilla, followed by 2.6-test8-Vanilla
> >>(-mm1 behaves similar), 2.4.22-Vanilla and 2.4.21+swsusp all compiled wo preempt.
> >>
> >>IO on 2.6 stops now for seconds at a time. -test8 is worse than -test9
> >>
> >
> >We see the same delta at OSDL. Try repeating your tests with 'elevator=deadline'
> >to confirm.
> >For example, on the 8-cpu platform:
> >STP id Kernel Name MaxJPM Change Options
> >281669 linux-2.6.0-test8 7014.42 0.0
> >281671 linux-2.6.0-test8 8294.94 +18.26% elevator=deadline
> >
> >The -mm kernels don't show this big delta. We also do not see this delta on
> >smaller machines
> >
>
> I'm working with Randy to fix this. Attached is what I have so far. See how
> you go with it.
>
>
This has been done without running a kernel compile, by $ ti-tests/ti stat ub17 ddw 4 5000
Seems to be more even on average but still drops IO too low and then gets overloaded.
By "too low" I mean io bo less than 10000.
By overloaded I mean io bo goes much above 40000. The disk can do maybe 60000.
When io bo is much above 40000, cpu scheduling is being impaired as indicated
by vmstat stopping output for a second or so...
Regards
Michael
19 0 2 0 16424 35736 66552 0 0 0 22524 1108 5664 12 88 0
19 1 3 0 22760 36004 72688 0 0 0 24888 1056 495 10 90 0
13 6 3 0 30568 36132 65752 0 0 8 37452 1310 957 21 79 0
17 0 3 0 20904 36432 62392 0 0 0 24260 1120 5254 11 89 0
18 3 5 0 4536 36040 79948 0 0 0 34364 1125 2832 16 84 0
24 4 3 0 34296 35552 63120 0 0 128 24848 1183 3312 21 79 0
15 6 3 0 51704 35580 46760 0 0 12 4176 1159 3910 33 67 0
17 4 5 0 17724 35340 69152 0 0 36 28940 1287 2030 26 74 0
21 4 3 0 17020 35404 69628 0 0 0 22952 1214 3587 24 76 0
11 9 2 0 13756 35528 72976 0 0 8 25028 1163 4727 22 78 0
19 2 5 0 23996 35260 83608 0 0 0 8016 1348 613 14 86 0
20 2 3 0 29884 35432 81048 0 0 64 40828 1307 809 9 91 0
20 5 3 0 31228 35616 72920 0 0 288 12340 1251 1215 29 71 0
24 2 1 0 12412 35840 91068 0 0 220 34636 1151 2531 28 72 0
18 8 3 0 5120 35816 112024 0 0 816 4472 1093 1172 24 76 0
22 4 1 0 26944 36116 86020 0 0 220 13088 1238 4180 9 91 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
25 2 1 0 29376 36376 73588 0 0 48 31044 1193 580 9 91 0
25 2 1 0 34944 36688 67940 0 0 168 0 1124 3117 10 90 0
18 6 1 0 22840 36784 70340 0 0 124 13284 1084 72949 31 69 0
21 3 2 0 4528 36900 77184 0 0 188 24404 1151 7793 24 76 0
19 2 1 0 30000 36968 54476 0 0 180 21040 1207 1138 25 75 0
24 1 1 0 9520 37084 70756 0 0 264 300 1070 586 23 77 0
20 1 1 0 27568 37368 57128 0 0 28 0 1027 438 14 86 0
21 1 2 0 8176 37392 68784 0 0 0 26764 1124 71428 21 79 0
22 3 2 0 6192 37524 70680 0 0 40 18932 1094 3965 30 70 0
18 3 1 0 4656 37588 78172 0 0 0 16160 1089 676 16 84 0
21 1 0 0 20400 37280 92132 0 0 8 4748 1140 792 11 89 0
21 2 0 0 11312 37560 107024 0 0 4 0 1020 439 9 91 0
22 2 0 0 36672 37848 81452 0 0 0 0 1021 457 9 91 0
21 2 0 0 16064 38156 102892 0 0 0 0 1020 430 9 91 0
18 4 1 0 39616 38336 81084 0 0 16 31052 1084 463 20 80 0
16 3 2 0 6528 38520 96212 0 0 12 36000 1100 2175 25 75 0
20 3 1 0 21200 38616 88876 0 0 4 16384 1110 2012 28 72 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
15 5 2 0 31568 38756 78676 0 0 12 30340 1154 3259 26 74 0
13 6 1 0 4560 38776 99032 0 0 0 1920 1146 172960 33 67 0
20 4 3 0 11920 38844 93312 0 0 0 30220 1428 94676 23 77 0
14 5 1 0 21520 38956 83376 0 0 0 5140 1188 84888 23 77 0
14 6 3 0 14480 39020 98828 0 0 0 44408 1172 78004 31 69 0
15 8 2 0 51472 39188 56580 0 0 0 8132 1176 2303 26 74 0
15 4 3 0 25680 39284 76676 0 0 0 25996 1226 79590 27 73 0
16 3 2 0 28704 39228 58988 0 0 0 22280 1135 6413 23 77 0
9 11 2 0 4768 39068 97644 0 0 0 41280 1141 54086 29 71 0
21 1 1 0 22112 39028 80196 0 0 20 236 1161 38466 18 82 0
9 7 3 0 9888 39080 83768 0 0 12 30784 1178 722 25 75 0
16 3 1 0 29196 39120 83860 0 0 12 31632 1130 1328 22 78 0
13 7 2 0 20320 39324 86424 0 0 0 50508 1123 2721 17 83 0
16 2 2 0 16864 39300 80372 0 0 0 544 1161 1129 24 76 0
14 2 3 0 15328 39460 92580 0 0 0 39468 1112 1863 25 75 0
16 2 2 0 29792 39644 84504 0 0 0 24400 1147 14905 19 81 0
19 1 1 0 23648 39656 73584 0 0 24 248 1099 294663 32 68 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
16 5 1 0 19040 39668 78276 0 0 0 5280 1313 97372 34 66 0
17 1 3 0 5024 39780 85960 0 0 12 31896 1207 132553 20 80 0
15 3 3 0 60448 39500 47960 0 0 36 7028 1223 96263 27 73 0
15 2 2 0 64992 39664 60344 0 0 128 27892 1111 580 13 87 0
15 3 2 0 32928 39972 92040 0 0 56 12288 1088 421 9 91 0
16 3 2 0 15520 40240 108900 0 0 0 8192 1037 417 9 91 0
16 3 2 0 24608 40496 99724 0 0 0 12288 1064 409 8 92 0
15 4 4 0 8608 40784 114284 0 0 0 63464 1052 462 9 91 0
15 6 4 0 32800 40816 83792 0 0 12 1984 1089 1810 21 79 0
15 8 1 0 21280 40904 82156 0 0 4 23720 1211 3472 31 69 0
24 1 3 0 23648 41028 108648 0 0 4 37940 1102 1696 13 87 0
19 2 2 0 34080 41028 93700 0 0 12 8508 1167 1878 28 72 0
14 4 3 0 23200 40916 105792 0 0 0 34040 1207 3732 24 76 0
17 6 2 0 74952 40932 54212 0 0 40 448 1192 1874 30 70 0
22 2 3 0 20424 41148 102468 0 0 0 32464 1220 4147 13 87 0
19 3 3 0 27464 41296 90572 0 0 0 21544 1097 633 16 84 0
15 6 4 0 19968 41116 107444 0 0 12 25248 1209 3191 26 74 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
21 5 4 0 40768 41152 83128 0 0 0 30176 1109 1661 21 79 0
15 8 3 0 42048 41292 77732 0 0 0 25916 1224 1490 19 81 0
24 3 2 0 35712 41380 83744 0 0 0 14996 1187 2938 24 76 0
29 6 4 0 23440 41472 94456 0 0 0 44032 1130 711 18 82 0
26 8 2 0 12816 41672 88500 0 0 0 17888 1157 2706 21 79 0
32 1 2 0 34000 41708 58700 0 0 0 5168 1338 190579 22 78 0
13 4 6 0 44368 41772 70596 0 0 20 33148 1228 94089 27 73 0
19 0 6 0 24208 41940 96484 0 0 4 30668 1205 79296 15 85 0
23 7 3 0 40720 41984 70228 0 0 0 17752 1096 35684 29 71 0
26 3 3 0 4112 42084 113204 0 0 0 50428 1104 36990 20 80 0
19 6 2 0 6032 42092 127112 0 0 0 8104 1126 86790 24 76 0
23 5 1 0 16720 42164 100772 0 0 0 50696 1199 4121 23 77 0
19 6 4 0 11944 42204 105984 0 0 0 16608 1107 78384 24 76 0
15 9 2 0 15400 42176 93236 0 0 0 8216 1145 4285 21 79 0
20 4 2 0 27688 41388 92252 0 0 8 19224 1150 2285 26 74 0
13 11 2 0 25000 41424 96764 0 0 12 49792 1138 2378 26 74 0
13 4 3 0 36808 41572 85604 0 0 0 8156 1159 6269 19 81 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
14 4 3 0 46040 41696 71356 0 0 0 29004 1283 1317 16 84 0
16 4 2 0 29784 41800 81516 0 0 0 20744 1212 4003 20 80 0
14 7 2 0 16360 41912 94368 0 0 12 27096 1347 1280 20 80 0
15 5 1 0 20456 41512 86216 0 0 0 19936 1285 2452 34 66 0
14 6 2 0 22120 41692 92032 0 0 4 33492 1130 2019 28 72 0
15 3 2 0 7208 41904 90848 0 0 0 27200 1160 3969 19 81 0
22 4 2 0 15904 42008 75116 0 0 0 16 1167 2638 19 81 0
16 2 3 0 23064 42080 91744 0 0 164 61468 3191 6787 19 81 0
17 4 1 0 22360 41824 92348 0 0 16 18512 1121 9184 32 68 0
21 3 2 0 23192 41804 85752 0 0 16 11908 1401 676 12 88 0
18 6 2 0 37648 40528 81016 0 0 36 38528 1163 80175 16 84 0
12 7 4 0 14656 40616 88612 0 0 164 20460 1143 29431 31 69 0
19 4 2 0 24448 40688 86760 0 0 16 22964 1202 140777 34 66 0
15 5 2 0 15872 40708 104980 0 0 0 26640 1129 4195 23 77 0
19 0 1 0 10560 40780 115620 0 0 4 8080 1288 57755 29 71 0
16 5 4 0 23616 40848 91952 0 0 0 38020 1204 44670 12 88 0
13 8 1 0 20736 40932 93992 0 0 16 29872 1135 54569 26 74 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
19 5 2 0 35392 40972 83120 0 0 4 11364 1107 172946 24 76 0
14 8 2 0 14976 41092 89096 0 0 8 45232 1148 1527 21 79 0
13 7 2 0 31296 41128 64760 0 0 0 224 1099 123517 29 71 0
18 0 0 0 38144 40176 87580 0 0 0 18512 1184 39251 25 75 0
15 5 3 0 39872 40300 91356 0 0 12 19932 1252 523 13 87 0
11 8 3 0 41408 40372 91876 0 0 0 41144 1267 927 25 75 0
17 7 3 0 8448 40628 108448 0 0 20 58064 2531 4759 27 73 0
17 4 2 0 6016 40740 116184 0 0 0 37424 1144 10750 28 72 0
20 5 4 0 24768 39780 116784 0 0 0 9660 1157 111315 11 89 0
17 4 2 0 48128 39880 91060 0 0 4 13232 1113 137719 19 81
26 2 0 0 19848 48908 83352 0 0 4 248 1059 72823 17 83 0
26 2 0 0 20104 49176 82924 0 0 0 0 1008 399 9 91 0
27 2 1 0 22088 49464 80672 0 0 0 0 1008 429 9 91 0
28 2 1 0 19016 49776 83472 0 0 0 0 1007 412 9 91 0
27 4 2 0 4872 50080 98964 0 0 4 25540 1044 735 8 92 0
20 10 1 0 47088 50116 49712 0 0 20 21360 1141 3492 34 66 0
17 8 2 0 45104 50220 71860 0 0 4 44284 1211 3694 30 70 0
19 2 2 0 28232 50260 59928 0 0 0 19624 1135 2130 30 70 0
9 11 2 0 33544 50308 54580 0 0 0 8012 1123 4169 34 66 0
15 7 2 0 23624 50248 94468 0 0 36 30592 1091 974 17 83 0
18 5 4 0 13192 50388 98268 0 0 12 22600 1259 2185 25 75 0
24 5 3 0 19192 50548 92424 0 0 0 28660 1276 1870 22 78 0
20 6 1 0 31032 50688 71356 0 0 44 9340 1204 880 12 88 0
17 3 3 0 29496 50880 66688 0 0 136 20432 1209 995 18 82 0
19 3 2 0 12824 51040 91012 0 0 0 28116 1275 3671 22 78 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
13 4 2 0 35992 51156 61128 0 0 0 20900 1156 7625 27 73 0
15 4 3 0 25856 51264 71368 0 0 0 33208 1173 79498 20 80 0
21 4 2 0 21888 51308 84324 0 0 0 7412 1113 83040 32 68 0
20 2 2 0 29688 51428 66888 0 0 0 9384 1439 7754 27 73 0
18 4 2 0 39992 51640 47992 0 0 4 19496 1129 2264 22 78 0
19 0 3 0 59448 51732 65064 0 0 0 42396 1262 100721 19 81 0
16 7 3 0 37592 51780 62768 0 0 28 4300 1131 52379 31 69 0
22 1 3 0 43080 51972 56672 0 0 24 26444 1124 34785 25 75 0
20 3 1 0 21616 52160 68732 0 0 12 9448 1104 6441 22 78 0
20 4 3 0 15728 52292 74672 0 0 0 25112 1123 116217 19 81 0
18 6 2 0 16048 52292 74672 0 0 0 0 1145 282658 36 64 0
21 4 1 0 11696 52320 75800 0 0 0 10300 1072 82012 25 75 0
25 1 1 0 5360 52324 79404 0 0 0 12628 1071 504 15 85 0
26 1 1 0 15024 52552 70720 0 0 0 0 1008 448 11 89 0
26 1 1 0 21360 52832 64588 0 0 0 0 1007 8016 10 90 0
28 1 1 0 20720 52832 64588 0 0 0 0 1012 317182 25 75 0
25 1 0 0 19368 52992 69852 0 0 8 536 1022 152851 19 81 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
25 1 0 0 9640 53104 71684 0 0 0 21140 1041 70482 15 85 0
24 1 1 0 12200 53352 68944 0 0 0 0 1008 394 11 89 0
24 2 1 0 11176 53628 69668 0 0 0 0 1017 408 10 90 0
24 2 1 0 13480 53932 68156 0 0 0 0 1007 419 9 91 0
25 2 1 0 14760 54236 66588 0 0 0 0 1008 409 9 91 0
27 0 1 0 12392 54328 82308 0 0 0 15840 1071 76839 18 82 0
28 0 1 0 8296 54508 86196 0 0 0 0 1008 386 10 90 0
31 0 1 0 5480 54752 89492 0 0 12 0 1012 981 20 80 0
28 1 2 0 5112 55008 85752 0 0 0 18428 1021 442 10 90 0
16 3 1 0 24544 55140 78332 0 0 0 25680 1081 3376 21 79 0
17 4 2 0 7264 55220 95080 0 0 0 24552 1317 2637 10 90 0
17 5 2 0 15680 55328 86304 0 0 0 21676 1085 9903 15 85 0
19 2 2 0 4304 55416 83680 0 0 0 17856 1183 618 15 85 0
21 1 2 0 15760 55552 72904 0 0 56 22336 1258 16832 8 92 0
19 0 0 0 20832 55752 80040 0 0 8 1808 1078 1068 13 87 0
17 1 2 0 16480 55852 92856 0 0 0 49504 1130 2205 26 74 0
17 5 2 0 49696 55872 51592 0 0 132 2916 1107 1862 28 72 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
21 2 2 0 22176 56084 78348 0 0 48 13296 1276 3631 19 81 0
19 3 2 0 25632 56332 78016 0 0 0 25784 1321 1092 13 87 0
22 7 2 0 10672 56456 96848 0 0 172 31936 1091 2935 24 76 0
26 2 3 0 19552 56620 84204 0 0 0 38656 1119 1711 15 85 0
25 3 2 0 5080 56792 91392 0 0 4 19996 1120 1875 23 77 0
19 7 1 0 52440 56736 70180 0 0 68 24276 1123 3578 27 73 0
17 7 2 0 42456 56812 80208 0 0 0 21252 1108 4099 32 68 0
15 5 2 0 30000 56912 91712 0 0 4 62240 1114 2093 28 72 0
22 6 3 0 10512 56956 91248 0 0 4 10260 1132 7823 35 65 0
10 12 1 0 40976 57072 63072 0 0 0 880 1072 3468 17 83 0
13 9 3 0 4560 57192 97196 0 0 28 50812 1152 3494 21 79 0
18 4 3 0 16336 57108 85944 0 0 16 2292 1064 3716 26 74 0
14 6 1 0 38864 57232 54536 0 0 28 20964 1203 8334 16 84 0
14 7 1 0 35280 57232 67020 0 0 0 12284 1177 78927 26 74 0
20 5 1 0 27976 57304 74740 0 0 0 22540 1197 20679 18 82 0
18 7 2 0 4200 57344 91024 0 0 12 26268 1134 77804 28 72 0
15 9 3 0 11304 57472 96116 0 0 8 22124 1203 6235 19 81 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
19 3 4 0 17504 57584 88192 0 0 16 10816 1272 8953 24 76 0
22 0 2 0 13856 57628 82256 0 0 0 12968 1289 67453 24 76 0
19 3 3 0 18080 57792 91496 0 0 8 38848 1219 3310 14 86 0
19 3 3 0 11104 58072 98664 0 0 0 0 1085 27253 9 91 0
20 3 3 0 21152 58360 88356 0 0 12 0 1008 1499 9 91 0
22 3 3 0 24800 58668 84536 0 0 12 0 1020 1240 9 91 0
23 3 3 0 11312 58864 97852 0 0 0 0 1007 40361 15 85 0
22 3 3 0 16880 59184 92736 0 0 8 0 1045 1320 10 90 0
21 3 3 0 16432 59484 92652 0 0 8 8 1021 1397 11 89 0
22 5 3 0 13104 59788 95116 0 0 16 32 1013 1174 12 88 0
17 2 2 0 30720 59924 77332 0 0 0 27948 1072 5686 35 65 0
18 7 2 0 23536 60012 72036 0 0 0 13652 1145 2749 38 62 0
18 10 2 0 24560 59992 68520 0 0 12 15876 1202 3901 27 73 0
17 8 3 0 12272 60188 101412 0 0 68 36856 1159 5031 30 70 0
22 4 2 0 21680 60332 90520 0 0 536 60 1141 4503 29 71 0
21 4 1 0 37296 60616 66560 0 0 540 18392 1129 2009 13 87 0
19 4 4 0 8560 60832 95192 0 0 544 17040 1109 3007 19 81 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
13 7 2 0 27184 60944 80272 0 0 300 38872 1084 3685 32 68 0
19 6 2 0 9944 60856 115212 0 0 344 0 1103 55713 37 63 0
17 2 3 0 42008 60984 69364 0 0 320 13100 1115 26498 38 62 0
10 6 2 0 29272 61068 87140 0 0 4 39540 1169 58939 34 66 0
18 3 3 0 36528 61088 73704 0 0 168 71308 4503 534135 26 74 0
19 2 1 0 19120 61100 95712 0 0 0 26052 1132 94936 35 65 0
21 0 2 0 8432 61224 81972 0 0 0 19984 1161 80658 16 84 0
21 2 2 0 22064 61300 88936 0 0 0 39816 1205 27946 29 71 0
23 2 1 0 19696 61352 92256 0 0 4 26128 1162 104133 21 79 0
18 7 3 0 42992 61436 68640 0 0 36 17900 1169 16911 20 80 0
23 3 4 0 19632 61464 91880 0 0 0 19468 1200 78743 28 72 0
21 5 3 0 38064 61548 73992 0 0 0 21212 1187 2879 26 74 0
18 7 4 0 20336 61720 94672 0 0 0 31936 1131 2234 21 79 0
23 1 2 0 5520 61896 95788 0 0 0 22628 1106 6286 24 76 0
21 2 3 0 10320 61920 95476 0 0 0 13224 1275 2145 24 76 0
24 3 2 0 5584 61928 112888 0 0 0 40552 1259 960 10 90 0
26 0 1 0 24272 61972 90292 0 0 12 288 1084 1322 14 86 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
22 3 2 0 8400 62012 105412 0 0 8 21640 1217 162006 23 77 0
25 3 2 0 22224 62020 94496 0 0 4 19092 1179 2317 28 72 0
25 2 2 0 11920 62176 104368 0 0 0 20232 1371 1748 21 79 0
27 2 2 0 15632 62288 98600 0 0 12 13976 1083 2869 14 86 0
26 2 1 0 12624 62460 100176 0 0 0 120 1035 1367 19 81 0
29 5 2 0 26120 62500 80564 0 0 12 30564 1129 86068 29 71 0
17 4 2 0 36616 61976 83172 0 0 68 32420 1201 45433 22 78 0
19 2 2 0 21512 62064 91708 0 0 0 35992 1219 39327 23 77 0
19 0 1 0 19208 62116 92132 0 0 0 8208 1248 87596 25 75 0
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 4:13 ` Michael Frank
@ 2003-10-28 4:30 ` Nick Piggin
2003-10-28 6:11 ` Michael Frank
2003-10-28 17:42 ` Martin Josefsson
0 siblings, 2 replies; 19+ messages in thread
From: Nick Piggin @ 2003-10-28 4:30 UTC (permalink / raw)
To: Michael Frank; +Cc: cliff white, linux-kernel, Nigel Cunningham
Michael Frank wrote:
>On Tuesday 28 October 2003 07:50, Nick Piggin wrote:
>
>>cliff white wrote:
>>
>>
>>>On Tue, 28 Oct 2003 05:52:45 +0800
>>>Michael Frank <mhf@linuxmail.org> wrote:
>>>
>>>
>>>
>>>>To my surprise 2.6 - which used to do better then 2.4 - does no longer
>>>>handle these test that well.
>>>>
>>>>Generally, IDE IO throughput is _very_ uneven and IO _stops_ at times with the
>>>>system cpu load very high (and the disk LED off).
>>>>
>>>>IMHO the CPU scheduling is OK but the IO scheduling acts up here.
>>>>
>>>>The test system is a 2.4GHz P4 with 512M RAM and a 55MB/s udma IDE harddisk.
>>>>
>>>>The tests load the system to loadavg > 30. IO should be about 20MB/s on avg.
>>>>
>>>>Enclosed are vmstat -1 logs for 2.6-test9-Vanilla, followed by 2.6-test8-Vanilla
>>>>(-mm1 behaves similar), 2.4.22-Vanilla and 2.4.21+swsusp all compiled wo preempt.
>>>>
>>>>IO on 2.6 stops now for seconds at a time. -test8 is worse than -test9
>>>>
>>>>
>>>We see the same delta at OSDL. Try repeating your tests with 'elevator=deadline'
>>>to confirm.
>>>For example, on the 8-cpu platform:
>>>STP id Kernel Name MaxJPM Change Options
>>>281669 linux-2.6.0-test8 7014.42 0.0
>>>281671 linux-2.6.0-test8 8294.94 +18.26% elevator=deadline
>>>
>>>The -mm kernels don't show this big delta. We also do not see this delta on
>>>smaller machines
>>>
>>>
>>I'm working with Randy to fix this. Attached is what I have so far. See how
>>you go with it.
>>
>>
>>
>
>This has been done without running a kernel compile, by $ ti-tests/ti stat ub17 ddw 4 5000
>
>Seems to be more even on average but still drops IO too low and then gets overloaded.
>
>By "too low" I mean io bo less than 10000.
>
>By overloaded I mean io bo goes much above 40000. The disk can do maybe 60000.
>
>When io bo is much above 40000, cpu scheduling is being impaired as indicated
>by vmstat stopping output for a second or so...
>
The bi / bo fields in vmstat aren't exactly what the disk is doing, rather
the requests the queue is taking. The interesting thing is how your final
measurement compares with 2.4.
I think 2.4 has quite a lot more free requests by default (512 vs 128 for
2.6). This might also be causing Nigel's writeout problems perhaps. Try
echo 512 > /sys/block/xxx/queue/nr_requests
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 4:30 ` Nick Piggin
@ 2003-10-28 6:11 ` Michael Frank
2003-10-28 17:42 ` Martin Josefsson
1 sibling, 0 replies; 19+ messages in thread
From: Michael Frank @ 2003-10-28 6:11 UTC (permalink / raw)
To: Nick Piggin; +Cc: cliff white, linux-kernel, Nigel Cunningham
On Tuesday 28 October 2003 12:30, Nick Piggin wrote:
>
> Michael Frank wrote:
>
> >On Tuesday 28 October 2003 07:50, Nick Piggin wrote:
> >
> >>cliff white wrote:
> >>
> >>
> >>>On Tue, 28 Oct 2003 05:52:45 +0800
> >>>Michael Frank <mhf@linuxmail.org> wrote:
> >>>
> >>>
> >>>
> >>>>To my surprise 2.6 - which used to do better then 2.4 - does no longer
> >>>>handle these test that well.
> >>>>
> >>>>Generally, IDE IO throughput is _very_ uneven and IO _stops_ at times with the
> >>>>system cpu load very high (and the disk LED off).
> >>>>
> >>>>IMHO the CPU scheduling is OK but the IO scheduling acts up here.
> >>>>
> >>>>The test system is a 2.4GHz P4 with 512M RAM and a 55MB/s udma IDE harddisk.
> >>>>
> >>>>The tests load the system to loadavg > 30. IO should be about 20MB/s on avg.
> >>>>
> >>>>Enclosed are vmstat -1 logs for 2.6-test9-Vanilla, followed by 2.6-test8-Vanilla
> >>>>(-mm1 behaves similar), 2.4.22-Vanilla and 2.4.21+swsusp all compiled wo preempt.
> >>>>
> >>>>IO on 2.6 stops now for seconds at a time. -test8 is worse than -test9
> >>>>
> >>>>
> >>>We see the same delta at OSDL. Try repeating your tests with 'elevator=deadline'
> >>>to confirm.
> >>>For example, on the 8-cpu platform:
> >>>STP id Kernel Name MaxJPM Change Options
> >>>281669 linux-2.6.0-test8 7014.42 0.0
> >>>281671 linux-2.6.0-test8 8294.94 +18.26% elevator=deadline
> >>>
> >>>The -mm kernels don't show this big delta. We also do not see this delta on
> >>>smaller machines
> >>>
> >>>
> >>I'm working with Randy to fix this. Attached is what I have so far. See how
> >>you go with it.
> >>
> >>
> >>
> >
> >This has been done without running a kernel compile, by $ ti-tests/ti stat ub17 ddw 4 5000
> >
> >Seems to be more even on average but still drops IO too low and then gets overloaded.
> >
> >By "too low" I mean io bo less than 10000.
> >
> >By overloaded I mean io bo goes much above 40000. The disk can do maybe 60000.
> >
> >When io bo is much above 40000, cpu scheduling is being impaired as indicated
> >by vmstat stopping output for a second or so...
> >
>
> The bi / bo fields in vmstat aren't exactly what the disk is doing, rather
> the requests the queue is taking. The interesting thing is how your final
> measurement compares with 2.4.
Well this is one easy "measurement", an other being hooking up an oscilloscope
to IDE - might do that actually as well ;)
Anyway, what matters is the user side. IO intensive tasks are the most affected.
Running on $ ti stat ddw 4 5000, any one out of 4 ddd's gets locked out at times
for several seconds. This was a problem on 2.4 with 2.4.18 - 2.4.20 and is
fixed again since 2.4.21 while at that time the focus was more on general
interactivity with regard to desktop use.
Going to improve the time info output of the ddd loops to get better data and
provide a comparison of -test9 with 2.4.22 and 2.4.23-pre-latest within a
few days.
>
> I think 2.4 has quite a lot more free requests by default (512 vs 128 for
> 2.6). This might also be causing Nigel's writeout problems perhaps. Try
> echo 512 > /sys/block/xxx/queue/nr_requests
>
Superficially, the effect between 128 up to 2048 is insignificant.
The effect down to the minimum of 4 is noticable both in userspace
and with physical disk activity.
BTW, with the patch the hang reported earlier was not encountered again so
far, but this needs more testing. Is this an anticipated or side-effect
of the patch?
Regards
Michael
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 3:54 ` Nick Piggin
@ 2003-10-28 7:48 ` Michael Frank
0 siblings, 0 replies; 19+ messages in thread
From: Michael Frank @ 2003-10-28 7:48 UTC (permalink / raw)
To: Nick Piggin, Nigel Cunningham; +Cc: cliff white, Linux Kernel Mailing List
On Tuesday 28 October 2003 11:54, Nick Piggin wrote:
>
> Nigel Cunningham wrote:
>
> >As you rightly guessed, I was forgetting there are now 1000 jiffies per
> >second.
> >
> >With your patch applied, I can achieve something close to 2.4
> >performance, but only if I set the limit on the number of pages to
> >submit at one time quite high. If I set it to 3000, I can get 20107 4K
> >pages written in 5267 jiffies (14MB/s) and can read them back at resume
> >time (so cache is not a factor) in 4620 jiffies (16MB/s). In 2.4, I
> >normally set the limit on async commits to 100, and achieve the same
> >performance. 100 here makes it very jerky and much slower.
> >
> >Could there be some timeout value on BIOs that I might be able to
> >tweak/disable during suspend?
> >
>
> Try setting /sys/block/xxx/queue/iosched/antic_expire to 0 on your
> device under IO and see how it goes. That shouldn't be causing the
> problem though, especially as you are mostly writing I think?
>
> Otherwise might possibly be the queue plugging stuff, or maybe a
> regression in the disk driver.
>
Haven't done much of 2.6 swsusp testing due to the little diversion
with the scheduler, however I did notice one of those dreaded DMA
timeouts with the SIS chipset again.
Regards
Michael
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-27 23:50 ` Nick Piggin
2003-10-28 1:37 ` Nigel Cunningham
2003-10-28 4:13 ` Michael Frank
@ 2003-10-28 17:26 ` cliff white
2003-10-28 21:18 ` Dave Olien
2 siblings, 1 reply; 19+ messages in thread
From: cliff white @ 2003-10-28 17:26 UTC (permalink / raw)
To: Nick Piggin; +Cc: mhf, linux-kernel
On Tue, 28 Oct 2003 10:50:04 +1100
Nick Piggin <piggin@cyberone.com.au> wrote:
>
>
> cliff white wrote:
>
> >On Tue, 28 Oct 2003 05:52:45 +0800
> >Michael Frank <mhf@linuxmail.org> wrote:
> >
> >
> >>To my surprise 2.6 - which used to do better then 2.4 - does no longer
> >>handle these test that well.
> >STP id Kernel Name MaxJPM Change Options
> >281669 linux-2.6.0-test8 7014.42 0.0
> >281671 linux-2.6.0-test8 8294.94 +18.26% elevator=deadline
> >
> >The -mm kernels don't show this big delta. We also do not see this delta on
> >smaller machines
> >
>
> I'm working with Randy to fix this. Attached is what I have so far. See how
> you go with it.
So far, looks quite good. ( i know 2.4.18 is wierd for some, but they were
hot off the stp, so i used 'em )
4-cpu
STP id Kernel Name MaxJPM Change Options
282391 linux-2.4.18 5250.23 0.00
282413 Nick's patch 5484.95 4.27 elevator=deadline
282413 Nick's patch 5416.51 3.06
282395 linux-2.4.18 6581.17 0.0
282415 Nick's patch 8293.95 20.6
282415 Nick's patch 8484.95 22.43 elevator=deadline
And, the graph is nice and flat!
http://khack.osdl.org/stp/282416/results/jpm.png
Full results: http://developer.osdl.org/cliffw/reaim/index.html
Any test detail: http://khack.osdl.org/stp/<STP id>
cliffw
>
--
The church is near, but the road is icy.
The bar is far, but i will walk carefully. - Russian proverb
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 4:30 ` Nick Piggin
2003-10-28 6:11 ` Michael Frank
@ 2003-10-28 17:42 ` Martin Josefsson
1 sibling, 0 replies; 19+ messages in thread
From: Martin Josefsson @ 2003-10-28 17:42 UTC (permalink / raw)
To: Nick Piggin; +Cc: Michael Frank, cliff white, linux-kernel, Nigel Cunningham
[-- Attachment #1.1: Type: text/plain, Size: 2867 bytes --]
On Tue, 2003-10-28 at 05:30, Nick Piggin wrote:
> The bi / bo fields in vmstat aren't exactly what the disk is doing, rather
> the requests the queue is taking. The interesting thing is how your final
> measurement compares with 2.4.
I've attached a small script I made to investigate some weird I/O
problems I'm seeing with both AS and deadline. Maybe the script can be
of some help, who knows.
It produces vmstat-like one line per second output.
It takes the data from /proc/diskstats (output should be the same order
as in that file, it's described in Documentation/iostats.txt)
Example output (from one of my tests that showed the problem):
$diskstat.sh hde
row reads rmerge rsect rms writes wmerge wsect wms io ms ms2
1 0 0 0 0 0 0 0 0 163 0 0
2 3 0 24 46 751 530 10032 241626 135 1463 223510
3 1 0 64 15 590 402 8056 147398 149 1019 148466
4 1 0 160 16 599 593 9592 144204 156 1016 147806
5 1 0 8 21 522 485 7896 134072 133 1016 152418
6 1 0 8 14 541 265 6504 163212 139 1014 149101
7 3 0 264 63 495 153 5248 127999 147 1016 145554
.....
It's a sequential write, only one writer, lots of I/O-requests for a
very small number of sectors and the drive isn't very active.
It's a file recieved via network and written to disk.
Corresponding 'vmstat 1' output (not at the same time as the output
above):
24 procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
25 r b swpd free buff cache si so bi bo in cs us sy id wa
26 2 5 20400 2256 42956 297704 0 0 12 2300 4452 6010 4 25 0 71
27 3 4 20544 2524 42536 298276 0 144 16 3476 3605 4366 3 21 0 76
28 0 10 20544 2104 41876 299380 0 0 16 2996 4094 5107 4 22 0 74
29 1 10 20548 2456 40656 300252 0 4 40 3228 4389 5725 4 27 0 69
30 0 5 20832 2176 40108 301252 0 288 16 2780 4530 6889 5 26 0 69
Normally I see bursts of >30MB/s with a few seconds interval.
In this example there's some swapping but I havn't seen any indication
that swapping is the cause since it happens when there's no swapping
going on as well.
When this happens (not all the time) the machine freezes for 1-2 seconds
every 3-15 seconds. X freezes and if I ssh into the machine that freezes
as well at these intervals.
I've tried profiling when this happens but it just shows all time in
default_idle.
Maybe my problem is related, or maybe not...
--
/Martin
[-- Attachment #1.2: diskstat.sh --]
[-- Type: text/x-sh, Size: 881 bytes --]
#/bin/bash
num=0
for loop in a b c d e f g h i j k
do
old[$num]=0
num=$(($num + 1))
done
row=0
while true; do grep "$1 " /proc/diskstats | awk '{print $4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14}'; sleep 1; done | while read a b c d e f g h i j k
do
if [ $(($row % 30)) == 0 ]; then
#echo -e "row\treads\trmerge\trsect\trms\t\twrites\twmerge\twsect\twms\t\tio\tms\tms2"
echo -e "row\treads\trmerge\trsect\trms\twrites\twmerge\twsect\twms\tio\tms\tms2"
fi
row=$(($row + 1))
num=0
for loop in a b c d e f g h i j k
do
if [ ${old[$num]} == 0 ]; then
old[$num]=${!loop}
diff[$num]=0
else
diff[$num]=$((${!loop} - ${old[$num]}))
old[$num]=${!loop}
fi
num=$(($num + 1))
done
echo -e "$row\t${diff[0]}\t${diff[1]}\t${diff[2]}\t${diff[3]}\t${diff[4]}\t${diff[5]}\t${diff[6]}\t${diff[7]}\t$i\t${diff[9]}\t${diff[10]}"
done
[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 17:26 ` cliff white
@ 2003-10-28 21:18 ` Dave Olien
2003-10-29 2:40 ` Nick Piggin
0 siblings, 1 reply; 19+ messages in thread
From: Dave Olien @ 2003-10-28 21:18 UTC (permalink / raw)
To: cliff white; +Cc: Nick Piggin, mhf, linux-kernel
Yes, it seems Nick's latest patch from 10/27, has helped reaim
considerably.
The dbt2 workload still has a problem though. Mary ran this patch today,
with deadline and with as-iosched:
2.6.0-test9, elevator=deadline 1644 NOTPM
2.6.0-test9, unpatched as-iosched 977 NOTPM
2.6.0-test9, as-iosched with as-fix.patch 980 NOTPM
Higher NOTPM numbers are better.
On Tue, Oct 28, 2003 at 09:26:12AM -0800, cliff white wrote:
> > cliff white wrote:
>
> So far, looks quite good. ( i know 2.4.18 is wierd for some, but they were
> hot off the stp, so i used 'em )
>
> 4-cpu
> STP id Kernel Name MaxJPM Change Options
> 282391 linux-2.4.18 5250.23 0.00
> 282413 Nick's patch 5484.95 4.27 elevator=deadline
> 282413 Nick's patch 5416.51 3.06
>
> 282395 linux-2.4.18 6581.17 0.0
> 282415 Nick's patch 8293.95 20.6
> 282415 Nick's patch 8484.95 22.43 elevator=deadline
>
> And, the graph is nice and flat!
> http://khack.osdl.org/stp/282416/results/jpm.png
>
> Full results: http://developer.osdl.org/cliffw/reaim/index.html
>
> Any test detail: http://khack.osdl.org/stp/<STP id>
>
> cliffw
>
> >
>
>
> --
> The church is near, but the road is icy.
> The bar is far, but i will walk carefully. - Russian proverb
> -
> To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
> Please read the FAQ at http://www.tux.org/lkml/
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-28 21:18 ` Dave Olien
@ 2003-10-29 2:40 ` Nick Piggin
2003-10-29 4:56 ` Michael Frank
0 siblings, 1 reply; 19+ messages in thread
From: Nick Piggin @ 2003-10-29 2:40 UTC (permalink / raw)
To: Dave Olien; +Cc: cliff white, mhf, linux-kernel
Dave Olien wrote:
>Yes, it seems Nick's latest patch from 10/27, has helped reaim
>considerably.
>
>The dbt2 workload still has a problem though. Mary ran this patch today,
>with deadline and with as-iosched:
>
>2.6.0-test9, elevator=deadline 1644 NOTPM
>2.6.0-test9, unpatched as-iosched 977 NOTPM
>2.6.0-test9, as-iosched with as-fix.patch 980 NOTPM
>
>Higher NOTPM numbers are better.
>
OK, how does 2.6.0-test5 go on these tests?
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-29 2:40 ` Nick Piggin
@ 2003-10-29 4:56 ` Michael Frank
2003-10-29 16:47 ` Michael Frank
0 siblings, 1 reply; 19+ messages in thread
From: Michael Frank @ 2003-10-29 4:56 UTC (permalink / raw)
To: Nick Piggin, Dave Olien; +Cc: cliff white, linux-kernel
On Wednesday 29 October 2003 10:40, Nick Piggin wrote:
>
> Dave Olien wrote:
>
> >Yes, it seems Nick's latest patch from 10/27, has helped reaim
> >considerably.
> >
> >The dbt2 workload still has a problem though. Mary ran this patch today,
> >with deadline and with as-iosched:
> >
> >2.6.0-test9, elevator=deadline 1644 NOTPM
> >2.6.0-test9, unpatched as-iosched 977 NOTPM
> >2.6.0-test9, as-iosched with as-fix.patch 980 NOTPM
> >
> >Higher NOTPM numbers are better.
> >
>
> OK, how does 2.6.0-test5 go on these tests?
>
-test5 looks good with 4 dd loops 4K x 5000. The disk is busy as expected.
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
3 2 3 0 114488 11816 165872 0 0 0 39352 1116 1047 34 66 0
0 4 1 0 114616 11848 165872 0 0 0 932 1162 1495 49 51 0
0 4 2 0 114848 11852 165872 0 0 0 25300 1283 11355 59 41 0
0 4 1 0 115168 11860 165872 0 0 0 14364 1328 1286 79 21 0
1 4 2 0 114464 11912 165872 0 0 4 12196 1424 1488 38 62 0
0 4 1 0 113968 11912 150412 0 0 0 31460 1316 1308 72 28 0
0 4 2 0 93424 11936 165872 0 0 0 25148 1249 1374 61 39 0
6 2 3 0 118704 11936 140748 0 0 0 19972 1156 1061 82 18 0
4 4 3 0 102128 11960 157672 0 0 0 16324 1217 655 43 57 0
0 4 2 0 93568 11980 165872 0 0 0 35948 1230 1199 50 50 0
0 4 4 0 95680 12056 148552 0 0 4 28840 1148 1443 42 58 0
0 4 2 0 75456 12068 165872 0 0 0 23936 1145 1354 61 39 0
1 3 4 0 95488 12076 146204 0 0 0 30080 1197 1410 60 40 0
0 4 4 0 75904 12120 165872 0 0 0 23476 1171 1338 48 52 0
0 4 4 0 76032 12120 165872 0 0 0 9312 1259 1169 83 17 0
1 4 4 0 76160 12120 165872 0 0 0 9008 1282 1324 83 17 0
5 2 4 0 84608 12120 150592 0 0 0 17116 1357 892 42 58 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
4 4 4 0 69312 12152 165872 0 0 0 12016 1437 892 39 61 0
0 4 4 0 68608 12152 165872 0 0 0 45376 1302 1236 67 33 0
1 4 4 0 71936 12160 155532 0 0 0 21628 1127 1233 72 28 0
4 0 0 0 128000 12176 100984 0 0 0 68 1160 1790 67 33 0
4 0 0 0 98752 12200 129524 0 0 0 31108 1216 1493 53 47 0
2 2 1 0 114048 12236 111252 0 0 0 49072 1103 1540 44 56 0
6 1 1 0 103424 12264 122360 0 0 0 22928 1089 1078 39 61 0
0 4 4 0 58752 12288 165872 0 0 0 20816 1208 1014 42 58 0
0 4 4 0 65216 12288 157616 0 0 0 12664 1279 1199 83 17 0
0 4 4 0 70016 12288 151924 0 0 0 10912 1466 1535 78 22 0
2 4 4 0 54144 12320 165872 0 0 0 33336 1367 1505 39 61 0
0 4 1 0 63296 12344 156824 0 0 0 35460 1239 1142 34 66 0
0 4 1 0 68480 12344 146736 0 0 0 12992 1087 1336 79 21 0
0 4 1 0 48192 12356 165872 0 0 0 27260 1226 1303 63 38 0
4 3 1 0 48448 12404 165872 0 0 0 20204 1118 1224 41 59 0
0 4 2 0 47872 12444 165872 0 0 0 15636 1260 1036 29 71 0
0 4 1 0 47360 12444 165872 0 0 0 39544 1550 1137 73 27 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 4 2 0 66176 12444 131464 0 0 0 7744 1154 1235 83 17 0
0 4 2 0 15248 12468 158968 0 0 0 38432 1330 1354 42 58 0
5 0 1 0 21096 12500 153784 0 0 0 10516 1173 1391 66 34 0
0 4 2 0 8288 12524 165872 0 0 0 43688 1167 1205 56 44 0
6 2 1 0 15208 12524 158452 0 0 0 24712 1150 1052 77 23 0
5 4 2 0 20896 12544 153536 0 0 0 23460 1109 2717 24 76 0
3 1 3 0 9088 12568 148068 0 0 0 10012 1182 1287 45 55 0
0 4 3 0 4736 11600 152360 0 0 76 41456 1290 1353 59 41 0
2 2 3 0 4884 11632 152228 0 0 0 21648 1116 1253 61 39 0
0 4 3 0 5096 11656 152360 0 0 0 13516 1180 1511 61 39 0
1 3 3 0 5192 11656 152360 0 0 0 30244 1187 1303 58 42 0
0 4 3 0 18728 11656 134228 0 0 0 13896 1188 1331 79 21 0
0 4 3 0 4720 11464 148240 0 0 0 25356 1363 1452 58 42 0
9 0 1 0 44384 11504 109656 0 0 0 1168 1202 493 37 63 0
0 4 3 0 4336 11564 148240 0 0 0 21092 1150 1169 38 62 0
0 4 1 0 4264 11564 137308 0 0 0 25324 1372 1298 76 24 0
1 3 3 0 6832 9220 119764 0 0 4 22976 1333 1546 48 52 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 3 4 0 9904 9244 108500 0 0 12 45416 1280 1561 53 47 0
0 4 2 0 13360 8880 111916 0 0 0 22484 1171 1421 58 42 0
1 3 4 0 14704 8928 116296 0 0 0 6052 1103 1480 48 52 0
0 4 1 0 15972 8924 114916 0 0 0 59612 1182 763 52 48 0
1 4 2 0 4772 8948 126216 0 0 0 20096 1092 1297 59 41 0
7 1 2 0 11684 8964 119392 0 0 0 9436 1113 1314 33 67 0
0 4 3 0 5096 8984 126216 0 0 0 17428 1300 963 54 46 0
1 4 3 0 4776 8984 126216 0 0 0 19224 1219 1356 79 21 0
0 4 3 0 4972 8984 126216 0 0 0 14704 1288 1173 83 17 0
0 4 3 0 26868 9020 94336 0 0 0 20624 1277 1462 55 45 0
0 4 3 0 5296 9036 126216 0 0 0 9892 1487 1434 54 46 0
0 4 3 0 4976 9036 126216 0 0 0 32264 1266 1285 75 25 0
0 4 1 0 5360 9036 126216 0 0 0 17008 1186 1358 80 20 0
6 2 1 0 42800 9064 89960 0 0 0 14196 1267 559 35 65 0
0 4 1 0 26224 9092 106216 0 0 0 20112 1178 1187 56 44 0
0 4 2 0 45940 9128 86220 0 0 0 40196 1179 1425 42 58 0
0 4 4 0 9840 8908 118928 0 0 0 28412 1195 9048 36 64 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 4 3 0 4432 8940 126160 0 0 0 46816 1290 1413 53 47 0
1 4 4 0 4624 8944 126112 0 0 0 5276 1224 1175 83 17 0
8 0 4 0 5144 8752 126068 0 0 0 2352 1258 1171 42 58 0
2 4 4 0 4828 8752 126068 0 0 0 33200 1308 975 70 30 0
0 4 1 0 5212 8760 126068 0 0 0 24304 1270 775 72 28 0
1 4 4 0 25316 8840 105672 0 0 0 29824 1209 1467 35 65 0
1 4 5 0 4964 8848 126068 0 0 0 14264 1271 1438 57 43 0
1 4 5 0 4900 8848 126068 0 0 0 31776 1197 1246 73 27 0
1 4 1 0 4964 8856 126068 0 0 0 16076 1173 1328 83 17 0
9 1 5 0 4260 8892 126068 0 0 0 23992 1252 2280 37 63 0
2 4 6 0 5284 8916 126068 0 0 0 12236 1250 846 40 60 0
1 4 6 0 5412 8916 126068 0 0 0 26204 1403 1263 75 25 0
2 3 6 0 11876 8916 119992 0 0 0 18996 1308 1281 60 40 0
4 1 3 0 36132 8960 95756 0 0 0 22516 1194 1658 49 51 0
1 4 6 0 4900 8988 126068 0 0 0 22708 1266 1446 44 56 0
4 4 4 0 9380 8988 115648 0 0 0 21944 1337 1489 72 28 0
6 2 4 0 4840 8988 126088 0 0 20 22176 1259 997 75 25 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
1 4 1 0 5096 8992 126088 0 0 0 12632 1269 1905 90 10 0
0 4 1 0 23592 9000 107920 0 0 0 18484 1397 1605 74 26 0
1 3 3 0 5224 9072 126088 0 0 4 20816 1464 1603 39 61 0
Back to -test9
Also used only noop. Does not make much of a difference as i do not wish to claim it is better ;)
Request queue size above 128 does not make a significant difference either.
I'd say going down to 4 still works about as well in terms of total throughput.
And it is still bloody impossible to make the disk LED stay on.
I'll be back with "measurements" by weekend.
Regards
Michael
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: 2.6.0-test8/test9 io scheduler needs tuning?
2003-10-29 4:56 ` Michael Frank
@ 2003-10-29 16:47 ` Michael Frank
0 siblings, 0 replies; 19+ messages in thread
From: Michael Frank @ 2003-10-29 16:47 UTC (permalink / raw)
To: Nick Piggin, Dave Olien; +Cc: cliff white, linux-kernel, Martin Josefsson
[-- Attachment #1: Type: text/plain, Size: 909 bytes --]
OK, here is the updated ti-tests suite first.
Changes:
- dd loops show now Real, System, User times one per line.
R 0.17 S 0.07 U 0.00
- diskstat function kindly provided by Martin Josefsson
row reads rmerge rsect rms writes wmerge wsect wms io ms ms2
1681 14 0 160 5227 90 6050 48704 23501 0 1183 23375
- Updated README-TI file included in archive
- Minor bug fixes and cleanups and 0 command speed up
- sleep function to help witj placing xterms
- clean and distclean functions
Usage example - Run 4 dd write loops @ 5000 4K blocks, show diskstat, show status such as vmstat, ps-a, messages.
$ ti ddw 4 5000 sleep 3 diskstat hda sleep 3 stat
Improved usage example - add 10 unixbench tests
$ ti ddw 4 5000 sleep 3 diskstat hda sleep 3 stat sleep 3 ub10
Please refer to README-TI in archive.
Regards
Michael
[-- Attachment #2: ti-tests-1.0-beta-2.tar.bz2 --]
[-- Type: application/x-tbz, Size: 39712 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2003-10-29 16:48 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-10-27 21:52 2.6.0-test8/test9 io scheduler needs tuning? Michael Frank
2003-10-27 22:55 ` cliff white
2003-10-27 23:50 ` Nick Piggin
2003-10-28 1:37 ` Nigel Cunningham
2003-10-28 1:48 ` Nick Piggin
2003-10-28 2:58 ` Michael Frank
2003-10-28 2:57 ` Michael Frank
2003-10-28 3:31 ` Nigel Cunningham
2003-10-28 3:54 ` Nick Piggin
2003-10-28 7:48 ` Michael Frank
2003-10-28 4:13 ` Michael Frank
2003-10-28 4:30 ` Nick Piggin
2003-10-28 6:11 ` Michael Frank
2003-10-28 17:42 ` Martin Josefsson
2003-10-28 17:26 ` cliff white
2003-10-28 21:18 ` Dave Olien
2003-10-29 2:40 ` Nick Piggin
2003-10-29 4:56 ` Michael Frank
2003-10-29 16:47 ` Michael Frank
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).