* (no subject)
@ 2003-09-08 18:58 Mike Sullivan
2003-09-08 20:26 ` I/O degredation with AS on 2.6.0-test3 Dave Hansen
0 siblings, 1 reply; 5+ messages in thread
From: Mike Sullivan @ 2003-09-08 18:58 UTC (permalink / raw)
To: linux-kernel
In 2.6.0-test3 I am seeing a significant slowdown (~40%) during
heavy database workloads when using the default anticipatory
I/O scheduler. When the same kernel is booted with the
elevator=deadline argument the slowdown does not occur.
I've attached excerpts from the database test for vmstat and
iostat output as well as readprofiles for each of the schedulers.
Is anyone else seeing similiar behaviors?
Hardware: IBM x440 8way with hypertheading enabled, 16GB RAM,
4 QLA2310 FC adapters attached to 2 FastT900 controllers
(112 disks total).
1a. vmstat for anticipatory scheduler
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
14 41 0 3489576 3840 168280 0 0 25 2 70 30 1 2 96 0
4 103 0 3329760 3860 172160 0 0 233549 97 6606 21284 27 6 0 67
7 91 0 3327496 3864 172156 0 0 137106 14 7364 32957 20 5 0 75
2 84 0 3325416 3868 172412 0 0 138979 73 7096 30445 19 5 0 76
6 78 0 3324528 3868 173192 0 0 143213 444 7328 24901 13 4 0 83
0 105 0 3316408 3876 173964 0 0 142413 524 7531 26854 12 5 0 83
7 94 0 3313664 3880 174220 0 0 134939 821 7336 29037 14 5 0 82
1 110 0 3313560 3884 174216 0 0 114651 923 6990 27063 11 4 0 84
5 109 0 3313008 3884 174216 0 0 110115 862 6937 26381 11 4 0 85
3 111 0 3325400 3920 174180 0 0 108664 1030 6892 26721 11 4 0 85
1b. vmstat for deadline scheduler
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa
19 0 0 3518364 3104 119096 0 0 43 1 78 62 1 2 97 1
70 23 0 3361380 3776 170684 0 0 351035 538 3521 19435 68 17 9 5
96 25 0 3321060 3816 171684 0 0 588377 349 4973 25000 91 9 0 0
79 41 0 3316492 3824 171676 0 0 621683 831 5037 26853 91 9 0 0
86 48 0 3318684 3836 172964 0 0 609092 3073 4882 27878 91 9 0 0
71 44 0 3310164 3848 173212 0 0 551741 3746 4291 28492 92 8 0 0
91 35 0 3297708 3852 174248 0 0 508096 1425 4410 29533 91 8 0 1
2a. iostat for anticipatory scheduler
Linux 2.6.0-test3lp
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/sdj 1.92 0.00 363.48 0.88 18908.11 29.12 9454.06 14.56 51.98 4.80 13.14 1.22 44.57
/dev/sdi 1.97 0.00 364.65 0.61 18541.72 20.66 9270.86 10.33 50.82 4.86 13.27 1.23 44.75
/dev/sdh 1.95 0.00 361.35 0.85 18901.05 28.30 9450.53 14.15 52.26 4.79 13.20 1.23 44.72
/dev/sdg 1.66 0.00 357.14 0.84 18155.59 28.06 9077.80 14.03 50.79 4.74 13.23 1.25 44.61
/dev/sdf 3.20 0.00 328.57 0.87 17293.65 28.98 8646.82 14.49 52.58 4.73 14.34 1.36 44.74
/dev/sde 3.44 0.00 330.51 0.75 17180.30 18.55 8590.15 9.28 51.92 4.76 14.35 1.36 45.08
/dev/sdd 3.36 0.00 323.76 0.42 16911.58 14.56 8455.79 7.28 52.21 4.80 14.78 1.38 44.75
/dev/sdc 3.04 0.00 325.47 0.93 16665.03 31.01 8332.51 15.50 51.15 4.94 15.13 1.37 44.74
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/sdj 4.80 0.00 863.10 2.00 39945.60 64.00 19972.80 32.00 46.25 11.25 13.08 1.16 100.02
/dev/sdi 4.50 0.00 787.50 14.90 34438.40 483.20 17219.20 241.60 43.52 14.13 17.58 1.25 100.02
/dev/sdh 3.40 0.00 851.00 2.10 33180.80 70.40 16590.40 35.20 38.98 11.65 13.60 1.17 100.02
/dev/sdg 5.10 0.00 840.60 9.10 37772.80 288.00 18886.40 144.00 44.79 13.01 15.27 1.18 100.02
/dev/sdf 8.90 0.00 740.70 3.90 33075.20 128.00 16537.60 64.00 44.59 11.66 15.67 1.34 100.02
/dev/sde 8.30 0.00 760.40 2.80 36169.60 96.00 18084.80 48.00 47.52 11.70 15.23 1.31 100.02
/dev/sdd 12.40 0.00 769.70 0.00 30147.20 0.00 15073.60 0.00 39.17 11.05 14.29 1.30 100.02
/dev/sdc 9.20 0.00 746.00 3.00 31568.00 92.80 15784.00 46.40 42.27 11.65 15.59 1.34 100.03
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/sdj 5.01 0.00 895.80 0.00 38073.27 0.00 19036.64 0.00 42.50 11.06 12.29 1.12 100.12
/dev/sdi 5.91 0.00 818.02 12.61 32515.72 406.81 16257.86 203.40 39.64 13.84 16.71 1.21 100.12
/dev/sdh 7.41 0.00 862.16 1.80 34652.25 54.45 17326.13 27.23 40.17 11.46 13.31 1.16 100.12
/dev/sdg 4.40 0.00 866.67 2.20 33271.67 70.47 16635.84 35.24 38.37 11.67 13.43 1.15 100.13
/dev/sdf 8.01 0.00 784.28 2.80 33242.84 86.49 16621.42 43.24 42.35 11.52 14.59 1.27 100.13
/dev/sde 6.11 0.20 728.53 17.62 30488.09 576.58 15244.04 288.29 41.63 15.00 20.15 1.34 100.13
/dev/sdd 6.01 0.40 705.51 12.11 29690.49 413.21 14845.25 206.61 41.95 13.73 19.17 1.40 100.13
/dev/sdc 7.41 0.80 738.04 7.61 33338.94 281.88 16669.47 140.94 45.09 12.13 16.17 1.34 100.12
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/sdj 4.29 0.00 864.37 0.00 33411.58 0.00 16705.79 0.00 38.65 11.17 12.97 1.15 99.83
/dev/sdi 6.29 0.00 826.35 6.49 35002.00 198.00 17501.00 99.00 42.27 12.34 14.75 1.20 99.83
/dev/sdh 6.69 0.00 844.91 0.90 36563.67 28.74 18281.84 14.37 43.26 10.92 13.03 1.18 99.83
/dev/sdg 3.39 0.00 860.78 0.70 33705.39 22.36 16852.69 11.18 39.15 11.27 13.09 1.16 99.82
/dev/sdf 8.28 0.00 749.40 1.20 32661.08 38.32 16330.54 19.16 43.56 11.24 14.97 1.33 99.82
/dev/sde 5.79 0.00 731.44 13.47 34328.14 427.94 17164.07 213.97 46.66 13.86 18.67 1.34 99.82
/dev/sdd 4.59 0.00 713.77 17.37 31853.09 552.50 15926.55 276.25 44.32 14.61 20.09 1.37 99.82
/dev/sdc 11.88 0.00 699.00 16.07 30914.17 514.17 15457.09 257.09 43.95 14.46 20.26 1.40 99.82
2b. io for deadline
Linux 2.6.0-test3lp
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/sdj 0.04 0.00 103.39 1.02 30230.24 36.05 15115.12 18.02 289.87 0.68 6.51 1.60 16.72
/dev/sdi 0.01 0.00 64.93 0.82 18642.75 29.55 9321.38 14.77 283.99 0.50 7.57 1.90 12.49
/dev/sdh 0.02 0.00 109.26 1.02 30215.96 36.05 15107.98 18.02 274.30 0.70 6.37 1.55 17.14
/dev/sdg 0.03 0.00 62.97 0.21 18346.02 10.04 9173.01 5.02 290.50 0.48 7.53 1.92 12.11
/dev/sdf 0.02 0.00 98.78 0.92 27471.17 32.80 13735.58 16.40 275.87 0.75 7.47 1.64 16.37
/dev/sde 1.64 0.00 105.42 1.75 27301.07 41.84 13650.54 20.92 255.15 0.77 7.21 1.63 17.46
/dev/sdd 0.02 0.00 103.89 1.02 28228.90 36.05 14114.45 18.02 269.40 0.77 7.36 1.58 16.54
/dev/sdc 0.05 0.00 108.91 1.63 31749.13 55.56 15874.56 27.78 287.71 0.77 6.94 1.67 18.49
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/sdj 0.10 0.00 581.30 0.00 152032.00 0.00 76016.00 0.00 261.54 4.16 7.16 1.62 94.39
/dev/sdi 0.20 0.00 497.70 0.00 152601.60 0.00 76300.80 0.00 306.61 3.75 7.52 1.92 95.35
/dev/sdh 0.20 0.00 552.40 0.00 150166.40 0.00 75083.20 0.00 271.84 3.97 7.18 1.71 94.71
/dev/sdg 0.20 0.00 545.70 0.00 148086.40 0.00 74043.20 0.00 271.37 3.93 7.19 1.74 95.15
/dev/sdf 0.20 0.00 661.90 0.00 160950.40 0.00 80475.20 0.00 243.16 4.97 7.50 1.47 97.56
/dev/sde 0.80 0.00 725.50 0.00 148051.20 0.00 74025.60 0.00 204.07 5.39 7.42 1.31 95.00
/dev/sdd 0.90 0.00 688.60 0.00 158950.40 0.00 79475.20 0.00 230.83 5.12 7.43 1.40 96.40
/dev/sdc 0.70 0.00 644.30 13.80 147622.40 441.60 73811.20 220.80 224.99 4.99 7.58 1.41 92.52
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/sdj 0.00 0.00 448.10 0.00 132969.60 0.00 66484.80 0.00 296.74 3.03 6.77 1.88 84.40
/dev/sdi 0.00 0.00 514.20 0.00 141683.20 0.00 70841.60 0.00 275.54 3.53 6.88 1.80 92.70
/dev/sdh 0.20 0.00 537.90 0.00 145718.40 0.00 72859.20 0.00 270.90 3.93 7.31 1.71 91.92
/dev/sdg 0.60 0.00 764.00 0.00 149760.00 0.00 74880.00 0.00 196.02 5.25 6.87 1.30 99.41
/dev/sdf 0.70 0.00 803.10 0.00 138851.20 0.00 69425.60 0.00 172.89 5.94 7.40 1.20 96.11
/dev/sde 1.00 0.00 993.30 0.00 164857.60 0.00 82428.80 0.00 165.97 6.97 7.02 1.01 99.96
/dev/sdd 1.80 0.00 899.70 0.00 166409.60 0.00 83204.80 0.00 184.96 5.93 6.59 1.11 99.49
/dev/sdc 0.20 0.00 445.30 0.00 131558.40 0.00 65779.20 0.00 295.44 3.37 7.57 1.93 85.86
Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s avgrq-sz avgqu-sz await svctm %util
/dev/sdj 0.10 0.00 402.60 0.00 154393.60 0.00 77196.80 0.00 383.49 2.77 6.89 2.01 80.84
/dev/sdi 0.00 0.00 506.60 9.20 132547.20 294.40 66273.60 147.20 257.54 3.47 6.73 1.49 76.64
/dev/sdh 0.20 0.00 323.90 0.00 103331.20 0.00 51665.60 0.00 319.02 2.28 7.04 2.04 66.11
/dev/sdg 0.10 0.00 553.00 0.00 174540.80 0.00 87270.40 0.00 315.63 3.56 6.44 1.71 94.53
/dev/sdf 0.00 0.00 447.70 0.00 154179.20 0.00 77089.60 0.00 344.38 3.33 7.44 1.99 88.89
/dev/sde 0.20 0.00 426.20 4.60 155136.00 147.20 77568.00 73.60 360.45 3.21 7.47 2.06 88.80
/dev/sdd 0.10 0.00 448.90 0.00 159312.00 0.00 79656.00 0.00 354.89 3.28 7.30 1.98 88.70
/dev/sdc 0.00 0.00 410.50 0.00 146473.60 0.00 73236.80 0.00 356.82 3.04 7.41 2.16 88.65
3a. readprofile for anticipatory scheduler
1699600 total 0.8206
1609934 poll_idle 27757.4828
12233 schedule 10.1772
11380 do_softirq 54.9758
9795 __make_request 8.0485
5708 scsi_request_fn 8.3450
5282 scsi_end_request 28.2460
3482 qla2x00_start_scsi 3.4960
3466 dio_bio_end_io 28.8833
2279 dio_await_one 14.6090
2065 cpu_idle 32.2656
1706 qla2x00_queuecommand 1.1685
1647 add_timer 11.6809
1572 put_io_context 18.2791
1503 unlock_page 17.0795
1433 follow_hugetlb_page 3.0042
1292 sys_msgsnd 1.7969
1260 current_kernel_time 18.2609
1081 scsi_dispatch_cmd 2.8078
918 qla2x00_get_new_sp 8.2703
758 try_to_wake_up 1.3512
702 system_call 15.9545
691 del_timer_sync 5.4841
687 scsi_io_completion 0.6314
628 do_no_page 0.5004
589 blk_recount_segments 0.8951
577 blk_run_queues 3.6752
560 __end_that_request_first 1.0219
533 get_io_context 4.1318
506 scsi_device_unbusy 4.0480
501 set_page_dirty_lock 5.5055
493 get_request 0.7515
451 direct_io_worker 0.4275
450 do_direct_IO 0.4306
441 dio_bio_complete 1.6036
409 worker_thread 0.6059
406 submit_page_section 0.9442
386 kmem_cache_alloc 5.0130
358 bio_alloc 0.8689
344 add_disk_randomness 6.6154
339 kmem_cache_free 3.8966
319 dio_bio_add_page 3.0673
317 __generic_file_aio_read 0.5671
316 ipc_lock 4.3288
296 kfree 2.2256
281 add_timer_randomness 1.2165
268 as_work_handler 3.3924
236 fsync_buffers_list 0.6396
230 get_offset_cyclone 4.1818
223 huge_pte_offset 5.4390
213 qla2x00_next 0.2359
211 io_schedule_timeout 4.1373
208 scsi_put_command 1.4545
197 mempool_alloc 0.5988
186 blockdev_direct_IO 0.5360
180 finished_one_bio 1.5517
173 testmsg 2.4714
173 batch_entropy_store 1.1234
171 sys_msgrcv 0.1913
168 blk_run_queue 2.3014
166 mempool_free_slab 7.9048
159 dio_refill_pages 0.5179
157 generic_make_request 0.3601
149 bio_add_page 0.4357
140 dnotify_parent 1.3592
136 fget_light 1.0543
135 vfs_read 0.4576
122 do_page_fault 0.1083
118 update_atime 0.5315
116 get_jiffies_64 2.0000
113 sd_rw_intr 0.2770
111 scsi_run_queue 0.5663
105 .text.lock.util 4.3750
97 blkdev_get_blocks 0.8220
92 __copy_from_user_ll 0.7541
90 generic_file_direct_IO 0.5172
88 add_wait_queue 1.2571
85 submit_bio 0.5556
84 sys_semtimedop 0.0635
84 __copy_to_user_ll 0.7000
82 remove_wait_queue 1.1081
82 dio_bio_submit 0.8119
81 io_schedule 1.6875
78 do_gettimeofday 0.5417
78 dio_send_cur_page 0.4561
77 find_vma 0.9506
75 page_add_rmap 0.1974
75 memcpy 1.3158
75 bio_put 1.3889
74 page_waitqueue 1.5745
72 scsi_init_cmd_errh 0.3564
72 generic_file_read 0.4186
71 max_block 0.6174
70 blkdev_direct_IO 0.6140
66 sys_ipc 0.0965
65 proc_pid_stat 0.0543
64 kmap_atomic 0.3951
63 scsi_softirq 0.2864
63 del_timer 0.6000
62 default_idle 1.3778
60 mempool_free 0.4878
60 bio_destructor 0.6383
56 get_user_pages 0.0556
56 find_extend_vma 0.4444
55 task_mem 0.2148
55 __d_lookup 0.2000
52 number 0.0692
51 syscall_exit 4.6364
51 dio_await_completion 0.6220
50 bio_get_nr_vecs 0.9434
49 as_set_request 0.4375
48 sys_pread64 0.3780
48 dio_get_page 0.5517
45 find_get_page 0.5769
41 do_anonymous_page 0.0429
40 do_getitimer 0.1455
38 dio_new_bio 0.3016
38 __set_page_dirty_buffers 0.1279
36 mempool_alloc_slab 1.7143
35 sys_setitimer 0.1268
35 do_clock_nanosleep 0.0448
35 __kmalloc 0.2869
34 __copy_user_intel 0.1977
33 adjust_abs_time 0.1158
31 page_remove_rmap 0.0571
31 blkdev_writepage 0.8857
30 inode_times_differ 0.4412
30 get_more_blocks 0.1911
29 s_show 0.0458
28 scsi_decide_disposition 0.0875
28 link_path_walk 0.0133
27 sys_nanosleep 0.1034
26 show_regs 0.0741
25 wait_on_page_bit 0.1196
25 do_setitimer 0.0529
25 bio_endio 0.2049
23 vsnprintf 0.0201
23 queue_delayed_work 0.1655
23 ipcperms 0.1411
22 do_readv_writev 0.0316
21 zap_pte_range 0.0297
21 scsi_finish_command 0.1167
20 scsi_add_timer 0.1980
20 generic_file_aio_write_nolock 0.0068
19 update_queue 0.1652
19 dio_zero_block 0.1131
19 blk_queue_bounce 0.2468
18 syscall_call 1.6364
18 store_msg 0.0783
17 try_atomic_semop 0.0570
17 do_wp_page 0.0130
17 buffered_rmqueue 0.0489
16 sys_gettimeofday 0.0812
16 handle_mm_fault 0.0329
15 scsi_free_sgtable 0.3333
15 qla2x00_calc_iocbs_64 0.2542
15 free_hot_cold_page 0.0600
15 flush_tlb_mm 0.1136
15 elv_set_request 0.2586
14 kmap_atomic_to_page 0.0921
14 .text.lock.direct_io 0.1386
13 wake_up_process 0.3095
13 dio_bio_alloc 0.1429
13 default_llseek 0.0491
12 scsi_next_command 0.2927
12 pid_revalidate 0.1121
11 show_cpuinfo 0.0162
11 render_sigset_t 0.0598
11 dio_bio_reap 0.0965
11 __lookup 0.0539
10 proc_pid_status 0.0090
10 path_lookup 0.0299
10 get_pid_list 0.0855
9 free_msg 0.1957
9 clear_page_tables 0.0201
9 atomic_dec_and_lock 0.1385
9 __pagevec_lru_add_active 0.0354
8 local_bh_enable 0.0597
8 load_msg 0.0193
8 __wake_up 0.1231
7 vfs_write 0.0237
7 release_pages 0.0216
7 pte_alloc_one 0.0547
7 pte_alloc_map 0.0215
7 kernel_to_ipc64_perm 0.1296
7 ipc_unlock 0.8750
7 fput 0.3333
7 copy_process 0.0027
6 vfs_readv 0.0600
6 tstojiffie 0.0594
6 sock_alloc_send_pskb 0.0117
6 generic_file_readv 0.0385
6 end_that_request_chunk 1.2000
6 dio_complete 0.0800
5 write_profile 0.0276
5 sock_wfree 0.0658
5 radix_tree_lookup 0.0676
5 pgd_ctor 0.1316
5 page_address 0.0289
5 lru_cache_add_active 0.0602
5 get_empty_filp 0.0209
5 generic_file_write_nolock 0.0321
5 filemap_nopage 0.0054
5 bad_range 0.0442
4 zap_pmd_range 0.0342
4 vma_link 0.0244
4 unix_stream_recvmsg 0.0029
4 unix_stream_data_wait 0.0134
4 unix_release_sock 0.0060
4 unix_create1 0.0113
4 tg3_poll 0.0137
4 sync_mapping_buffers 0.0889
4 split_vma 0.0137
4 skb_recv_datagram 0.0169
4 qla2x00_blink_led 0.0214
4 pte_chain_alloc 0.0625
4 get_wchan 0.0328
4 generic_osync_inode 0.0156
4 follow_page 0.0095
4 follow_mount 0.0286
4 flush_scheduled_work 0.4000
4 elv_put_request 0.1212
4 copy_page_range 0.0030
4 __pte_chain_free 0.0455
4 __get_user_4 0.1739
4 __constant_c_and_count_memset 0.0312
4 .text.lock.time 0.1212
3 unix_stream_sendmsg 0.0030
3 tcp_v4_rcv 0.0013
3 tcp_sendmsg 0.0006
3 sys_llseek 0.0118
3 sys_fstat64 0.0526
3 sys_close 0.0265
3 strncpy_from_user 0.0303
3 sock_fasync 0.0052
3 scsi_init_io 0.0095
3 poll_freewait 0.0469
3 may_open 0.0068
3 kmap_high 0.0065
3 ipc_buildid 0.2500
3 generic_fillattr 0.0176
3 fget 0.0476
3 exit_notify 0.0016
3 eth_type_trans 0.0179
3 dput 0.0074
3 do_poll 0.0152
3 do_mmap_pgoff 0.0017
3 dio_cleanup 0.0195
3 bdev_read_only 0.0517
3 alloc_ldt 0.0071
3 access_process_vm 0.0065
3 __mark_inode_dirty 0.0123
3 __filemap_copy_from_user_iovec 0.0171
3 __alloc_pages 0.0038
2 wake_up_process_kick 0.0476
2 vsprintf 0.0465
2 vfs_writev 0.0200
2 unmap_region 0.0058
2 unix_write_space 0.0142
2 unix_poll 0.0129
2 unix_find_socket_byinode 0.0233
2 sys_stat64 0.0351
2 sys_readv 0.0202
2 sys_open 0.0144
2 sys_clock_nanosleep 0.0067
2 ss_wakeup 0.0426
2 sock_poll 0.0408
2 sock_map_fd 0.0061
2 set_page_address 0.0060
2 remove_suid 0.0182
2 remove_shared_vm_struct 0.0177
2 release_task 0.0049
2 raw_file_aio_write 0.0345
2 prepare_to_wait 0.0222
2 pid_alive 0.0571
2 normal_poll 0.0056
2 mmput 0.0120
2 iput 0.0161
2 ip_rcv 0.0017
2 get_write_access 0.0351
2 fn_hash_lookup 0.0085
2 eventpoll_init_file 0.0741
2 end_that_request_first 0.2000
2 do_sync_read 0.0109
2 do_aic7xxx_isr 0.0129
2 d_alloc 0.0044
2 cp_new_stat64 0.0075
2 copy_files 0.0030
2 change_protection 0.0035
2 cache_grow 0.0025
2 __block_prepare_write 0.0019
2 __blk_queue_bounce 0.0034
1 work_resched 0.0455
1 wait_for_packet 0.0031
1 vm_acct_memory 0.0185
1 vfs_permission 0.0031
1 vfs_follow_link 0.0022
1 unix_stream_connect 0.0010
1 unix_peer_get 0.0238
1 unix_accept 0.0033
1 tg3_rx 0.0011
1 tcp_transmit_skb 0.0007
1 tcp_poll 0.0026
1 sys_timer_settime 0.0012
1 sys_select 0.0007
1 sys_pwrite64 0.0079
1 sys_munmap 0.0098
1 sys_mprotect 0.0017
1 sys_getuid 0.0625
1 sys_getdents64 0.0048
1 strnlen_user 0.0147
1 sock_wmalloc 0.0106
1 sock_destroy_inode 0.0323
1 sock_def_readable 0.0079
1 sock_create 0.0027
1 skip_atoi 0.0154
1 skb_clone 0.0030
1 show_interrupts 0.0014
1 shm_get_stat 0.0064
1 set_huge_pte 0.0057
1 scsi_setup_cmd_retry 0.0093
1 schedule_timeout 0.0053
1 schedule_delayed_work 0.0714
1 restore_fpu 0.0323
1 recalc_bh_state 0.0076
1 rb_insert_color 0.0044
1 qla2x00_build_scsi_iocbs_32 0.0018
1 pty_write 0.0024
1 pty_chars_in_buffer 0.0137
1 profile_exit_task 0.0132
1 proc_lookup 0.0037
1 proc_calc_metrics 0.0123
1 prepare_to_wait_exclusive 0.0112
1 prep_new_page 0.0122
1 pid_fd_revalidate 0.0051
1 path_release 0.0159
1 parse_table 0.0034
1 page_cache_readahead 0.0026
1 old_mmap 0.0032
1 nr_running 0.0244
1 notify_change 0.0026
1 next_thread 0.0127
1 net_rx_action 0.0039
1 monotonic_clock_cyclone 0.0061
1 mm_init 0.0040
1 math_state_restore 0.0152
1 lru_add_drain 0.0119
1 locks_remove_posix 0.0036
1 load_elf_binary 0.0003
1 kobj_lookup 0.0034
1 kill_fasync 0.0169
1 kill_bdev 0.0147
1 ipc_checkid 0.0303
1 invalidate_inode_buffers 0.0111
1 inode_init_once 0.0027
1 inet_sendmsg 0.0112
1 in_group_p 0.0222
1 i_waitq_head 0.0455
1 handle_signal 0.0032
1 get_unused_fd 0.0025
1 get_signal_to_deliver 0.0012
1 generic_file_open 0.0111
1 flush_tlb_page 0.0072
1 find_lock_page 0.0036
1 filp_close 0.0083
1 filldir64 0.0032
1 fd_install 0.0179
1 ext2_update_inode 0.0013
1 ext2_get_inode 0.0033
1 ext2_alloc_block 0.0050
1 dup_task_struct 0.0050
1 do_sys_settimeofday 0.0051
1 do_select 0.0014
1 do_page_cache_readahead 0.0027
1 do_flush_tlb_all 0.0108
1 disk_round_stats 0.0047
1 device_not_available 0.0244
1 dev_seq_printf_stats 0.0044
1 detach_vmas_to_be_unmapped 0.0101
1 destroy_context 0.0058
1 dentry_open 0.0027
1 dcache_readdir 0.0021
1 d_rehash 0.0111
1 d_lookup 0.0125
1 d_instantiate 0.0111
1 csum_partial 0.0035
1 create_elf_tables 0.0011
1 count_semncnt 0.0091
1 copy_strings 0.0016
1 copy_mm 0.0011
1 check_unthrottle 0.0164
1 can_vma_merge_before 0.0119
1 cache_init_objs 0.0085
1 c_start 0.0278
1 bd_claim 0.0172
1 batch_entropy_process 0.0047
1 alloc_skb 0.0045
1 alloc_inode 0.0030
1 add_softcursor 0.0042
1 __vma_link 0.0063
1 __unix_remove_socket 0.0084
1 __tcp_select_window 0.0040
1 __posix_lock_file 0.0007
1 __kfree_skb 0.0046
1 __fput 0.0050
1 __copy_user_zeroing_intel 0.0058
1 __block_commit_write 0.0068
1 __alloc_percpu 0.0047
1 SHATransform 0.0033
1 .text.lock.tty_io 0.0023
1 .text.lock.read_write 0.0192
1 .text.lock.open 0.0058
1 .text.lock.dnotify 0.0233
3b. readprofile for deadline scheduler
171653 total 0.0829
19968 do_softirq 96.4638
18055 schedule 15.0208
14925 scsi_request_fn 21.8202
11718 qla2x00_start_scsi 11.7651
9323 follow_hugetlb_page 19.5451
5626 __make_request 4.6228
4567 sys_msgsnd 6.3519
4108 poll_idle 70.8276
4070 dio_bio_end_io 33.9167
4044 unlock_page 45.9545
3882 do_direct_IO 3.7148
3690 scsi_end_request 19.7326
3577 submit_page_section 8.3186
3146 set_page_dirty_lock 34.5714
2502 dio_bio_complete 9.0982
2313 qla2x00_queuecommand 1.5842
2298 dio_bio_add_page 22.0962
2230 system_call 50.6818
2163 del_timer_sync 17.1667
1905 try_to_wake_up 3.3957
1782 get_offset_cyclone 32.4000
1721 fsync_buffers_list 4.6640
1677 get_request 2.5564
1658 bio_add_page 4.8480
1394 add_timer 9.8865
1181 dio_await_one 7.5705
1159 dio_refill_pages 3.7752
1121 scsi_dispatch_cmd 2.9117
1056 current_kernel_time 15.3043
939 bio_alloc 2.2791
884 qla2x00_get_new_sp 7.9640
863 direct_io_worker 0.8180
858 ipc_lock 11.7534
812 do_no_page 0.6470
802 put_io_context 9.3256
798 blkdev_get_blocks 6.7627
765 .text.lock.util 31.8750
751 max_block 6.5304
717 __generic_file_aio_read 1.2826
691 scsi_io_completion 0.6351
683 kmem_cache_alloc 8.8701
677 scsi_device_unbusy 5.4160
658 kfree 4.9474
644 sys_msgrcv 0.7204
624 kmem_cache_free 7.1724
610 get_jiffies_64 10.5172
582 blk_run_queue 7.9726
581 __end_that_request_first 1.0602
563 page_add_rmap 1.4816
546 find_vma 6.7407
541 dio_send_cur_page 3.1637
517 scsi_put_command 3.6154
502 generic_unplug_device 4.7810
483 blk_run_queues 3.0764
471 testmsg 6.7286
471 page_waitqueue 10.0213
461 blk_recount_segments 0.7006
455 __copy_from_user_ll 3.7295
433 default_llseek 1.6340
405 get_user_pages 0.4018
379 huge_pte_offset 9.2439
360 fget_light 2.7907
348 copy_page_range 0.2632
337 add_disk_randomness 6.4808
301 get_io_context 2.3333
301 do_readv_writev 0.4325
301 blockdev_direct_IO 0.8674
289 do_page_fault 0.2564
280 __copy_to_user_ll 2.3333
273 sd_rw_intr 0.6691
266 add_timer_randomness 1.1515
258 qla2x00_next 0.2857
254 mempool_alloc 0.7720
239 generic_make_request 0.5482
236 update_atime 1.0631
235 batch_entropy_store 1.5260
231 __kmalloc 1.8934
227 do_gettimeofday 1.5764
223 sys_semtimedop 0.1687
219 do_wp_page 0.1681
211 get_more_blocks 1.3439
208 sys_ipc 0.3041
198 io_schedule 4.1250
186 __set_page_dirty_buffers 0.6263
183 finished_one_bio 1.5776
183 del_timer 1.7429
182 kmap_atomic 1.1235
181 find_extend_vma 1.4365
180 wait_on_page_bit 0.8612
180 generic_file_direct_IO 1.0345
180 dio_get_page 2.0690
179 dnotify_parent 1.7379
178 io_schedule_timeout 3.4902
172 generic_file_readv 1.1026
158 blkdev_direct_IO 1.3860
152 submit_bio 0.9935
146 sys_llseek 0.5748
144 do_anonymous_page 0.1506
137 scsi_run_queue 0.6990
137 ipcperms 0.8405
134 bio_destructor 1.4255
126 do_getitimer 0.4582
120 find_get_page 1.5385
118 mempool_free 0.9593
116 syscall_exit 10.5455
113 do_clock_nanosleep 0.1445
112 do_setitimer 0.2368
108 scsi_softirq 0.4909
108 deadline_set_request 1.1020
102 store_msg 0.4435
102 scsi_init_cmd_errh 0.5050
101 sys_setitimer 0.3659
97 proc_pid_stat 0.0810
97 dio_bio_submit 0.9604
95 page_remove_rmap 0.1750
92 bio_put 1.7037
92 __d_lookup 0.3345
88 dio_await_completion 1.0732
85 memcpy 1.4912
80 sys_gettimeofday 0.4061
79 wake_up_process 1.8810
76 blk_queue_bounce 0.9870
73 vfs_read 0.2475
72 pte_alloc_one 0.5625
69 load_msg 0.1663
68 task_mem 0.2656
68 scsi_decide_disposition 0.2125
65 bio_get_nr_vecs 1.2264
64 number 0.0852
63 mempool_alloc_slab 3.0000
61 zap_pte_range 0.0863
60 handle_mm_fault 0.1235
60 dio_new_bio 0.4762
58 adjust_abs_time 0.2035
57 .text.lock.direct_io 0.5644
56 sys_readv 0.5657
54 scsi_finish_command 0.3000
53 blkdev_writepage 1.5143
52 huge_page_release 1.2381
50 vsnprintf 0.0436
49 scsi_next_command 1.1951
48 try_atomic_semop 0.1611
46 syscall_call 4.1818
46 scsi_add_timer 0.4554
44 vfs_readv 0.4400
44 inode_times_differ 0.6471
43 sys_select 0.0318
42 sys_nanosleep 0.1609
42 elv_set_request 0.7241
41 find_lock_page 0.1475
40 generic_file_read 0.2326
39 __copy_user_intel 0.2267
37 buffered_rmqueue 0.1063
36 update_queue 0.3130
36 sys_pread64 0.2835
36 radix_tree_lookup 0.4865
36 link_path_walk 0.0171
36 do_select 0.0495
35 work_resched 1.5909
35 qla2x00_calc_iocbs_64 0.5932
35 device_not_available 0.8537
33 s_show 0.0521
32 dio_bio_alloc 0.3516
30 pte_chain_alloc 0.4688
28 free_msg 0.6087
27 kmap_atomic_to_page 0.1776
27 bio_endio 0.2213
26 free_hot_cold_page 0.1040
26 __lookup 0.1275
24 release_pages 0.0741
24 pte_alloc_map 0.0738
24 flush_tlb_mm 0.1818
24 __pte_chain_free 0.2727
23 mempool_free_slab 1.0952
23 generic_file_aio_write_nolock 0.0078
23 filemap_nopage 0.0249
23 dio_zero_block 0.1369
22 ipc_unlock 2.7500
22 dio_complete 0.2933
21 tstojiffie 0.2079
20 fget 0.3175
20 clear_page_tables 0.0446
20 __alloc_pages 0.0255
19 set_huge_pte 0.1092
18 dio_bio_reap 0.1579
18 bad_range 0.1593
18 __wake_up 0.2769
17 shmem_getpage 0.0062
17 fput 0.8095
17 find_vma_prev 0.2099
16 local_bh_enable 0.1194
16 kstat_read_proc 0.0175
16 end_that_request_chunk 3.2000
16 copy_files 0.0237
15 queue_delayed_work 0.1079
15 pid_revalidate 0.1402
15 may_open 0.0339
14 tcp_poll 0.0365
14 scsi_free_sgtable 0.3111
14 render_sigset_t 0.0761
14 cache_grow 0.0175
14 __pagevec_lru_add_active 0.0551
13 shmem_nopage 0.1204
13 math_state_restore 0.1970
13 copy_mm 0.0140
13 .text.lock.time 0.3939
12 pid_alive 0.3429
12 hugetlb_prefault 0.0327
12 atomic_dec_and_lock 0.1846
11 get_unused_fd 0.0279
11 get_pid_list 0.0940
11 generic_file_write_nolock 0.0705
11 cpu_idle 0.1719
11 __get_user_4 0.4783
10 show_interrupts 0.0136
10 remove_shared_vm_struct 0.0885
10 ipc_checkid 0.3030
10 end_that_request_first 1.0000
9 vfs_write 0.0305
9 sock_poll 0.1837
9 restore_fpu 0.2903
9 qla2x00_build_scsi_iocbs_32 0.0161
9 generic_osync_inode 0.0350
9 d_alloc 0.0199
9 __posix_lock_file 0.0060
9 __mark_inode_dirty 0.0370
8 sys_close 0.0708
8 strncpy_from_user 0.0808
8 qla2x00_blink_led 0.0428
8 proc_pid_status 0.0072
8 kernel_to_ipc64_perm 0.1481
8 ipc_parse_version 0.2759
8 flush_scheduled_work 0.8000
7 unmap_hugepage_range 0.0281
7 schedule_delayed_work 0.5000
7 poll_freewait 0.1094
7 nr_blockdev_pages 0.0700
7 free_pages 0.0909
7 do_sys_settimeofday 0.0354
7 copy_process 0.0027
7 bdev_read_only 0.1207
7 __alloc_percpu 0.0327
6 write_profile 0.0331
6 vfs_writev 0.0600
6 unix_write_space 0.0426
6 sys_shmat 0.0083
6 ss_wakeup 0.1277
6 pgd_ctor 0.1579
6 mark_page_accessed 0.1154
6 lru_cache_add_active 0.0723
6 ipc_buildid 0.5000
6 flush_tlb_page 0.0432
6 do_sigaction 0.0095
6 do_mmap_pgoff 0.0034
6 dio_cleanup 0.0390
6 cp_new_stat64 0.0226
5 unmap_vmas 0.0069
5 unix_stream_recvmsg 0.0036
5 sys_open 0.0360
5 schedule_timeout 0.0267
5 path_lookup 0.0150
5 locks_remove_posix 0.0182
5 do_aic7xxx_isr 0.0323
5 chrdev_open 0.0121
5 access_process_vm 0.0108
5 __pollwait 0.0254
5 __free_pages 0.0694
4 unix_stream_data_wait 0.0134
4 unix_release_sock 0.0060
4 tg3_rx 0.0043
4 show_cpuinfo 0.0059
4 setup_sigcontext 0.0135
4 setup_rt_frame 0.0059
4 remove_wait_queue 0.0541
4 proc_lookup 0.0148
4 prep_new_page 0.0488
4 page_cache_readahead 0.0105
4 notify_change 0.0102
4 new_inode 0.0345
4 max_select_fd 0.0177
4 kill_bdev 0.0588
4 ip_route_input_slow 0.0018
4 get_empty_filp 0.0167
4 eventpoll_init_file 0.1481
4 dput 0.0099
4 do_generic_mapping_read 0.0031
4 alloc_skb 0.0179
4 __constant_c_and_count_memset 0.0312
3 wake_up_process_kick 0.0714
3 unix_stream_sendmsg 0.0030
3 unix_poll 0.0194
3 tg3_poll 0.0103
3 sys_pwrite64 0.0236
3 split_vma 0.0103
3 sock_map_fd 0.0091
3 sock_alloc_send_pskb 0.0059
3 sigprocmask 0.0142
3 scsi_init_io 0.0095
3 page_address 0.0173
3 open_namei 0.0029
3 lru_add_drain 0.0357
3 kmap_high 0.0065
3 hugetlbfs_statfs 0.0370
3 generic_fillattr 0.0176
3 follow_mount 0.0214
3 find_vma_prepare 0.0297
3 file_move 0.0526
3 file_kill 0.0682
3 fd_install 0.0536
3 exit_notify 0.0016
3 dup_task_struct 0.0149
3 do_settimeofday 0.0099
3 do_munmap 0.0071
3 disk_stats_read 0.0041
3 count_semncnt 0.0273
3 calc_crc_errors 0.0155
3 alloc_inode 0.0090
3 __get_free_pages 0.0303
3 __fput 0.0151
3 __copy_user_zeroing_intel 0.0174
3 __blk_queue_bounce 0.0052
3 .text.lock.read_write 0.0577
2 wait_for_packet 0.0062
2 vma_link 0.0122
2 vm_acct_memory 0.0370
2 unmap_region 0.0058
2 unix_accept 0.0066
2 task_prio 0.1429
2 sys_fstat64 0.0351
2 sys_connect 0.0113
2 sys_brk 0.0073
2 sys_alarm 0.0241
2 sys_accept 0.0063
2 sync_mapping_buffers 0.0444
2 sock_def_readable 0.0157
2 skb_recv_datagram 0.0084
2 shmem_swp_alloc 0.0032
2 set_page_address 0.0060
2 seq_read 0.0027
2 schedule_tail 0.0130
2 proc_pident_lookup 0.0037
2 old_mmap 0.0063
2 nr_free_pages 0.0290
2 memcpy_fromiovec 0.0118
2 lookup_mnt 0.0192
2 locks_remove_flock 0.0103
2 load_elf_binary 0.0007
2 kunmap_atomic 0.1667
2 kobj_lookup 0.0069
2 kfree_skbmem 0.0455
2 invalidate_vcache 0.0139
2 init_new_context 0.0085
2 get_signal_to_deliver 0.0024
2 free_pages_and_swap_cache 0.0153
2 fn_hash_lookup 0.0085
2 filp_close 0.0165
2 file_read_actor 0.0085
2 ext2_get_inode 0.0066
2 eligible_child 0.0114
2 elf_map 0.0107
2 do_sync_read 0.0109
2 do_lookup 0.0124
2 do_irq_balance 0.0019
2 dnotify_flush 0.0128
2 dentry_open 0.0053
2 create_elf_tables 0.0022
2 copy_strings 0.0031
2 convert_fxsr_to_user 0.0057
2 change_protection 0.0035
2 bounce_copy_vec 0.0175
2 alloc_ldt 0.0047
2 add_wait_queue 0.0286
2 __user_walk 0.0213
2 __bio_map_user 0.0030
1 yield 0.0455
1 wake_up_inode 0.0156
1 vsscanf 0.0008
1 vgacon_cursor 0.0024
1 vfs_stat 0.0110
1 vfs_permission 0.0031
1 vfs_fstat 0.0125
1 unix_stream_connect 0.0010
1 unix_sock_destructor 0.0036
1 unix_peer_get 0.0238
1 uart_wait_until_sent 0.0045
1 sys_write 0.0101
1 sys_wait4 0.0016
1 sys_socketcall 0.0015
1 sys_shmdt 0.0030
1 sys_rt_sigreturn 0.0032
1 sys_read 0.0101
1 sys_mmap2 0.0059
1 sys_ioctl 0.0016
1 sys_dup2 0.0044
1 sys_clock_nanosleep 0.0034
1 sys_access 0.0030
1 sync_sb_inodes 0.0017
1 supplemental_group_member 0.0204
1 submit_bh 0.0029
1 sockfd_lookup 0.0084
1 sock_wfree 0.0132
1 sock_init_data 0.0030
1 sock_destroy_inode 0.0323
1 sock_create 0.0027
1 sock_alloc_send_skb 0.0196
1 sock_alloc 0.0106
1 skip_atoi 0.0154
1 skb_release_data 0.0045
1 skb_free_datagram 0.0256
1 shm_open 0.0111
1 shm_get_stat 0.0064
1 shm_close 0.0063
1 serial_in 0.0156
1 seq_printf 0.0116
1 sched_migrate_task 0.0127
1 save_i387_fxsave 0.0050
1 save_i387 0.0055
1 s_start 0.0024
1 run_timer_softirq 0.0026
1 restore_i387 0.0079
1 reserve_blocks 0.0055
1 release_task 0.0024
1 rb_insert_color 0.0044
1 rb_erase 0.0044
1 raw_release 0.0069
1 raw_open 0.0033
1 proc_readdir 0.0022
1 proc_info_read 0.0029
1 proc_destroy_inode 0.0323
1 prepare_binprm 0.0044
1 prep_compound_page 0.0185
1 posix_locks_conflict 0.0058
1 pipe_read 0.0017
1 path_release 0.0159
1 netif_receive_skb 0.0022
1 mpage_writepages 0.0014
1 move_addr_to_kernel 0.0069
1 monotonic_clock_cyclone 0.0061
1 mmput 0.0060
1 mm_init 0.0040
1 meminfo_read_proc 0.0020
1 locks_insert_lock 0.0143
1 locks_conflict 0.0323
1 load_elf_interp 0.0017
1 lease_alloc 0.0044
1 kmap 0.0455
1 ipc_addid 0.0058
1 inode_update_time 0.0059
1 huge_pte_alloc 0.0244
1 hash_vcache 0.0400
1 handle_signal 0.0032
1 handle_IRQ_event 0.0100
1 get_zeroed_page 0.0082
1 get_wchan 0.0082
1 get_unmapped_area 0.0029
1 generic_forget_inode 0.0033
1 follow_page 0.0024
1 flush_signal_handlers 0.0149
1 flush_old_exec 0.0005
1 flush_all_zero_pkmaps 0.0056
1 finish_wait 0.0111
1 find_task_by_pid 0.0130
1 find_group_orlov 0.0012
1 find_busiest_node 0.0015
1 filldir64 0.0032
1 file_ra_state_init 0.0312
1 fcntl_setlk 0.0014
1 fasync_helper 0.0043
1 ext2_new_block 0.0007
1 ext2_free_blocks 0.0014
1 ext2_find_entry 0.0015
1 ext2_block_to_path 0.0031
1 ext2_alloc_branch 0.0019
1 ext2_add_link 0.0009
1 exit_thread 0.0185
1 error_code 0.0179
1 end_buffer_async_write 0.0051
1 elv_put_request 0.0303
1 do_sync_write 0.0055
1 do_signal 0.0038
1 do_sigaltstack 0.0028
1 do_rw_proc 0.0062
1 do_pollfd 0.0069
1 do_poll 0.0051
1 destroy_inode 0.0125
1 destroy_context 0.0058
1 deny_write_access 0.0159
1 default_idle 0.0222
1 dcache_readdir 0.0021
1 d_rehash 0.0111
1 copy_siginfo_to_user 0.0025
1 con_write_room 0.0370
1 complete 0.0123
1 cap_vm_enough_memory 0.0053
1 cap_task_post_setuid 0.0034
1 can_vma_merge_after 0.0109
1 can_share_swap_page 0.0097
1 call_rcu 0.0192
1 batch_entropy_process 0.0047
1 assign_type 0.0417
1 alloc_buffer_head 0.0135
1 __vma_link 0.0063
1 __unix_insert_socket 0.0109
1 __rb_erase_color 0.0024
1 __pagevec_lru_add 0.0048
1 __kfree_skb 0.0046
1 __find_get_block_slow 0.0029
1 __filemap_copy_from_user_iovec 0.0057
1 SHATransform 0.0033
1 .text.lock.swap 0.0103
1 .text.lock.file_table 0.0092
Mike Sullivan
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: I/O degredation with AS on 2.6.0-test3
2003-09-08 18:58 Mike Sullivan
@ 2003-09-08 20:26 ` Dave Hansen
2003-09-09 2:44 ` Nick Piggin
0 siblings, 1 reply; 5+ messages in thread
From: Dave Hansen @ 2003-09-08 20:26 UTC (permalink / raw)
To: Mike Sullivan; +Cc: Linux Kernel Mailing List
You might want to try Martin Bligh's diffprofile utility. It's a bit
hard to compare 2 500-line profiles without it.
ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/tools/
Also, there have evidently been a few I/O scheduler fixes since -test3.
Please retry with -test5.
--
Dave Hansen
haveblue@us.ibm.com
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: I/O degredation with AS on 2.6.0-test3
2003-09-08 20:26 ` I/O degredation with AS on 2.6.0-test3 Dave Hansen
@ 2003-09-09 2:44 ` Nick Piggin
0 siblings, 0 replies; 5+ messages in thread
From: Nick Piggin @ 2003-09-09 2:44 UTC (permalink / raw)
To: Dave Hansen; +Cc: Mike Sullivan, Linux Kernel Mailing List
Dave Hansen wrote:
>You might want to try Martin Bligh's diffprofile utility. It's a bit
>hard to compare 2 500-line profiles without it.
>
>ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/tools/
>
>Also, there have evidently been a few I/O scheduler fixes since -test3.
>Please retry with -test5.
>
More important, are they regressions vs. previous kernels with AS?
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: I/O degredation with AS on 2.6.0-test3
2003-09-17 15:22 Mike Sullivan
@ 2003-09-17 22:57 ` Nick Piggin
0 siblings, 0 replies; 5+ messages in thread
From: Nick Piggin @ 2003-09-17 22:57 UTC (permalink / raw)
To: Mike Sullivan; +Cc: linux-kernel
Mike Sullivan wrote:
>
>>>Dave Hansen wrote:
>>>
>>>You might want to try Martin Bligh's diffprofile utility. It's a bit
>>>hard to compare 2 500-line profiles without it.
>>>
>>>ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/tools/
>>>
>>>Also, there have evidently been a few I/O scheduler fixes since -test3.
>>>Please retry with -test5.
>>>
>>>
>
>>Nick Piggen wrote:
>>More important, are they regressions vs. previous kernels with AS?
>>
>
>I've checked 2.5.69mm9, 2.5.75, 2.6.0test3, and 2.6.0test5 and they all
>show a
>significant degradation in IO performance (>40%) when using the as
>scheduler
>compared to the deadline scheduler.
>
>Configuration: IBM x440 8way with hypertheading enabled, 16GB RAM,
>4 QLA2310 FC adapters attached to 2 FastT900 controllers
>(112 disks total). The 112 physical disks are striped as 8 raid 0
>logical disks. There are a total of 40 raw devices setup (5 per raid0
>disk).
>
>I've reattached excerpts from the database test for vmstat and
>the readprofile diffs created from Martin's diffprofile utility
>(diffprofile readp.as readp.dl).
>
Thanks Mike, I'll get a patch for you to try soon. I'm a bit busy ATM.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: I/O degredation with AS on 2.6.0-test3
@ 2003-09-17 15:22 Mike Sullivan
2003-09-17 22:57 ` Nick Piggin
0 siblings, 1 reply; 5+ messages in thread
From: Mike Sullivan @ 2003-09-17 15:22 UTC (permalink / raw)
To: piggin; +Cc: linux-kernel, Mike Sullivan
>> Dave Hansen wrote:
>>
>>You might want to try Martin Bligh's diffprofile utility. It's a bit
>>hard to compare 2 500-line profiles without it.
>>
>>ftp://ftp.kernel.org/pub/linux/kernel/people/mbligh/tools/
>>
>>Also, there have evidently been a few I/O scheduler fixes since -test3.
>>Please retry with -test5.
>>
>Nick Piggen wrote:
>More important, are they regressions vs. previous kernels with AS?
I've checked 2.5.69mm9, 2.5.75, 2.6.0test3, and 2.6.0test5 and they all
show a
significant degradation in IO performance (>40%) when using the as
scheduler
compared to the deadline scheduler.
Configuration: IBM x440 8way with hypertheading enabled, 16GB RAM,
4 QLA2310 FC adapters attached to 2 FastT900 controllers
(112 disks total). The 112 physical disks are striped as 8 raid 0
logical disks. There are a total of 40 raw devices setup (5 per raid0
disk).
I've reattached excerpts from the database test for vmstat and
the readprofile diffs created from Martin's diffprofile utility
(diffprofile readp.as readp.dl).
vmstat for anticipatory scheduler
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
14 41 0 3489576 3840 168280 0 0 25 2 70 30 1 2
96 0
4 103 0 3329760 3860 172160 0 0 233549 97 6606 21284 27 6
0 67
7 91 0 3327496 3864 172156 0 0 137106 14 7364 32957 20 5
0 75
2 84 0 3325416 3868 172412 0 0 138979 73 7096 30445 19 5
0 76
6 78 0 3324528 3868 173192 0 0 143213 444 7328 24901 13 4
0 83
0 105 0 3316408 3876 173964 0 0 142413 524 7531 26854 12 5
0 83
7 94 0 3313664 3880 174220 0 0 134939 821 7336 29037 14 5
0 82
1 110 0 3313560 3884 174216 0 0 114651 923 6990 27063 11 4
0 84
5 109 0 3313008 3884 174216 0 0 110115 862 6937 26381 11 4
0 85
3 111 0 3325400 3920 174180 0 0 108664 1030 6892 26721 11 4
0 85
vmstat for deadline scheduler
procs -----------memory---------- ---swap-- -----io---- --system--
----cpu----
r b swpd free buff cache si so bi bo in cs us sy id
wa
19 0 0 3518364 3104 119096 0 0 43 1 78 62 1 2
97 1
70 23 0 3361380 3776 170684 0 0 351035 538 3521 19435 68 17
9 5
96 25 0 3321060 3816 171684 0 0 588377 349 4973 25000 91 9
0 0
79 41 0 3316492 3824 171676 0 0 621683 831 5037 26853 91 9
0 0
86 48 0 3318684 3836 172964 0 0 609092 3073 4882 27878 91 9
0 0
71 44 0 3310164 3848 173212 0 0 551741 3746 4291 28492 92 8
0 0
91 35 0 3297708 3852 174248 0 0 508096 1425 4410 29533 91 8
0 1
9217 161.5% scsi_request_fn
8588 75.5% do_softirq
8236 236.5% qla2x00_start_scsi
7890 550.6% follow_hugetlb_page
5822 47.6% schedule
3432 762.7% do_direct_IO
3275 253.5% sys_msgsnd
3171 781.0% submit_page_section
2645 527.9% set_page_dirty_lock
2541 169.1% unlock_page
2061 467.3% dio_bio_complete
1979 620.4% dio_bio_add_page
1552 674.8% get_offset_cyclone
1528 217.7% system_call
1509 1012.8% bio_add_page
1485 629.2% fsync_buffers_list
1472 213.0% del_timer_sync
1184 240.2% get_request
1147 151.3% try_to_wake_up
1000 628.9% dio_refill_pages
701 722.7% blkdev_get_blocks
680 957.7% max_block
660 628.6% .text.lock.util
607 35.6% qla2x00_queuecommand
604 17.4% dio_bio_end_io
581 162.3% bio_alloc
542 171.5% ipc_lock
502 0.0% generic_unplug_device
494 425.9% get_jiffies_64
488 650.7% page_add_rmap
473 276.6% sys_msgrcv
469 609.1% find_vma
463 593.6% dio_send_cur_page
420 3230.8% default_llseek
414 246.4% blk_run_queue
412 91.4% direct_io_worker
400 126.2% __generic_file_aio_read
397 536.5% page_waitqueue
363 394.6% __copy_from_user_ll
362 122.3% kfree
349 623.2% get_user_pages
344 8600.0% copy_page_range
309 148.6% scsi_put_command
298 172.3% testmsg
297 76.9% kmem_cache_alloc
285 84.1% kmem_cache_free
279 1268.2% do_readv_writev
224 164.7% fget_light
202 1188.2% do_wp_page
196 233.3% __copy_to_user_ll
196 560.0% __kmalloc
184 29.3% do_no_page
181 603.3% get_more_blocks
171 33.8% scsi_device_unbusy
167 136.9% do_page_fault
166 2766.7% generic_file_readv
160 141.6% sd_rw_intr
156 70.0% huge_pte_offset
155 620.0% wait_on_page_bit
149 191.0% do_gettimeofday
148 389.5% __set_page_dirty_buffers
143 4766.7% sys_llseek
142 215.2% sys_ipc
139 165.5% sys_semtimedop
132 275.0% dio_get_page
125 223.2% find_extend_vma
120 190.5% del_timer
118 100.0% update_atime
118 184.4% kmap_atomic
117 144.4% io_schedule
115 61.8% blockdev_direct_IO
114 495.7% ipcperms
108 0.0% deadline_set_request
103 251.2% do_anonymous_page
90 100.0% generic_file_direct_IO
88 125.7% blkdev_direct_IO
87 348.0% do_setitimer
86 215.0% do_getitimer
84 466.7% store_msg
82 52.2% generic_make_request
78 222.9% do_clock_nanosleep
75 166.7% find_get_page
74 123.3% bio_destructor
67 78.8% submit_bio
66 507.7% wake_up_process
66 188.6% sys_setitimer
65 928.6% pte_alloc_one
65 127.5% syscall_exit
64 206.5% page_remove_rmap
64 400.0% sys_gettimeofday
62 35.8% batch_entropy_store
61 762.5% load_msg
58 96.7% mempool_free
57 28.9% mempool_alloc
57 300.0% blk_queue_bounce
54 2700.0% sys_readv
52 0.0% huge_page_release
45 21.1% qla2x00_next
45 71.4% scsi_softirq
44 275.0% handle_mm_fault
43 307.1% .text.lock.direct_io
42 4200.0% sys_select
40 142.9% scsi_decide_disposition
40 3.7% scsi_dispatch_cmd
40 190.5% zap_pte_range
40 4000.0% find_lock_page
39 27.9% dnotify_parent
38 633.3% vfs_readv
37 72.5% dio_await_completion
37 308.3% scsi_next_command
37 67.3% __d_lookup
35 3500.0% do_select
34 3400.0% device_not_available
34 3400.0% work_resched
33 157.1% scsi_finish_command
32 49.2% proc_pid_stat
31 182.4% try_atomic_semop
31 620.0% radix_tree_lookup
30 41.7% scsi_init_cmd_errh
28 155.6% syscall_call
27 180.0% elv_set_request
27 75.0% mempool_alloc_slab
27 117.4% vsnprintf
26 650.0% pte_chain_alloc
26 130.0% scsi_add_timer
26 23.4% scsi_run_queue
25 75.8% adjust_abs_time
22 57.9% dio_new_bio
22 71.0% blkdev_writepage
21 3.8% __end_that_request_first
20 117.6% buffered_rmqueue
20 133.3% qla2x00_calc_iocbs_64
20 500.0% __pte_chain_free
19 146.2% dio_bio_alloc
19 211.1% free_msg
18 1800.0% set_huge_pte
18 360.0% filemap_nopage
17 89.5% update_queue
17 242.9% pte_alloc_map
17 0.0% shmem_getpage
17 566.7% fget
17 242.9% release_pages
17 22.7% bio_put
17 566.7% __alloc_pages
17 0.0% find_vma_prev
16 266.7% dio_complete
16 0.0% kstat_read_proc
15 136.4% __lookup
15 214.3% ipc_unlock
15 250.0% tstojiffie
15 30.0% bio_get_nr_vecs
15 55.6% sys_nanosleep
15 18.3% dio_bio_submit
14 46.7% inode_times_differ
14 700.0% copy_files
13 0.0% shmem_nopage
13 23.6% task_mem
13 260.0% bad_range
13 1300.0% tcp_poll
13 92.9% kmap_atomic_to_page
12 23.1% number
12 600.0% cache_grow
12 1200.0% math_state_restore
12 0.0% hugetlb_prefault
12 400.0% may_open
12 1200.0% copy_mm
11 73.3% free_hot_cold_page
11 122.2% clear_page_tables
10 166.7% end_that_request_chunk
10 13.3% memcpy
10 125.0% __wake_up
10 500.0% pid_alive
10 142.9% fput
10 1000.0% get_unused_fd
9 60.0% flush_tlb_mm
9 225.0% .text.lock.time
9 900.0% show_interrupts
9 900.0% ipc_checkid
8 400.0% remove_shared_vm_struct
8 800.0% restore_fpu
8 400.0% end_that_request_first
8 28.6% link_path_walk
8 800.0% qla2x00_build_scsi_iocbs_32
8 100.0% local_bh_enable
8 800.0% __posix_lock_file
8 0.0% ipc_parse_version
7 63.6% dio_bio_reap
7 0.0% free_pages
7 0.0% unmap_hugepage_range
7 0.0% nr_blockdev_pages
7 175.0% __get_user_4
7 350.0% sock_poll
7 350.0% d_alloc
6 0.0% do_sigaction
6 200.0% __mark_inode_dirty
6 600.0% schedule_delayed_work
6 600.0% __alloc_percpu
6 0.0% sys_shmat
6 600.0% do_sys_settimeofday
6 120.0% generic_file_write_nolock
6 0.0% mark_page_accessed
-7 -2.0% add_disk_randomness
-7 -63.6% show_cpuinfo
-8 -34.8% queue_delayed_work
-12 -25.0% sys_pread64
-15 -5.3% add_timer_randomness
-26 -100.0% show_regs
-32 -44.4% generic_file_read
-33 -15.6% io_schedule_timeout
-34 -3.7% qla2x00_get_new_sp
-49 -100.0% as_set_request
-61 -98.4% default_idle
-62 -45.9% vfs_read
-78 -95.1% remove_wait_queue
-86 -97.7% add_wait_queue
-94 -16.3% blk_run_queues
-128 -21.7% blk_recount_segments
-143 -86.1% mempool_free_slab
-204 -16.2% current_kernel_time
-232 -43.5% get_io_context
-253 -15.4% add_timer
-268 -100.0% as_work_handler
-409 -100.0% worker_thread
-770 -49.0% put_io_context
-1098 -48.2% dio_await_one
-1592 -30.1% scsi_end_request
-2054 -99.5% cpu_idle
-4169 -42.6% __make_request
-1527947 -89.9% total
-1605826 -99.7% poll_idle
Mike Sullivan
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2003-09-17 22:57 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2003-09-08 18:58 Mike Sullivan
2003-09-08 20:26 ` I/O degredation with AS on 2.6.0-test3 Dave Hansen
2003-09-09 2:44 ` Nick Piggin
2003-09-17 15:22 Mike Sullivan
2003-09-17 22:57 ` Nick Piggin
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).