* regression: 100% io-wait with 2.6.24-rcX @ 2008-01-07 10:51 Joerg Platte 2008-01-07 11:19 ` Ingo Molnar 0 siblings, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-07 10:51 UTC (permalink / raw) To: linux-kernel Hi, when booting kernel 2.6.24-rc{4,5,6,7} top reports up to 100% iowait, even if no program accesses the disc on my Thinkpad T40p. Kernel 2.6.23.12 does not suffer from this. Is there anything I can do to find out which process or which part of the kernel is responsible for this? I can try to bisect it, but maybe there are other possibilities to debug this, since I cannot boot this computer frequently. I discovered, that there is no iowait within the first few seconds after waking up from suspend to ram... regards, Jörg ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-07 10:51 regression: 100% io-wait with 2.6.24-rcX Joerg Platte @ 2008-01-07 11:19 ` Ingo Molnar 2008-01-07 13:24 ` Joerg Platte 0 siblings, 1 reply; 59+ messages in thread From: Ingo Molnar @ 2008-01-07 11:19 UTC (permalink / raw) To: jplatte; +Cc: linux-kernel * Joerg Platte <lists@naasa.net> wrote: > Hi, > > when booting kernel 2.6.24-rc{4,5,6,7} top reports up to 100% iowait, > even if no program accesses the disc on my Thinkpad T40p. Kernel > 2.6.23.12 does not suffer from this. Is there anything I can do to > find out which process or which part of the kernel is responsible for > this? I can try to bisect it, but maybe there are other possibilities > to debug this, since I cannot boot this computer frequently. I > discovered, that there is no iowait within the first few seconds after > waking up from suspend to ram... do: echo t > /proc/sysrq-trigger and send us the dmesg output. If the dmesg output does not include the bootup bits then increase CONFIG_LOG_BUF_SHIFT to 20 or so: CONFIG_LOG_BUF_SHIFT=20 to have a large enough kernel messages buffer. Ingo ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-07 11:19 ` Ingo Molnar @ 2008-01-07 13:24 ` Joerg Platte 2008-01-07 13:32 ` Peter Zijlstra 0 siblings, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-07 13:24 UTC (permalink / raw) To: Ingo Molnar; +Cc: linux-kernel [-- Attachment #1: Type: text/plain, Size: 459 bytes --] Am Montag, 7. Januar 2008 schrieb Ingo Molnar: > do: > > echo t > /proc/sysrq-trigger > > and send us the dmesg output. If the dmesg output does not include the > bootup bits then increase CONFIG_LOG_BUF_SHIFT to 20 or so: > > CONFIG_LOG_BUF_SHIFT=20 > > to have a large enough kernel messages buffer. The buffer was too small, hence I copied the relevant parts of /var/log/kern.log, I hope it contains all required information as well. regards, Jörg [-- Attachment #2: dmesg --] [-- Type: text/plain, Size: 17530 bytes --] kernel: f509eb84 f50dc700 f50f3be4 00000246 f50f3be4 f5c37b80 c01672ef kernel: Call Trace: kernel: [<c0271f11>] schedule_timeout+0x13/0x8b kernel: [<c01672ef>] pipe_poll+0x24/0x7f kernel: [<c016d53f>] do_select+0x370/0x3b9 kernel: [<c016daf0>] __pollwait+0x0/0xa9 kernel: [<c0118a51>] default_wake_function+0x0/0x5 last message repeated 4 times kernel: [<c0146ba7>] __rmqueue+0x14/0x196 kernel: [<c01169ce>] enqueue_entity+0x2b/0x3d kernel: [<c01169f6>] enqueue_task_fair+0x16/0x24 kernel: [<c011642e>] enqueue_task+0x3f/0x4a kernel: [<c01162b8>] __wake_up_common+0x31/0x56 kernel: [<c015e1ef>] __slab_free+0x5b/0x279 kernel: [<c0215e3a>] __kfree_skb+0x8/0x63 kernel: [<c0214a2c>] sock_wfree+0x20/0x34 kernel: [<c0216521>] skb_release_all+0xa1/0xf6 kernel: [<f8892813>] unix_stream_recvmsg+0x3e9/0x4d6 [unix] kernel: [<c016d774>] core_sys_select+0x1ec/0x282 kernel: [<c0210d34>] sock_aio_read+0xf2/0xfa kernel: [<c0123d9a>] recalc_sigpending+0xa/0x2d kernel: [<c0162077>] do_sync_read+0xc6/0x109 kernel: [<c012be2b>] autoremove_wake_function+0x0/0x33 kernel: [<f88929a4>] unix_ioctl+0xa4/0xac [unix] kernel: [<c0210adb>] sock_ioctl+0x1b2/0x1d7 kernel: [<c016dc39>] sys_select+0xa0/0x167 kernel: [<c0103d7a>] sysenter_past_esp+0x5f/0x85 kernel: [<c0270000>] netlbl_unlabel_list+0x9f/0x1bf kernel: ======================= kernel: bash R running 0 8034 7712 kernel: bash S 000000fd 0 8062 7713 kernel: dfcc0a80 00000082 12210264 000000fd f537ace0 01366da4 00000000 7fffffff kernel: dfc77000 00000001 bf9130cf c0271f11 f78127b0 00000000 f535ffb8 f535ff90 kernel: f537b034 00000000 bf9130cf f535ffb8 f535f000 c010348a 00000000 080c5c60 kernel: Call Trace: kernel: [<c0271f11>] schedule_timeout+0x13/0x8b kernel: [<c010348a>] do_notify_resume+0x81/0x5f6 kernel: [<c01f014b>] read_chan+0x2fc/0x521 kernel: [<c0118a51>] default_wake_function+0x0/0x5 kernel: [<c01efe4f>] read_chan+0x0/0x521 kernel: [<c01ed351>] tty_read+0x6f/0xad kernel: [<c01ed2e2>] tty_read+0x0/0xad kernel: [<c01628e5>] vfs_read+0x9f/0x149 kernel: [<c0162c74>] sys_read+0x41/0x67 kernel: [<c0103d7a>] sysenter_past_esp+0x5f/0x85 kernel: ======================= kernel: kwalletmanage S 00000648 0 8371 1 kernel: f4128600 00200086 fcf08470 00000648 f52fd860 0006eb74 00000000 7fffffff kernel: f407a500 00000400 f4176f4c c0271f11 f414b200 00000000 00000000 00200200 kernel: 0016e496 c0123251 f4123e00 f4176be4 00200246 f4176be4 f407a500 001672ef kernel: Call Trace: kernel: [<c0271f11>] schedule_timeout+0x13/0x8b kernel: [<c0123251>] process_timeout+0x0/0x5 kernel: [<c016d53f>] do_select+0x370/0x3b9 kernel: [<c016daf0>] __pollwait+0x0/0xa9 kernel: [<c0118a51>] default_wake_function+0x0/0x5 last message repeated 4 times kernel: [<c01abd85>] elv_insert+0xa6/0x146 kernel: [<c01169ce>] enqueue_entity+0x2b/0x3d kernel: [<c01169f6>] enqueue_task_fair+0x16/0x24 kernel: [<c011642e>] enqueue_task+0x3f/0x4a kernel: [<c01169ce>] enqueue_entity+0x2b/0x3d kernel: [<c01169f6>] enqueue_task_fair+0x16/0x24 kernel: [<c011642e>] enqueue_task+0x3f/0x4a kernel: [<c01162b8>] __wake_up_common+0x31/0x56 kernel: [<c02131ab>] sock_alloc_send_skb+0x7c/0x193 kernel: [<c01162b8>] __wake_up_common+0x31/0x56 kernel: [<c021474d>] sock_def_readable+0x31/0x6b kernel: [<f8892c62>] unix_stream_sendmsg+0x263/0x32c [unix] kernel: [<c016d774>] core_sys_select+0x1ec/0x282 kernel: [<c0210c3a>] sock_aio_write+0xe0/0xe8 kernel: [<c01681c4>] pipe_read+0x32c/0x338 kernel: [<c0161f6e>] do_sync_write+0xc6/0x109 kernel: [<c012be2b>] autoremove_wake_function+0x0/0x33 kernel: [<c01508b4>] free_pgtables+0x90/0xa0 kernel: [<f88929a4>] unix_ioctl+0xa4/0xac [unix] kernel: [<c0210adb>] sock_ioctl+0x1b2/0x1d7 kernel: [<c016dc39>] sys_select+0xa0/0x167 kernel: [<c0103d7a>] sysenter_past_esp+0x5f/0x85 kernel: ======================= kernel: kttsd S 00000655 0 8384 1 kernel: dfcc0900 00200086 af7cbb2c 00000655 f52bcce0 00044e4d 00000000 7fffffff kernel: f4131180 00000400 f4050f4c c0271f11 c010478b f889407c 00000000 f4ba7000 kernel: f416a600 00000020 f4075c40 f4050be4 00200246 f4050be4 f4131180 c01672ef kernel: Call Trace: kernel: [<c0271f11>] schedule_timeout+0x13/0x8b kernel: [<c010478b>] common_interrupt+0x23/0x28 kernel: [<c01672ef>] pipe_poll+0x24/0x7f kernel: [<c016d53f>] do_select+0x370/0x3b9 kernel: [<c016daf0>] __pollwait+0x0/0xa9 kernel: [<c0118a51>] default_wake_function+0x0/0x5 last message repeated 4 times kernel: [<c017e48d>] __getblk+0x14/0x1cd kernel: [<c0160960>] get_unused_fd_flags+0x55/0xcb kernel: [<f9a154ff>] ext3_getblk+0xb9/0x173 [ext3] kernel: [<c0218cb7>] scm_detach_fds+0xec/0x125 kernel: [<c0218982>] __scm_destroy+0x23/0x38 kernel: [<c01169ce>] enqueue_entity+0x2b/0x3d kernel: [<c01169f6>] enqueue_task_fair+0x16/0x24 kernel: [<c011642e>] enqueue_task+0x3f/0x4a kernel: [<c01162b8>] __wake_up_common+0x31/0x56 kernel: [<f889357c>] unix_write_space+0x32/0x6a [unix] kernel: [<c0214a2c>] sock_wfree+0x20/0x34 kernel: [<c0216521>] skb_release_all+0xa1/0xf6 kernel: [<f8892813>] unix_stream_recvmsg+0x3e9/0x4d6 [unix] kernel: [<c016d774>] core_sys_select+0x1ec/0x282 kernel: [<c0210d34>] sock_aio_read+0xf2/0xfa kernel: [<c0123d9a>] recalc_sigpending+0xa/0x2d kernel: [<c0162077>] do_sync_read+0xc6/0x109 kernel: [<c012be2b>] autoremove_wake_function+0x0/0x33 kernel: [<f88929a4>] unix_ioctl+0xa4/0xac [unix] kernel: [<c0210adb>] sock_ioctl+0x1b2/0x1d7 kernel: [<c016dc39>] sys_select+0xa0/0x167 kernel: [<c0103d7a>] sysenter_past_esp+0x5f/0x85 kernel: [<c0270000>] netlbl_unlabel_list+0x9f/0x1bf kernel: ======================= kernel: konqueror S 00000648 0 8665 7259 kernel: f40fb000 00000086 fa0dc86a 00000648 f418d2a0 00003d78 00000000 7fffffff kernel: f5049c80 00000800 f41b5f4c c0271f11 00000200 f41b5f4c 00000000 00200200 kernel: 0016a41f c0123251 f5cf31c0 00000246 f5cf31c0 f41b5be4 f889106f 001672ef kernel: Call Trace: kernel: [<c0271f11>] schedule_timeout+0x13/0x8b kernel: [<c0123251>] process_timeout+0x0/0x5 kernel: [<f889106f>] unix_poll+0x17/0x8b [unix] kernel: [<c016d53f>] do_select+0x370/0x3b9 kernel: [<c016daf0>] __pollwait+0x0/0xa9 kernel: [<c0118a51>] default_wake_function+0x0/0x5 last message repeated 4 times kernel: [<c017e46f>] __find_get_block+0x13d/0x147 kernel: [<c01169ce>] enqueue_entity+0x2b/0x3d kernel: [<c017e76b>] __set_page_dirty+0x125/0x132 kernel: [<c01169ce>] enqueue_entity+0x2b/0x3d kernel: [<c01169f6>] enqueue_task_fair+0x16/0x24 kernel: [<c011642e>] enqueue_task+0x3f/0x4a kernel: [<c01162b8>] __wake_up_common+0x31/0x56 kernel: [<f88927ca>] unix_stream_recvmsg+0x3a0/0x4d6 [unix] kernel: [<c0214a2c>] sock_wfree+0x20/0x34 kernel: [<f88927ca>] unix_stream_recvmsg+0x3a0/0x4d6 [unix] kernel: [<f88927ca>] unix_stream_recvmsg+0x3a0/0x4d6 [unix] kernel: [<f8892813>] unix_stream_recvmsg+0x3e9/0x4d6 [unix] kernel: [<c016d774>] core_sys_select+0x1ec/0x282 kernel: [<c0210d34>] sock_aio_read+0xf2/0xfa kernel: [<c0162077>] do_sync_read+0xc6/0x109 kernel: [<c012be2b>] autoremove_wake_function+0x0/0x33 kernel: [<c0150079>] handle_mm_fault+0x268/0x57d kernel: [<f88929a4>] unix_ioctl+0xa4/0xac [unix] kernel: [<c0210adb>] sock_ioctl+0x1b2/0x1d7 kernel: [<c016dc39>] sys_select+0xa0/0x167 kernel: [<c0103d7a>] sysenter_past_esp+0x5f/0x85 kernel: [<c0270000>] netlbl_unlabel_list+0x9f/0x1bf kernel: ======================= kernel: kio_file S 00000195 0 10548 7259 kernel: f52e0d80 00000082 97e3a48e 00000195 f5032160 00000000 00000000 7fffffff kernel: f4099380 00000020 f509af4c c0271f11 00000000 00000000 00000000 00000000 kernel: 00000000 00000000 f66b1a80 00000246 f66b1a80 f509abe4 f889106f 0009abe4 kernel: Call Trace: kernel: [<c0271f11>] schedule_timeout+0x13/0x8b kernel: [<f889106f>] unix_poll+0x17/0x8b [unix] kernel: [<c016d53f>] do_select+0x370/0x3b9 kernel: [<c016daf0>] __pollwait+0x0/0xa9 kernel: [<c0118a51>] default_wake_function+0x0/0x5 kernel: [<c0147e3f>] __alloc_pages+0x5d/0x2d9 kernel: [<c0146ba7>] __rmqueue+0x14/0x196 kernel: [<c01169ce>] enqueue_entity+0x2b/0x3d kernel: [<c01169f6>] enqueue_task_fair+0x16/0x24 kernel: [<c011642e>] enqueue_task+0x3f/0x4a kernel: [<c01162b8>] __wake_up_common+0x31/0x56 kernel: [<c02131ab>] sock_alloc_send_skb+0x7c/0x193 kernel: [<c01162b8>] __wake_up_common+0x31/0x56 kernel: [<c021474d>] sock_def_readable+0x31/0x6b kernel: [<f8892c62>] unix_stream_sendmsg+0x263/0x32c [unix] kernel: [<c016d774>] core_sys_select+0x1ec/0x282 kernel: [<c0210c3a>] sock_aio_write+0xe0/0xe8 kernel: [<c0161f6e>] do_sync_write+0xc6/0x109 kernel: [<c012be2b>] autoremove_wake_function+0x0/0x33 kernel: [<c014fffc>] handle_mm_fault+0x1eb/0x57d kernel: [<c016dc39>] sys_select+0xa0/0x167 kernel: [<c0162cdb>] sys_write+0x41/0x67 kernel: [<c0103d7a>] sysenter_past_esp+0x5f/0x85 kernel: ======================= kernel: pdflush D f41c2f14 0 18822 2 kernel: f673f000 00000046 00000286 f41c2f14 f5194ce0 00000286 00000286 f41c2f14 kernel: 00175279 f41c2f6c 00000000 c0271f6c f5ff363c f5ff3644 c0354a90 c0354a90 kernel: 00175279 c0123251 f5194b80 c03546c0 c0271f67 6c666470 00687375 00000000 kernel: Call Trace: kernel: [<c0271f6c>] schedule_timeout+0x6e/0x8b kernel: [<c0123251>] process_timeout+0x0/0x5 kernel: [<c0271f67>] schedule_timeout+0x69/0x8b kernel: [<c027179a>] __sched_text_start+0x3a/0x70 kernel: [<c014d34b>] congestion_wait+0x4e/0x62 kernel: [<c012be2b>] autoremove_wake_function+0x0/0x33 kernel: [<c014971e>] pdflush+0x0/0x1bf kernel: [<c01493c6>] wb_kupdate+0x8c/0xd1 kernel: [<c014971e>] pdflush+0x0/0x1bf kernel: [<c0149839>] pdflush+0x11b/0x1bf kernel: [<c014933a>] wb_kupdate+0x0/0xd1 kernel: [<c012bd71>] kthread+0x36/0x5d kernel: [<c012bd3b>] kthread+0x0/0x5d kernel: [<c010493b>] kernel_thread_helper+0x7/0x10 kernel: ======================= kernel: Sched Debug Version: v0.07, 2.6.24-rc7 #1 kernel: now at 5394978.742321 msecs kernel: .sysctl_sched_latency : 20.000000 kernel: .sysctl_sched_min_granularity : 4.000000 kernel: .sysctl_sched_wakeup_granularity : 10.000000 kernel: .sysctl_sched_batch_wakeup_granularity : 10.000000 kernel: .sysctl_sched_child_runs_first : 0.000001 kernel: .sysctl_sched_features : 7 kernel: kernel: cpu#0, 599.501 MHz kernel: .nr_running : 3 kernel: .load : 11596 kernel: .nr_switches : 3472897 kernel: .nr_load_updates : 789297 kernel: .nr_uninterruptible : 1 kernel: .jiffies : 1528423 kernel: .next_balance : 0.000000 kernel: .curr->pid : 8034 kernel: .clock : 6967153.202929 kernel: .idle_clock : 4447303.677171 kernel: .prev_clock_raw : 4676606.867330 kernel: .clock_warps : 103 kernel: .clock_overflows : 4982682 kernel: .clock_deep_idle_events : 944576 kernel: .clock_max_delta : 3.333116 kernel: .cpu_load[0] : 11596 kernel: .cpu_load[1] : 5798 kernel: .cpu_load[2] : 2899 kernel: .cpu_load[3] : 1450 kernel: .cpu_load[4] : 744 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 0.000001 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 0.000001 kernel: .spread : 0.000000 kernel: .spread0 : 0.000000 kernel: .nr_running : 1 kernel: .load : 2048 kernel: .nr_spread_over : 0 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 0.000001 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 0.000001 kernel: .spread : 0.000000 kernel: .spread0 : 0.000000 kernel: .nr_running : 0 kernel: .load : 0 kernel: .nr_spread_over : 0 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 0.000001 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 0.000001 kernel: .spread : 0.000000 kernel: .spread0 : 0.000000 kernel: .nr_running : 0 kernel: .load : 0 kernel: .nr_spread_over : 0 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 0.000001 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 0.000001 kernel: .spread : 0.000000 kernel: .spread0 : 0.000000 kernel: .nr_running : 0 kernel: .load : 0 kernel: .nr_spread_over : 0 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 0.000001 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 0.000001 kernel: .spread : 0.000000 kernel: .spread0 : 0.000000 kernel: .nr_running : 0 kernel: .load : 0 kernel: .nr_spread_over : 0 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 0.000001 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 0.000001 kernel: .spread : 0.000000 kernel: .spread0 : 0.000000 kernel: .nr_running : 0 kernel: .load : 0 kernel: .nr_spread_over : 0 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 0.000001 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 0.000001 kernel: .spread : 0.000000 kernel: .spread0 : 0.000000 kernel: .nr_running : 0 kernel: .load : 0 kernel: .nr_spread_over : 0 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 0.000001 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 0.000001 kernel: .spread : 0.000000 kernel: .spread0 : 0.000000 kernel: .nr_running : 0 kernel: .load : 0 kernel: .nr_spread_over : 0 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 0.000001 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 0.000001 kernel: .spread : 0.000000 kernel: .spread0 : 0.000000 kernel: .nr_running : 0 kernel: .load : 0 kernel: .nr_spread_over : 0 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 0.000001 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 0.000001 kernel: .spread : 0.000000 kernel: .spread0 : 0.000000 kernel: .nr_running : 0 kernel: .load : 0 kernel: .nr_spread_over : 0 kernel: kernel: cfs_rq kernel: .exec_clock : 0.000000 kernel: .MIN_vruntime : 214548.087264 kernel: .min_vruntime : 616382.544882 kernel: .max_vruntime : 214549.007330 kernel: .spread : 0.920066 kernel: .spread0 : 0.000000 kernel: .nr_running : 3 kernel: .load : 11596 kernel: .nr_spread_over : 0 kernel: kernel: runnable tasks: kernel: task PID tree-key switches prio exec-runtime sum-exec sum-sleep kernel: ---------------------------------------------------------------------------------------------------------- kernel: klogd 4715 214548.087264 4266 120 0 0 0.000000 0.000000 0.000000 kernel: Xorg 5872 214549.007330 983644 110 0 0 0.000000 0.000000 0.000000 kernel: R bash 8034 214548.926777 1418 120 0 0 0.000000 0.000000 0.000000 kernel: ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-07 13:24 ` Joerg Platte @ 2008-01-07 13:32 ` Peter Zijlstra 2008-01-07 13:40 ` Joerg Platte 0 siblings, 1 reply; 59+ messages in thread From: Peter Zijlstra @ 2008-01-07 13:32 UTC (permalink / raw) To: jplatte; +Cc: Ingo Molnar, linux-kernel, Fengguang Wu On Mon, 2008-01-07 at 14:24 +0100, Joerg Platte wrote: This is from: 2.6.24-rc7 > kernel: pdflush D f41c2f14 0 18822 2 > kernel: f673f000 00000046 00000286 f41c2f14 f5194ce0 00000286 00000286 f41c2f14 > kernel: 00175279 f41c2f6c 00000000 c0271f6c f5ff363c f5ff3644 c0354a90 c0354a90 > kernel: 00175279 c0123251 f5194b80 c03546c0 c0271f67 6c666470 00687375 00000000 > kernel: Call Trace: > kernel: [<c0271f6c>] schedule_timeout+0x6e/0x8b > kernel: [<c0123251>] process_timeout+0x0/0x5 > kernel: [<c0271f67>] schedule_timeout+0x69/0x8b > kernel: [<c027179a>] __sched_text_start+0x3a/0x70 > kernel: [<c014d34b>] congestion_wait+0x4e/0x62 > kernel: [<c012be2b>] autoremove_wake_function+0x0/0x33 > kernel: [<c014971e>] pdflush+0x0/0x1bf > kernel: [<c01493c6>] wb_kupdate+0x8c/0xd1 > kernel: [<c014971e>] pdflush+0x0/0x1bf > kernel: [<c0149839>] pdflush+0x11b/0x1bf > kernel: [<c014933a>] wb_kupdate+0x0/0xd1 > kernel: [<c012bd71>] kthread+0x36/0x5d > kernel: [<c012bd3b>] kthread+0x0/0x5d > kernel: [<c010493b>] kernel_thread_helper+0x7/0x10 > kernel: ======================= What filesystem are you using? ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-07 13:32 ` Peter Zijlstra @ 2008-01-07 13:40 ` Joerg Platte [not found] ` <E1JCRbA-0002bh-3c@localhost.localdomain> 0 siblings, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-07 13:40 UTC (permalink / raw) To: Peter Zijlstra; +Cc: Ingo Molnar, linux-kernel, Fengguang Wu Am Montag, 7. Januar 2008 schrieb Peter Zijlstra: > On Mon, 2008-01-07 at 14:24 +0100, Joerg Platte wrote: > > This is from: 2.6.24-rc7 > > > kernel: pdflush D f41c2f14 0 18822 2 > > kernel: f673f000 00000046 00000286 f41c2f14 f5194ce0 00000286 > > 00000286 f41c2f14 kernel: 00175279 f41c2f6c 00000000 c0271f6c > > f5ff363c f5ff3644 c0354a90 c0354a90 kernel: 00175279 c0123251 > > f5194b80 c03546c0 c0271f67 6c666470 00687375 00000000 kernel: Call Trace: > > kernel: [<c0271f6c>] schedule_timeout+0x6e/0x8b > > kernel: [<c0123251>] process_timeout+0x0/0x5 > > kernel: [<c0271f67>] schedule_timeout+0x69/0x8b > > kernel: [<c027179a>] __sched_text_start+0x3a/0x70 > > kernel: [<c014d34b>] congestion_wait+0x4e/0x62 > > kernel: [<c012be2b>] autoremove_wake_function+0x0/0x33 > > kernel: [<c014971e>] pdflush+0x0/0x1bf > > kernel: [<c01493c6>] wb_kupdate+0x8c/0xd1 > > kernel: [<c014971e>] pdflush+0x0/0x1bf > > kernel: [<c0149839>] pdflush+0x11b/0x1bf > > kernel: [<c014933a>] wb_kupdate+0x0/0xd1 > > kernel: [<c012bd71>] kthread+0x36/0x5d > > kernel: [<c012bd3b>] kthread+0x0/0x5d > > kernel: [<c010493b>] kernel_thread_helper+0x7/0x10 > > kernel: ======================= > > What filesystem are you using? Here you can see all currently mounted filesystems: /dev/sda6 on / type ext3 (rw,noatime,errors=remount-ro,acl) tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) procbususb on /proc/bus/usb type usbfs (rw) udev on /dev type tmpfs (rw,mode=0755) tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620) fusectl on /sys/fs/fuse/connections type fusectl (rw) /dev/sda7 on /tmp type ext2 (rw,noatime,errors=remount-ro,acl) /dev/sda8 on /export type ext3 (rw,noatime,errors=remount-ro,acl) /dev/sda1 on /winxp type ntfs (rw,umask=002,gid=10000,nls=utf8) rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw) automount(pid4936) on /amnt type autofs (rw,fd=4,pgrp=4936,minproto=2,maxproto=4) automount(pid4965) on /home type autofs (rw,fd=4,pgrp=4965,minproto=2,maxproto=4) nfsd on /proc/fs/nfsd type nfsd (rw) /export/home/jplatte on /home/jplatte type none (rw,bind) binfmt_misc on /proc/sys/fs/binfmt_misc type binfmt_misc (rw,noexec,nosuid,nodev) regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JCRbA-0002bh-3c@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JCRbA-0002bh-3c@localhost.localdomain> @ 2008-01-09 3:27 ` Fengguang Wu 2008-01-09 6:13 ` Joerg Platte 0 siblings, 1 reply; 59+ messages in thread From: Fengguang Wu @ 2008-01-09 3:27 UTC (permalink / raw) To: jplatte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel On Mon, Jan 07, 2008 at 02:40:13PM +0100, Joerg Platte wrote: > Am Montag, 7. Januar 2008 schrieb Peter Zijlstra: > > On Mon, 2008-01-07 at 14:24 +0100, Joerg Platte wrote: > > > > This is from: 2.6.24-rc7 > > > > > kernel: pdflush D f41c2f14 0 18822 2 > > > kernel: f673f000 00000046 00000286 f41c2f14 f5194ce0 00000286 > > > 00000286 f41c2f14 kernel: 00175279 f41c2f6c 00000000 c0271f6c > > > f5ff363c f5ff3644 c0354a90 c0354a90 kernel: 00175279 c0123251 > > > f5194b80 c03546c0 c0271f67 6c666470 00687375 00000000 kernel: Call Trace: > > > kernel: [<c0271f6c>] schedule_timeout+0x6e/0x8b > > > kernel: [<c0123251>] process_timeout+0x0/0x5 > > > kernel: [<c0271f67>] schedule_timeout+0x69/0x8b > > > kernel: [<c027179a>] __sched_text_start+0x3a/0x70 > > > kernel: [<c014d34b>] congestion_wait+0x4e/0x62 > > > kernel: [<c012be2b>] autoremove_wake_function+0x0/0x33 > > > kernel: [<c014971e>] pdflush+0x0/0x1bf > > > kernel: [<c01493c6>] wb_kupdate+0x8c/0xd1 > > > kernel: [<c014971e>] pdflush+0x0/0x1bf > > > kernel: [<c0149839>] pdflush+0x11b/0x1bf > > > kernel: [<c014933a>] wb_kupdate+0x0/0xd1 > > > kernel: [<c012bd71>] kthread+0x36/0x5d > > > kernel: [<c012bd3b>] kthread+0x0/0x5d > > > kernel: [<c010493b>] kernel_thread_helper+0x7/0x10 > > > kernel: ======================= > > > > What filesystem are you using? > > Here you can see all currently mounted filesystems: > > /dev/sda6 on / type ext3 (rw,noatime,errors=remount-ro,acl) > tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) > proc on /proc type proc (rw,noexec,nosuid,nodev) > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > procbususb on /proc/bus/usb type usbfs (rw) > udev on /dev type tmpfs (rw,mode=0755) > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620) > fusectl on /sys/fs/fuse/connections type fusectl (rw) > /dev/sda7 on /tmp type ext2 (rw,noatime,errors=remount-ro,acl) > /dev/sda8 on /export type ext3 (rw,noatime,errors=remount-ro,acl) > /dev/sda1 on /winxp type ntfs (rw,umask=002,gid=10000,nls=utf8) So they are ext3/ext2/ntfs. What if you umount ntfs? and ext2 if possible? ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-09 3:27 ` Fengguang Wu @ 2008-01-09 6:13 ` Joerg Platte [not found] ` <E1JCZg2-0001DE-RP@localhost.localdomain> 0 siblings, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-09 6:13 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu: > > /dev/sda6 on / type ext3 (rw,noatime,errors=remount-ro,acl) > > tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > procbususb on /proc/bus/usb type usbfs (rw) > > udev on /dev type tmpfs (rw,mode=0755) > > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620) > > fusectl on /sys/fs/fuse/connections type fusectl (rw) > > /dev/sda7 on /tmp type ext2 (rw,noatime,errors=remount-ro,acl) > > /dev/sda8 on /export type ext3 (rw,noatime,errors=remount-ro,acl) > > /dev/sda1 on /winxp type ntfs (rw,umask=002,gid=10000,nls=utf8) > > So they are ext3/ext2/ntfs. What if you umount ntfs? and ext2 if possible? Unmounting ntfs doesn't help, hence I converted the remaining ext2 filesystem to ext3, modified the fstab entry accordingly and rebooted. Now everything seems to be fine! Top reports an idle system and there is no abnormal iowait any longer! Seems to be ext2 was causing this! Later today I can try to remount the filesystem as ext2 to be sure the bug shows up again. regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JCZg2-0001DE-RP@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JCZg2-0001DE-RP@localhost.localdomain> @ 2008-01-09 12:04 ` Fengguang Wu 2008-01-09 12:22 ` Joerg Platte 0 siblings, 1 reply; 59+ messages in thread From: Fengguang Wu @ 2008-01-09 12:04 UTC (permalink / raw) To: jplatte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel On Wed, Jan 09, 2008 at 07:13:14AM +0100, Joerg Platte wrote: > Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu: > > > /dev/sda6 on / type ext3 (rw,noatime,errors=remount-ro,acl) > > > tmpfs on /lib/init/rw type tmpfs (rw,nosuid,mode=0755) > > > proc on /proc type proc (rw,noexec,nosuid,nodev) > > > sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) > > > procbususb on /proc/bus/usb type usbfs (rw) > > > udev on /dev type tmpfs (rw,mode=0755) > > > tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev) > > > devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=620) > > > fusectl on /sys/fs/fuse/connections type fusectl (rw) > > > /dev/sda7 on /tmp type ext2 (rw,noatime,errors=remount-ro,acl) > > > /dev/sda8 on /export type ext3 (rw,noatime,errors=remount-ro,acl) > > > /dev/sda1 on /winxp type ntfs (rw,umask=002,gid=10000,nls=utf8) > > > > So they are ext3/ext2/ntfs. What if you umount ntfs? and ext2 if possible? > > Unmounting ntfs doesn't help, hence I converted the remaining ext2 filesystem > to ext3, modified the fstab entry accordingly and rebooted. Now everything > seems to be fine! Top reports an idle system and there is no abnormal iowait > any longer! Seems to be ext2 was causing this! Later today I can try to > remount the filesystem as ext2 to be sure the bug shows up again. Thank you for the clue. However I cannot reproduce the bug on ext2/2.6.24-rc7. Can you provide more details? Thank you. Fengguang ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-09 12:04 ` Fengguang Wu @ 2008-01-09 12:22 ` Joerg Platte [not found] ` <E1JCaUd-0001Ko-Tt@localhost.localdomain> 0 siblings, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-09 12:22 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel [-- Attachment #1: Type: text/plain, Size: 783 bytes --] Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu: Thank your for the hint with the filesystems! > Thank you for the clue. However I cannot reproduce the bug on > ext2/2.6.24-rc7. Can you provide more details? Thank you. I attached some more information. I'm using the ata_piix driver for my PATA disk and cdrom drive and booted with hpet=force. Kernel 2.6.23.X has been patched with the -hrt patches to enable the hidden hpet time on the ICH4-M chipset. I just rebooted the notebook and mounted /tmp again as ext2 and now the iowait problem is back. Seems to be reproducible on my computer. What additional information do you need? regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 [-- Attachment #2: lspci --] [-- Type: text/plain, Size: 1620 bytes --] 00:00.0 Host bridge: Intel Corporation 82855PM Processor to I/O Controller (rev 03) 00:01.0 PCI bridge: Intel Corporation 82855PM Processor to AGP Controller (rev 03) 00:1d.0 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #1 (rev 01) 00:1d.1 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #2 (rev 01) 00:1d.2 USB Controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) USB UHCI Controller #3 (rev 01) 00:1d.7 USB Controller: Intel Corporation 82801DB/DBM (ICH4/ICH4-M) USB2 EHCI Controller (rev 01) 00:1e.0 PCI bridge: Intel Corporation 82801 Mobile PCI Bridge (rev 81) 00:1f.0 ISA bridge: Intel Corporation 82801DBM (ICH4-M) LPC Interface Bridge (rev 01) 00:1f.1 IDE interface: Intel Corporation 82801DBM (ICH4-M) IDE Controller (rev 01) 00:1f.3 SMBus: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) SMBus Controller (rev 01) 00:1f.5 Multimedia audio controller: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97 Audio Controller (rev 01) 00:1f.6 Modem: Intel Corporation 82801DB/DBL/DBM (ICH4/ICH4-L/ICH4-M) AC'97 Modem Controller (rev 01) 01:00.0 VGA compatible controller: ATI Technologies Inc Radeon R250 [Mobility FireGL 9000] (rev 02) 02:00.0 CardBus bridge: Texas Instruments PCI1520 PC card Cardbus Controller (rev 01) 02:00.1 CardBus bridge: Texas Instruments PCI1520 PC card Cardbus Controller (rev 01) 02:01.0 Ethernet controller: Intel Corporation 82540EP Gigabit Ethernet Controller (Mobile) (rev 03) 02:02.0 Ethernet controller: Atheros Communications, Inc. AR5211 802.11ab NIC (rev 01) [-- Attachment #3: lsmod --] [-- Type: text/plain, Size: 7475 bytes --] Module Size Used by radeon 112644 2 drm 73684 3 radeon binfmt_misc 10568 1 rfcomm 35924 1 l2cap 22660 5 rfcomm bluetooth 51872 4 rfcomm,l2cap snd_rtctimer 3616 1 nfsd 201652 13 auth_rpcgss 39492 1 nfsd exportfs 4544 1 nfsd af_packet 20292 4 autofs4 20356 2 ipt_MASQUERADE 3328 3 iptable_nat 6660 1 nf_nat 17948 2 ipt_MASQUERADE,iptable_nat nf_conntrack_ipv4 16328 11 iptable_nat xt_state 2304 9 ipt_LOG 5632 9 xt_limit 2432 9 ipt_REJECT 4288 2 xt_mark 1728 2 xt_tcpudp 2944 12 xt_mac 1728 38 iptable_filter 2752 1 xt_MARK 2048 3 xt_multiport 2880 8 iptable_mangle 2560 1 ip_tables 12168 3 iptable_nat,iptable_filter,iptable_mangle x_tables 13828 12 ipt_MASQUERADE,iptable_nat,xt_state,ipt_LOG,xt_limit,ipt_REJECT,xt_mark,xt_tcpudp,xt_mac,xt_MARK,xt_multiport,ip_tables nf_conntrack_ftp 8224 0 nf_conntrack 61200 6 ipt_MASQUERADE,iptable_nat,nf_nat,nf_conntrack_ipv4,xt_state,nf_conntrack_ftp cpufreq_userspace 3800 0 cpufreq_stats 4868 0 cpufreq_powersave 1600 0 nfs 227136 0 lockd 59576 3 nfsd,nfs nfs_acl 3264 2 nfsd,nfs sunrpc 169308 11 nfsd,auth_rpcgss,nfs,lockd,nfs_acl deflate 3520 0 zlib_deflate 17960 1 deflate zlib_inflate 15488 1 deflate twofish 6592 0 twofish_common 36416 1 twofish camellia 28736 0 serpent 18368 0 blowfish 8128 0 des_generic 16128 0 cbc 4288 0 ecb 3328 0 aes_generic 26536 0 aes_i586 32436 0 geode_aes 5700 0 blkcipher 6404 3 cbc,ecb,geode_aes xcbc 5384 0 sha256_generic 11008 0 sha1_generic 2432 0 md5 3840 0 crypto_null 2368 0 hmac 4352 0 crypto_hash 2048 2 xcbc,hmac cryptomgr 3520 0 crypto_algapi 16256 19 deflate,twofish,camellia,serpent,blowfish,des_generic,cbc,ecb,aes_generic,aes_i586,geode_aes,blkcipher,xcbc,sha256_generic,sha1_generic,md5,crypto_null,hmac,cryptomgr af_key 35536 0 nls_utf8 1920 1 ntfs 92900 1 nls_base 7040 2 nls_utf8,ntfs ext2 65160 1 fuse 44948 1 ebtable_broute 1984 0 bridge 49112 1 ebtable_broute llc 7128 1 bridge ebtable_nat 2176 0 ebtable_filter 2176 0 ebtables 17024 3 ebtable_broute,ebtable_nat,ebtable_filter deadline_iosched 5376 0 as_iosched 12488 0 ircomm_tty 22728 0 ircomm 12868 1 ircomm_tty tun 10816 0 acpi_cpufreq 6732 1 sbs 14024 0 sbshc 6656 1 sbs joydev 11008 0 snd_intel8x0 31260 2 snd_intel8x0m 16588 4 snd_ac97_codec 92644 2 snd_intel8x0,snd_intel8x0m ac97_bus 1920 1 snd_ac97_codec snd_pcm_oss 43040 0 snd_pcm 75976 6 snd_intel8x0,snd_intel8x0m,snd_ac97_codec,snd_pcm_oss snd_mixer_oss 15552 1 snd_pcm_oss irtty_sir 5952 0 sir_dev 13508 1 irtty_sir nsc_ircc 17040 0 irda 108680 4 ircomm_tty,ircomm,sir_dev,nsc_ircc crc_ccitt 1984 1 irda snd_seq_oss 29460 0 parport_pc 36092 0 parport 33544 1 parport_pc snd_seq_midi 8800 0 snd_rawmidi 23200 1 snd_seq_midi 8250_pnp 9472 0 snd_seq_midi_event 6656 2 snd_seq_oss,snd_seq_midi snd_seq 48732 6 snd_seq_oss,snd_seq_midi,snd_seq_midi_event snd_timer 21572 3 snd_rtctimer,snd_pcm,snd_seq snd_seq_device 7820 4 snd_seq_oss,snd_seq_midi,snd_rawmidi,snd_seq yenta_socket 24268 2 rsrc_nonstatic 11584 1 yenta_socket pcmcia 36660 0 firmware_class 9216 1 pcmcia snd 50644 22 snd_intel8x0,snd_intel8x0m,snd_ac97_codec,snd_pcm_oss,snd_pcm,snd_mixer_oss,snd_seq_oss,snd_rawmidi,snd_seq,snd_timer,snd_seq_device pcmcia_core 36688 3 yenta_socket,rsrc_nonstatic,pcmcia serio_raw 6404 0 pcspkr 2624 0 psmouse 36176 0 soundcore 7364 1 snd snd_page_alloc 10184 3 snd_intel8x0,snd_intel8x0m,snd_pcm 8250_pci 22912 0 8250 21880 2 8250_pnp,8250_pci serial_core 20480 1 8250 i2c_i801 8272 0 rng_core 4804 0 iTCO_wdt 11616 0 iTCO_vendor_support 3652 1 iTCO_wdt battery 13252 0 ac 5892 0 power_supply 9604 3 sbs,battery,ac video 18192 0 output 3456 1 video button 8080 0 intel_agp 22868 1 agpgart 30896 2 drm,intel_agp evdev 10752 7 ext3 121032 2 jbd 44116 1 ext3 mbcache 8192 2 ext2,ext3 sg 32800 0 sr_mod 15396 0 cdrom 32604 1 sr_mod sd_mod 25680 6 ata_generic 7172 0 usbhid 40100 0 hid 26628 1 usbhid ff_memless 5128 1 usbhid ata_piix 17476 5 floppy 51632 0 pata_acpi 6976 0 e1000 109824 0 libata 144016 3 ata_generic,ata_piix,pata_acpi scsi_mod 140948 4 sg,sr_mod,sd_mod,libata ehci_hcd 31372 0 uhci_hcd 22668 0 usbcore 131256 4 usbhid,ehci_hcd,uhci_hcd thermal 15580 0 processor 28016 3 acpi_cpufreq,thermal fan 4356 0 unix 27220 1085 cpufreq_conservative 6752 0 cpufreq_ondemand 7608 1 freq_table 4100 3 cpufreq_stats,acpi_cpufreq,cpufreq_ondemand thinkpad_acpi 49296 0 hwmon 3092 1 thinkpad_acpi nvram 8136 1 thinkpad_acpi fbcon 37976 72 tileblit 2368 1 fbcon font 8320 1 fbcon bitblit 5184 1 fbcon softcursor 2112 1 bitblit radeonfb 97392 1 fb 45768 5 fbcon,tileblit,bitblit,softcursor,radeonfb fb_ddc 2432 1 radeonfb backlight 4932 3 video,thinkpad_acpi,radeonfb i2c_algo_bit 5828 1 radeonfb cfbcopyarea 3264 1 radeonfb i2c_core 22224 4 i2c_i801,radeonfb,fb_ddc,i2c_algo_bit cfbimgblt 2624 1 radeonfb cfbfillrect 3392 1 radeonfb [-- Attachment #4: dmesg --] [-- Type: text/plain, Size: 15481 bytes --] river thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for hkey thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle HKEY for hkey thinkpad_acpi: wan_init: wan is not supported, status 0xf7ed8e5c thinkpad_acpi: ibm_init: probing for video thinkpad_acpi: video_init: initializing video subdriver thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for vid thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle \_SB.PCI0.AGP.VID for vid thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for vid2 thinkpad_acpi: drv_acpi_handle_init: ACPI handle for vid2 not found thinkpad_acpi: video_init: video is supported, mode 3 thinkpad_acpi: ibm_init: video installed thinkpad_acpi: ibm_init: probing for light thinkpad_acpi: light_init: initializing light subdriver thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for ledb thinkpad_acpi: drv_acpi_handle_init: ACPI handle for ledb not found thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for lght thinkpad_acpi: drv_acpi_handle_init: ACPI handle for lght not found thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for cmos thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle \UCMS for cmos thinkpad_acpi: light_init: light is supported thinkpad_acpi: ibm_init: light installed thinkpad_acpi: ibm_init: probing for bay thinkpad_acpi: bay_init: initializing bay subdriver thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for bay thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle \_SB.PCI0.IDE0.SCND.MSTR for bay thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for bay_ej thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle _EJ0 for bay_ej thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for bay2 thinkpad_acpi: drv_acpi_handle_init: ACPI handle for bay2 not found thinkpad_acpi: bay_init: bay 1: status supported, eject supported; bay 2: status not supported, eject not supported thinkpad_acpi: setup_acpi_notify: setting up ACPI notify for bay thinkpad_acpi: ibm_init: bay installed thinkpad_acpi: ibm_init: probing for cmos thinkpad_acpi: cmos_init: initializing cmos commands subdriver thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for cmos thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle \UCMS for cmos thinkpad_acpi: cmos_init: cmos commands are supported thinkpad_acpi: ibm_init: cmos installed thinkpad_acpi: ibm_init: probing for led thinkpad_acpi: led_init: initializing LED subdriver thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for led thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle LED for led thinkpad_acpi: led_init: LED commands are supported, mode 3 thinkpad_acpi: ibm_init: led installed thinkpad_acpi: ibm_init: probing for beep thinkpad_acpi: beep_init: initializing beep subdriver thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for beep thinkpad_acpi: drv_acpi_handle_init: Found ACPI handle BEEP for beep thinkpad_acpi: beep_init: beep is supported thinkpad_acpi: ibm_init: beep installed thinkpad_acpi: ibm_init: probing for thermal thinkpad_acpi: thermal_init: initializing thermal subdriver thinkpad_acpi: thermal_init: thermal is supported, mode 3 thinkpad_acpi: ibm_init: thermal installed thinkpad_acpi: ibm_init: probing for ecdump thinkpad_acpi: ibm_init: ecdump installed thinkpad_acpi: ibm_init: probing for brightness thinkpad_acpi: brightness_init: initializing brightness subdriver thinkpad_acpi: brightness_init: selected brightness_mode=3 thinkpad_acpi: brightness_init: brightness is supported thinkpad_acpi: ibm_init: brightness installed thinkpad_acpi: ibm_init: probing for volume thinkpad_acpi: ibm_init: volume installed thinkpad_acpi: ibm_init: probing for fan thinkpad_acpi: fan_init: initializing fan subdriver thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for fans thinkpad_acpi: drv_acpi_handle_init: ACPI handle for fans not found thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for gfan thinkpad_acpi: drv_acpi_handle_init: ACPI handle for gfan not found thinkpad_acpi: drv_acpi_handle_init: trying to locate ACPI handle for sfan thinkpad_acpi: drv_acpi_handle_init: ACPI handle for sfan not found thinkpad_acpi: fan_init: fan is supported, modes 2, 2 thinkpad_acpi: ibm_init: fan installed input: ThinkPad Extra Buttons as /class/input/input1 NET: Registered protocol family 1 ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3]) ACPI: Processor [CPU] (supports 8 throttling states) Marking TSC unstable due to: TSC halts in idle. Time: hpet clocksource has been installed. ACPI: Thermal Zone [THM0] (50 C) usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb USB Universal Host Controller Interface driver v3.0 ACPI: PCI Interrupt 0000:00:1d.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1d.0 to 64 uhci_hcd 0000:00:1d.0: UHCI Host Controller uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1 uhci_hcd 0000:00:1d.0: irq 11, io base 0x00001800 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 ACPI: PCI Interrupt 0000:00:1d.1[B] -> Link [LNKD] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1d.1 to 64 uhci_hcd 0000:00:1d.1: UHCI Host Controller uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2 uhci_hcd 0000:00:1d.1: irq 11, io base 0x00001820 usb usb2: configuration #1 chosen from 1 choice hub 2-0:1.0: USB hub found hub 2-0:1.0: 2 ports detected SCSI subsystem initialized ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 ACPI: PCI Interrupt 0000:00:1d.2[C] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1d.2 to 64 uhci_hcd 0000:00:1d.2: UHCI Host Controller uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3 uhci_hcd 0000:00:1d.2: irq 11, io base 0x00001840 usb usb3: configuration #1 chosen from 1 choice hub 3-0:1.0: USB hub found hub 3-0:1.0: 2 ports detected Intel(R) PRO/1000 Network Driver - version 7.3.20-k2 Copyright (c) 1999-2006 Intel Corporation. libata version 3.00 loaded. ACPI: PCI Interrupt Link [LNKH] enabled at IRQ 11 ACPI: PCI Interrupt 0000:00:1d.7[D] -> Link [LNKH] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1d.7 to 64 ehci_hcd 0000:00:1d.7: EHCI Host Controller ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 4 ehci_hcd 0000:00:1d.7: debug port 1 PCI: cache line size of 32 is not supported by device 0000:00:1d.7 ehci_hcd 0000:00:1d.7: irq 11, io mem 0xc0000000 ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 usb usb4: configuration #1 chosen from 1 choice hub 4-0:1.0: USB hub found hub 4-0:1.0: 6 ports detected Floppy drive(s): fd0 is 1.44M FDC 0 is a National Semiconductor PC87306 ACPI: PCI Interrupt 0000:02:01.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11 e1000: 0000:02:01.0: e1000_validate_option: PHY Smart Power Down Enabled e1000: 0000:02:01.0: e1000_probe: (PCI:33MHz:32-bit) 00:16:41:10:19:11 usb 2-1: new low speed USB device using uhci_hcd and address 2 e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection PCI: Enabling device 0000:00:1f.1 (0005 -> 0007) ACPI: PCI Interrupt 0000:00:1f.1[A] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1f.1 to 64 ACPI: PCI interrupt for device 0000:00:1f.1 disabled ata_piix 0000:00:1f.1: version 2.12 ACPI: PCI Interrupt 0000:00:1f.1[A] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1f.1 to 64 scsi0 : ata_piix scsi1 : ata_piix ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x1860 irq 14 ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x1868 irq 15 usb 2-1: configuration #1 chosen from 1 choice usbcore: registered new interface driver hiddev input: Logitech Optical USB Mouse as /class/input/input2 input: USB HID v1.10 Mouse [Logitech Optical USB Mouse] on usb-0000:00:1d.1-1 usbcore: registered new interface driver usbhid drivers/hid/usbhid/hid-core.c: v2.6:USB HID core driver ata1.00: ATA-7: SAMSUNG HM160JC, AP100-16, max UDMA/100 ata1.00: 312581808 sectors, multi 16: LBA48 ata1.00: configured for UDMA/100 ata2.00: ATAPI: UJDA745 DVD/CDRW, 1.06, max UDMA/33 ata2.00: configured for UDMA/33 scsi 0:0:0:0: Direct-Access ATA SAMSUNG HM160JC AP10 PQ: 0 ANSI: 5 scsi 1:0:0:0: CD-ROM MATSHITA UJDA745 DVD/CDRW 1.06 PQ: 0 ANSI: 5 Driver 'sd' needs updating - please use bus_type methods sd 0:0:0:0: [sda] 312581808 512-byte hardware sectors (160042 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sd 0:0:0:0: [sda] 312581808 512-byte hardware sectors (160042 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 Driver 'sr' needs updating - please use bus_type methods sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sda: sda1 sda2 < sda5 sda6 sda7 sda8 > sd 0:0:0:0: [sda] Attached SCSI disk sr0: scsi3-mmc drive: 24x/24x writer cd/rw xa/form2 cdda tray Uniform CD-ROM driver Revision: 3.20 sr 1:0:0:0: Attached scsi CD-ROM sr0 sd 0:0:0:0: Attached scsi generic sg0 type 0 sr 1:0:0:0: Attached scsi generic sg1 type 5 Attempting manual resume swsusp: Marking nosave pages: 000000000009f000 - 0000000000100000 swsusp: Basic memory bitmaps created swsusp: Basic memory bitmaps freed kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. Linux agpgart interface v0.102 agpgart: Detected an Intel 855PM Chipset. agpgart: AGP aperture is 256M @ 0xd0000000 input: Power Button (FF) as /class/input/input3 ACPI: Power Button (FF) [PWRF] input: Lid Switch as /class/input/input4 ACPI: Lid Switch [LID] input: Sleep Button (CM) as /class/input/input5 ACPI: Sleep Button (CM) [SLPB] input: Video Bus as /class/input/input6 ACPI: Video Device [VID] (multi-head: yes rom: no post: no) ACPI: AC Adapter [AC] (on-line) ACPI: Battery Slot [BAT0] (battery present) iTCO_vendor_support: vendor-support=0 iTCO_wdt: Intel TCO WatchDog Timer Driver v1.02 (26-Jul-2007) iTCO_wdt: Found a ICH4-M TCO device (Version=1, TCOBASE=0x1060) iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) intel_rng: FWH not detected ACPI: PCI Interrupt 0000:00:1f.3[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11 Serial: 8250/16550 driver $Revision: 1.90 $ 2 ports, IRQ sharing disabled serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A ACPI: PCI Interrupt 0000:00:1f.6[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11 ACPI: PCI interrupt for device 0000:00:1f.6 disabled input: PC Speaker as /class/input/input7 Yenta: CardBus bridge found at 0000:02:00.0 [1014:0512] Yenta: Using INTVAL to route CSC interrupts to PCI Yenta: Routing CardBus interrupts to PCI Yenta TI: socket 0000:02:00.0, mfunc 0x01d21022, devctl 0x64 Yenta: ISA IRQ mask 0x04b8, PCI irq 11 Socket status: 30000006 pcmcia: parent PCI bridge I/O window: 0x4000 - 0x8fff cs: IO port probe 0x4000-0x8fff: clean. pcmcia: parent PCI bridge Memory window: 0xc0200000 - 0xcfffffff pcmcia: parent PCI bridge Memory window: 0xe8000000 - 0xefffffff 00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A Synaptics Touchpad, model: 1, fw: 5.9, id: 0x2c6ab1, caps: 0x884793/0x0 serio: Synaptics pass-through port at isa0060/serio1/input0 Yenta: CardBus bridge found at 0000:02:00.1 [1014:0512] Yenta: Using INTVAL to route CSC interrupts to PCI Yenta: Routing CardBus interrupts to PCI Yenta TI: socket 0000:02:00.1, mfunc 0x01d21022, devctl 0x64 input: SynPS/2 Synaptics TouchPad as /class/input/input8 parport_pc 00:0b: reported by Plug and Play ACPI parport0: PC-style at 0x3bc (0x7bc), irq 7, dma 0 [PCSPP,TRISTATE,COMPAT,ECP,DMA] NET: Registered protocol family 23 nsc-ircc 00:0c: activated nsc-ircc, chip->init nsc-ircc, Found chip at base=0x02e nsc-ircc, driver loaded (Dag Brattli) IrDA: Registered device irda0 nsc-ircc, Using dongle: IBM31T1100 or Temic TFDS6000/TFDS6500 Yenta: ISA IRQ mask 0x0430, PCI irq 11 Socket status: 30000006 pcmcia: parent PCI bridge I/O window: 0x4000 - 0x8fff cs: IO port probe 0x4000-0x8fff: clean. pcmcia: parent PCI bridge Memory window: 0xc0200000 - 0xcfffffff pcmcia: parent PCI bridge Memory window: 0xe8000000 - 0xefffffff ACPI: PCI Interrupt 0000:00:1f.6[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1f.6 to 64 ACPI: PCI Interrupt 0000:00:1f.5[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1f.5 to 64 cs: IO port probe 0xc00-0xcff: clean. cs: IO port probe 0x800-0x8ff: clean. cs: IO port probe 0x100-0x4ff: excluding 0x4d0-0x4d7 cs: IO port probe 0xa00-0xaff: clean. cs: IO port probe 0xc00-0xcff: clean. cs: IO port probe 0x800-0x8ff: clean. cs: IO port probe 0x100-0x4ff: excluding 0x4d0-0x4d7 cs: IO port probe 0xa00-0xaff: clean. intel8x0_measure_ac97_clock: measured 52955 usecs intel8x0: measured clock 207 rejected intel8x0: clocking to 48000 Adding 1951856k swap on /dev/sda5. Priority:-1 extents:1 across:1951856k EXT3 FS on sda6, internal journal tun: Universal TUN/TAP device driver, 1.6 tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> IrCOMM protocol (Dag Brattli) io scheduler anticipatory registered io scheduler deadline registered Ebtables v2.0 registered Bridge firewalling registered fuse init (API version 7.9) EXT2-fs warning (device sda7): ext2_fill_super: mounting ext3 filesystem as ext2 kjournald starting. Commit interval 5 seconds EXT3 FS on sda8, internal journal EXT3-fs: mounted filesystem with ordered data mode. NTFS driver 2.1.29 [Flags: R/O MODULE]. NTFS volume version 3.1. IBM TrackPoint firmware: 0x0e, buttons: 3/3 input: TPPS/2 IBM TrackPoint as /class/input/input9 NET: Registered protocol family 15 padlock: VIA PadLock Hash Engine not detected. padlock: VIA PadLock Hash Engine not detected. padlock: VIA PadLock not detected. RPC: Registered udp transport module. RPC: Registered tcp transport module. nf_conntrack version 0.5.0 (16384 buckets, 65536 max) ip_tables: (C) 2000-2006 Netfilter Core Team Clocksource tsc unstable (delta = -197916262 ns) e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX NET: Registered protocol family 17 Installing knfsd (copyright (C) 1996 okir@monad.swb.de). NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory NFSD: starting 90-second grace period Bluetooth: Core ver 2.11 NET: Registered protocol family 31 Bluetooth: HCI device and connection manager initialized Bluetooth: HCI socket layer initialized Bluetooth: L2CAP ver 2.9 Bluetooth: L2CAP socket layer initialized Bluetooth: RFCOMM socket layer initialized Bluetooth: RFCOMM TTY layer initialized Bluetooth: RFCOMM ver 1.8 [drm] Initialized drm 1.1.0 20060810 [drm] Initialized radeon 1.28.0 20060524 on minor 0 agpgart: Found an AGP 2.0 compliant device at 0000:00:00.0. agpgart: Putting AGP V2 device at 0000:00:00.0 into 4x mode agpgart: Putting AGP V2 device at 0000:01:00.0 into 4x mode [drm] Setting GART location based on new memory map [drm] Loading R200 Microcode [drm] writeback test succeeded in 2 usecs ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JCaUd-0001Ko-Tt@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JCaUd-0001Ko-Tt@localhost.localdomain> @ 2008-01-09 12:57 ` Fengguang Wu 2008-01-09 13:04 ` Joerg Platte 0 siblings, 1 reply; 59+ messages in thread From: Fengguang Wu @ 2008-01-09 12:57 UTC (permalink / raw) To: jplatte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel On Wed, Jan 09, 2008 at 01:22:33PM +0100, Joerg Platte wrote: > Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu: > > Thank your for the hint with the filesystems! > > > Thank you for the clue. However I cannot reproduce the bug on > > ext2/2.6.24-rc7. Can you provide more details? Thank you. > > I attached some more information. I'm using the ata_piix driver for my PATA > disk and cdrom drive and booted with hpet=force. Kernel 2.6.23.X has been ~~~~~~~~ not 2.6.24-rc7? > patched with the -hrt patches to enable the hidden hpet time on the ICH4-M > chipset. I just rebooted the notebook and mounted /tmp again as ext2 and now > the iowait problem is back. Seems to be reproducible on my computer. What > additional information do you need? I mounted an ext2 as tmp and find no problem. My config options are: CONFIG_EXT2_FS=y CONFIG_EXT2_FS_XATTR=y CONFIG_EXT2_FS_POSIX_ACL=y CONFIG_EXT2_FS_SECURITY=y # CONFIG_EXT2_FS_XIP is not set Fengguang ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-09 12:57 ` Fengguang Wu @ 2008-01-09 13:04 ` Joerg Platte [not found] ` <E1JCrMj-0001HR-SZ@localhost.localdomain> ` (2 more replies) 0 siblings, 3 replies; 59+ messages in thread From: Joerg Platte @ 2008-01-09 13:04 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu: > On Wed, Jan 09, 2008 at 01:22:33PM +0100, Joerg Platte wrote: > > Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu: > > > > Thank your for the hint with the filesystems! > > > > > Thank you for the clue. However I cannot reproduce the bug on > > > ext2/2.6.24-rc7. Can you provide more details? Thank you. > > > > I attached some more information. I'm using the ata_piix driver for my > > PATA disk and cdrom drive and booted with hpet=force. Kernel 2.6.23.X has > > been > > ~~~~~~~~ > not 2.6.24-rc7? > No, 2.6.23.X was the last working kernel without this problem, the bug shows up with 2.6.24-rcX. I just wanted to emphasis, that I don't think that it has something to do with the hpet stuff. Kernel 2.6.24-rcX is unpatched, because the hrt stuff has been merged. > > patched with the -hrt patches to enable the hidden hpet time on the > > ICH4-M chipset. I just rebooted the notebook and mounted /tmp again as > > ext2 and now the iowait problem is back. Seems to be reproducible on my > > computer. What additional information do you need? > > I mounted an ext2 as tmp and find no problem. My config options are: > > CONFIG_EXT2_FS=y > CONFIG_EXT2_FS_XATTR=y > CONFIG_EXT2_FS_POSIX_ACL=y > CONFIG_EXT2_FS_SECURITY=y > # CONFIG_EXT2_FS_XIP is not set > > Fengguang CONFIG_EXT2_FS=m CONFIG_EXT2_FS_XATTR=y CONFIG_EXT2_FS_POSIX_ACL=y CONFIG_EXT2_FS_SECURITY=y CONFIG_EXT2_FS_XIP=y Here it is modular and I enabled CONFIG_EXT2_FS_XIP, but the same is enabled on 2.6.23.X. Maybe it is related to libata? I additionally discovered, that the problem disappears for a few seconds when I press the eject button for the ultra bay of my thinkpad. Pressing the button unregisters the cdrom drive to be able to replace it with a hard drive or a battery. Maybe this bug is thinkpad relared? regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JCrMj-0001HR-SZ@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JCrMj-0001HR-SZ@localhost.localdomain> @ 2008-01-10 6:58 ` Fengguang Wu 0 siblings, 0 replies; 59+ messages in thread From: Fengguang Wu @ 2008-01-10 6:58 UTC (permalink / raw) To: jplatte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel On Wed, Jan 09, 2008 at 02:04:29PM +0100, Joerg Platte wrote: > Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu: > > On Wed, Jan 09, 2008 at 01:22:33PM +0100, Joerg Platte wrote: > > > Am Mittwoch, 9. Januar 2008 schrieb Fengguang Wu: > > > > > > Thank your for the hint with the filesystems! > > > > > > > Thank you for the clue. However I cannot reproduce the bug on > > > > ext2/2.6.24-rc7. Can you provide more details? Thank you. > > > > > > I attached some more information. I'm using the ata_piix driver for my > > > PATA disk and cdrom drive and booted with hpet=force. Kernel 2.6.23.X has > > > been > > > > ~~~~~~~~ > > not 2.6.24-rc7? > > > No, 2.6.23.X was the last working kernel without this problem, the bug shows > up with 2.6.24-rcX. I just wanted to emphasis, that I don't think that it has > something to do with the hpet stuff. Kernel 2.6.24-rcX is unpatched, because > the hrt stuff has been merged. > > > > patched with the -hrt patches to enable the hidden hpet time on the > > > ICH4-M chipset. I just rebooted the notebook and mounted /tmp again as > > > ext2 and now the iowait problem is back. Seems to be reproducible on my > > > computer. What additional information do you need? > > > > I mounted an ext2 as tmp and find no problem. My config options are: > > > > CONFIG_EXT2_FS=y > > CONFIG_EXT2_FS_XATTR=y > > CONFIG_EXT2_FS_POSIX_ACL=y > > CONFIG_EXT2_FS_SECURITY=y > > # CONFIG_EXT2_FS_XIP is not set > > > > Fengguang > > CONFIG_EXT2_FS=m > CONFIG_EXT2_FS_XATTR=y > CONFIG_EXT2_FS_POSIX_ACL=y > CONFIG_EXT2_FS_SECURITY=y > CONFIG_EXT2_FS_XIP=y > > Here it is modular and I enabled CONFIG_EXT2_FS_XIP, but the same is enabled > on 2.6.23.X. Maybe it is related to libata? I additionally discovered, that > the problem disappears for a few seconds when I press the eject button for > the ultra bay of my thinkpad. Pressing the button unregisters the cdrom drive > to be able to replace it with a hard drive or a battery. Maybe this bug is > thinkpad relared? At last I caught it :-) [ 1862.219189] requeue_io 301: inode 50948 size 0 at 03:02(hda2) [ 1862.219199] requeue_io 301: inode 51616 size 0 at 03:02(hda2) [ 1862.219204] requeue_io 301: inode 51656 size 0 at 03:02(hda2) [ 1862.219208] requeue_io 301: inode 51655 size 0 at 03:02(hda2) [ 1862.219216] mm/page-writeback.c 668 wb_kupdate: pdflush(182) 10768 global 3100 0 0 wc _M tw 1024 sk 1 [ 1862.319039] requeue_io 301: inode 50948 size 0 at 03:02(hda2) [ 1862.319050] requeue_io 301: inode 51616 size 0 at 03:02(hda2) [ 1862.319055] requeue_io 301: inode 51656 size 0 at 03:02(hda2) [ 1862.319059] requeue_io 301: inode 51655 size 0 at 03:02(hda2) [ 1862.319068] mm/page-writeback.c 668 wb_kupdate: pdflush(182) 10768 global 3100 0 0 wc _M tw 1024 sk 1 They are some zero sized files, maybe something goes wrong with the truncate code. ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JCrsE-0000v4-Dz@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JCrsE-0000v4-Dz@localhost.localdomain> @ 2008-01-10 7:30 ` Fengguang Wu 0 siblings, 0 replies; 59+ messages in thread From: Fengguang Wu @ 2008-01-10 7:30 UTC (permalink / raw) To: jplatte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel [-- Attachment #1: Type: text/plain, Size: 105 bytes --] Joerg, Can you try the attached patches? Thank you. I cannot reliably reproduce the bug yet. Fengguang [-- Attachment #2: writeback-ext2-fix.patch --] [-- Type: text/plain, Size: 437 bytes --] mm/filemap_xip.c | 1 + 1 files changed, 1 insertion(+) Index: linux/mm/filemap_xip.c =================================================================== --- linux.orig/mm/filemap_xip.c +++ linux/mm/filemap_xip.c @@ -431,6 +431,7 @@ xip_truncate_page(struct address_space * return PTR_ERR(page); } zero_user_page(page, offset, length, KM_USER0); + set_page_dirty(page); return 0; } EXPORT_SYMBOL_GPL(xip_truncate_page); [-- Attachment #3: writeback-debug.patch --] [-- Type: text/plain, Size: 2677 bytes --] --- mm/page-writeback.c | 29 +++++++++++++++++++++++++++++ 1 file changed, 29 insertions(+) --- linux-2.6.24-git17.orig/mm/page-writeback.c +++ linux-2.6.24-git17/mm/page-writeback.c @@ -98,6 +98,26 @@ EXPORT_SYMBOL(laptop_mode); /* End of sysctl-exported parameters */ +#define writeback_debug_report(n, wbc) do { \ + __writeback_debug_report(n, wbc, __FILE__, __LINE__, __FUNCTION__); \ +} while (0) + +void __writeback_debug_report(long n, struct writeback_control *wbc, + const char *file, int line, const char *func) +{ + printk(KERN_DEBUG "%s %d %s: %s(%d) %ld " + "global %lu %lu %lu " + "wc %c%c tw %ld sk %ld\n", + file, line, func, + current->comm, current->pid, n, + global_page_state(NR_FILE_DIRTY), + global_page_state(NR_WRITEBACK), + global_page_state(NR_UNSTABLE_NFS), + wbc->encountered_congestion ? 'C':'_', + wbc->more_io ? 'M':'_', + wbc->nr_to_write, + wbc->pages_skipped); +} static void background_writeout(unsigned long _min_pages); @@ -395,6 +415,7 @@ static void balance_dirty_pages(struct a pages_written += write_chunk - wbc.nr_to_write; get_dirty_limits(&background_thresh, &dirty_thresh, &bdi_thresh, bdi); + writeback_debug_report(pages_written, &wbc); } /* @@ -421,6 +442,7 @@ static void balance_dirty_pages(struct a break; /* We've done our duty */ congestion_wait(WRITE, HZ/10); + writeback_debug_report(-pages_written, &wbc); } if (bdi_nr_reclaimable + bdi_nr_writeback < bdi_thresh && @@ -515,6 +537,11 @@ void throttle_vm_writeout(gfp_t gfp_mask global_page_state(NR_WRITEBACK) <= dirty_thresh) break; congestion_wait(WRITE, HZ/10); + printk(KERN_DEBUG "throttle_vm_writeout: " + "congestion_wait on %lu+%lu > %lu\n", + global_page_state(NR_UNSTABLE_NFS), + global_page_state(NR_WRITEBACK), + dirty_thresh); /* * The caller might hold locks which can prevent IO completion @@ -557,6 +584,7 @@ static void background_writeout(unsigned wbc.pages_skipped = 0; writeback_inodes(&wbc); min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write; + writeback_debug_report(min_pages, &wbc); if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) { /* Wrote less than expected */ if (wbc.encountered_congestion || wbc.more_io) @@ -630,6 +658,7 @@ static void wb_kupdate(unsigned long arg wbc.encountered_congestion = 0; wbc.nr_to_write = MAX_WRITEBACK_PAGES; writeback_inodes(&wbc); + writeback_debug_report(nr_to_write, &wbc); if (wbc.nr_to_write > 0) { if (wbc.encountered_congestion || wbc.more_io) congestion_wait(WRITE, HZ/10); [-- Attachment #4: requeue_io-debug.patch --] [-- Type: text/plain, Size: 1140 bytes --] Subject: track redirty_tail() calls It helps a lot to know how redirty_tail() are called. Cc: Ken Chen <kenchen@google.com> Cc: Andrew Morton <akpm@linux-foundation.org> Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn> --- fs/fs-writeback.c | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) --- linux-2.6.24-git17.orig/fs/fs-writeback.c +++ linux-2.6.24-git17/fs/fs-writeback.c @@ -164,12 +164,26 @@ static void redirty_tail(struct inode *i list_move(&inode->i_list, &sb->s_dirty); } +#define requeue_io(inode) \ + do { \ + __requeue_io(inode, __LINE__); \ + } while (0) + /* * requeue inode for re-scanning after sb->s_io list is exhausted. */ -static void requeue_io(struct inode *inode) +static void __requeue_io(struct inode *inode, int line) { list_move(&inode->i_list, &inode->i_sb->s_more_io); + + printk(KERN_DEBUG "requeue_io %d: inode %lu size %llu at %02x:%02x(%s)\n", + line, + inode->i_ino, + i_size_read(inode), + MAJOR(inode->i_sb->s_dev), + MINOR(inode->i_sb->s_dev), + inode->i_sb->s_id + ); } static void inode_sync_complete(struct inode *inode) ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <20080110073046.GA3432@mail.ustc.edu.cn>]
[parent not found: <E1JCsDr-0002cl-0e@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JCsDr-0002cl-0e@localhost.localdomain> @ 2008-01-10 7:53 ` Fengguang Wu 2008-01-10 8:37 ` Joerg Platte 0 siblings, 1 reply; 59+ messages in thread From: Fengguang Wu @ 2008-01-10 7:53 UTC (permalink / raw) To: jplatte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel On Thu, Jan 10, 2008 at 03:30:46PM +0800, Fengguang Wu wrote: > Joerg, > > Can you try the attached patches? Thank you. > I cannot reliably reproduce the bug yet. Please ignore the first patch and only apply the two debugging patches. They will produce many printk messages. The output of `dmesg` will be enough for me. Thank you, Fengguang ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-10 7:53 ` Fengguang Wu @ 2008-01-10 8:37 ` Joerg Platte [not found] ` <E1JCt0n-00048n-AD@localhost.localdomain> 0 siblings, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-10 8:37 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu: > On Thu, Jan 10, 2008 at 03:30:46PM +0800, Fengguang Wu wrote: > > Joerg, > > > > Can you try the attached patches? Thank you. > > I cannot reliably reproduce the bug yet. > > Please ignore the first patch and only apply the two debugging > patches. They will produce many printk messages. The output of > `dmesg` will be enough for me. Too late, I'm already compiling the kernel. But I have currently another problem, because the iowait problem disappeared today after the regular Debian update. I'll try to install the old package versions to make it show up again. Maybe that helps to debug it. Grüße, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JCt0n-00048n-AD@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JCt0n-00048n-AD@localhost.localdomain> @ 2008-01-10 8:43 ` Fengguang Wu 2008-01-10 10:03 ` Joerg Platte 0 siblings, 1 reply; 59+ messages in thread From: Fengguang Wu @ 2008-01-10 8:43 UTC (permalink / raw) To: jplatte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel On Thu, Jan 10, 2008 at 09:37:53AM +0100, Joerg Platte wrote: > Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu: > > On Thu, Jan 10, 2008 at 03:30:46PM +0800, Fengguang Wu wrote: > > > Joerg, > > > > > > Can you try the attached patches? Thank you. > > > I cannot reliably reproduce the bug yet. > > > > Please ignore the first patch and only apply the two debugging > > patches. They will produce many printk messages. The output of > > `dmesg` will be enough for me. > > Too late, I'm already compiling the kernel. But I have currently another It won't hurt, as you don't use XIP. > problem, because the iowait problem disappeared today after the regular > Debian update. I'll try to install the old package versions to make it show > up again. Maybe that helps to debug it. Thank you. I'm running sid, ext2 as rootfs now ;-) ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-10 8:43 ` Fengguang Wu @ 2008-01-10 10:03 ` Joerg Platte [not found] ` <E1JDBk4-0000UF-03@localhost.localdomain> 0 siblings, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-10 10:03 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu: > > problem, because the iowait problem disappeared today after the regular > > Debian update. I'll try to install the old package versions to make it > > show up again. Maybe that helps to debug it. > > Thank you. I'm running sid, ext2 as rootfs now ;-) The error is back and I'm getting thousands of messages like this with the patched kernel: mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3937 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3937 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3937 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3937 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3937 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3937 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3937 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3937 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3938 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3938 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3939 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3939 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3939 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3939 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3939 0 0 wc _M tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3939 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3939 0 0 wc _M tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3939 0 0 wc _M tw 1024 sk 2 regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JDBk4-0000UF-03@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JDBk4-0000UF-03@localhost.localdomain> @ 2008-01-11 4:43 ` Fengguang Wu 2008-01-11 5:29 ` Joerg Platte 2008-01-12 23:32 ` Joerg Platte 0 siblings, 2 replies; 59+ messages in thread From: Fengguang Wu @ 2008-01-11 4:43 UTC (permalink / raw) To: jplatte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel On Thu, Jan 10, 2008 at 11:03:05AM +0100, Joerg Platte wrote: > Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu: > > > problem, because the iowait problem disappeared today after the regular > > > Debian update. I'll try to install the old package versions to make it > > > show up again. Maybe that helps to debug it. > > > > Thank you. I'm running sid, ext2 as rootfs now ;-) > > The error is back and I'm getting thousands of messages like this with the > patched kernel: > > mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc _M > tw 1024 sk 0 > requeue_io 301: inode 81441 size 0 at 08:07(sda7) > mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M > tw 1024 sk 2 > requeue_io 301: inode 81441 size 0 at 08:07(sda7) > mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc _M > tw 1024 sk 2 > requeue_io 301: inode 81441 size 0 at 08:07(sda7) Joerg, what's the output of `dumpe2fs /dev/sda7` and `lsof|grep /tmp`? ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-11 4:43 ` Fengguang Wu @ 2008-01-11 5:29 ` Joerg Platte 2008-01-11 6:41 ` Joerg Platte 2008-01-12 23:32 ` Joerg Platte 1 sibling, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-11 5:29 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel Am Freitag, 11. Januar 2008 schrieb Fengguang Wu: > Joerg, what's the output of `dumpe2fs /dev/sda7` and `lsof|grep /tmp`? Fengang, here is the output (kernel 2.6.24-rc7 without your patches): Filesystem volume name: TMP Last mounted on: <not available> Filesystem UUID: e23ae961-bbdc-44bc-b662-f28f7314939b Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal resize_inode dir_index filetype sparse_super large_file Filesystem flags: signed directory hash Default mount options: (none) Filesystem state: not clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 244320 Block count: 487966 Reserved block count: 24398 Free blocks: 468728 Free inodes: 244162 First block: 0 Block size: 4096 Fragment size: 4096 Reserved GDT blocks: 119 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 16288 Inode blocks per group: 509 Filesystem created: Thu Feb 8 18:46:19 2007 Last mount time: Thu Jan 10 11:09:35 2008 Last write time: Fri Jan 11 06:12:58 2008 Mount count: 25 Maximum mount count: 39 Last checked: Tue Dec 11 22:07:03 2007 Check interval: 15552000 (6 months) Next check after: Sun Jun 8 23:07:03 2008 Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 128 Journal inode: 8 Default directory hash: tea Directory Hash Seed: 79e271e9-b874-4e11-92b0-cdb36d07e1c1 Journal backup: inode blocks Journal size: 32M Group 0: (Blocks 0-32767) Primary superblock at 0, Group descriptors at 1-1 Reserved GDT blocks at 2-120 Block bitmap at 121 (+121), Inode bitmap at 122 (+122) Inode table at 123-631 (+123) 23926 free blocks, 16275 free inodes, 2 directories Free blocks: 638-10239, 16384, 18444-30719, 30721-32767 Free inodes: 14-16288 Group 1: (Blocks 32768-65535) Backup superblock at 32768, Group descriptors at 32769-32769 Reserved GDT blocks at 32770-32888 Block bitmap at 32889 (+121), Inode bitmap at 32890 (+122) Inode table at 32891-33399 (+123) 32135 free blocks, 16287 free inodes, 1 directories Free blocks: 33400-57343, 57345-65535 Free inodes: 16290-32576 Group 2: (Blocks 65536-98303) Block bitmap at 65536 (+0), Inode bitmap at 65537 (+1) Inode table at 65538-66046 (+2) 32256 free blocks, 16286 free inodes, 1 directories Free blocks: 66047-96255, 96257-98303 Free inodes: 32579-48864 Group 3: (Blocks 98304-131071) Backup superblock at 98304, Group descriptors at 98305-98305 Reserved GDT blocks at 98306-98424 Block bitmap at 98425 (+121), Inode bitmap at 98426 (+122) Inode table at 98427-98935 (+123) 32135 free blocks, 16286 free inodes, 1 directories Free blocks: 98936-120831, 120833-131071 Free inodes: 48867-65152 Group 4: (Blocks 131072-163839) Block bitmap at 131072 (+0), Inode bitmap at 131073 (+1) Inode table at 131074-131582 (+2) 32256 free blocks, 16287 free inodes, 1 directories Free blocks: 131583-133119, 133121-163839 Free inodes: 65154-81440 Group 5: (Blocks 163840-196607) Backup superblock at 163840, Group descriptors at 163841-163841 Reserved GDT blocks at 163842-163960 Block bitmap at 163961 (+121), Inode bitmap at 163962 (+122) Inode table at 163963-164471 (+123) 32135 free blocks, 16286 free inodes, 1 directories Free blocks: 164472-182271, 182273-196607 Free inodes: 81443-97728 Group 6: (Blocks 196608-229375) Block bitmap at 196608 (+0), Inode bitmap at 196609 (+1) Inode table at 196610-197118 (+2) 32087 free blocks, 16265 free inodes, 1 directories Free blocks: 197119-200703, 200705-210943, 210945-210951, 210970-210975, 210988-215039, 215041-215047, 215058-215071, 215090-217087, 217089-219135, 219137-219143, 219177-219199, 219226-219255, 219257-219263, 219265-219271, 219274-219279, 219305-219335, 219337-219343, 219345-219351, 219354-219359, 219371-219383, 219385-227327, 227331-229375 Free inodes: 97752-114016 Group 7: (Blocks 229376-262143) Backup superblock at 229376, Group descriptors at 229377-229377 Reserved GDT blocks at 229378-229496 Block bitmap at 229497 (+121), Inode bitmap at 229498 (+122) Inode table at 229499-230007 (+123) 32134 free blocks, 16278 free inodes, 1 directories Free blocks: 230008-253951, 253953-260095, 260097-262143 Free inodes: 114022, 114025-114027, 114031-130304 Group 8: (Blocks 262144-294911) Block bitmap at 262144 (+0), Inode bitmap at 262145 (+1) Inode table at 262146-262654 (+2) 29809 free blocks, 16193 free inodes, 17 directories Free blocks: 262661-264191, 264244, 264250, 264263, 264272, 264281, 264288-266239, 266870-266871, 266877-268287, 268289-270335, 270965-274431, 274434-282623, 282625-290815, 291897-292863, 292866-294911 Free inodes: 130316, 130319, 130388, 130402-130406, 130408-146592 Group 9: (Blocks 294912-327679) Backup superblock at 294912, Group descriptors at 294913-294913 Reserved GDT blocks at 294914-295032 Block bitmap at 295033 (+121), Inode bitmap at 295034 (+122) Inode table at 295035-295543 (+123) 32135 free blocks, 16286 free inodes, 1 directories Free blocks: 295544-313343, 313345-327679 Free inodes: 146595-162880 Group 10: (Blocks 327680-360447) Block bitmap at 327680 (+0), Inode bitmap at 327681 (+1) Inode table at 327682-328190 (+2) 32256 free blocks, 16283 free inodes, 1 directories Free blocks: 328191-342015, 342017-360447 Free inodes: 162886-179168 Group 11: (Blocks 360448-393215) Block bitmap at 360448 (+0), Inode bitmap at 360449 (+1) Inode table at 360450-360958 (+2) 32248 free blocks, 16286 free inodes, 1 directories Free blocks: 360959-372735, 372744-380927, 380929-393215 Free inodes: 179171-195456 Group 12: (Blocks 393216-425983) Block bitmap at 393216 (+0), Inode bitmap at 393217 (+1) Inode table at 393218-393726 (+2) 32257 free blocks, 16288 free inodes, 0 directories Free blocks: 393727-425983 Free inodes: 195457-211744 Group 13: (Blocks 425984-458751) Block bitmap at 425984 (+0), Inode bitmap at 425985 (+1) Inode table at 425986-426494 (+2) 32256 free blocks, 16287 free inodes, 1 directories Free blocks: 426495-436223, 436225-458751 Free inodes: 211746-228032 Group 14: (Blocks 458752-487965) Block bitmap at 458752 (+0), Inode bitmap at 458753 (+1) Inode table at 458754-459262 (+2) 28703 free blocks, 16288 free inodes, 0 directories Free blocks: 459263-487965 Free inodes: 228033-244320 konqueror 987 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca konqueror 987 jplatte 12r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca konqueror 987 jplatte 13u REG 8,7 579 97731 /tmp/kde-jplatte/konqueror-crash-Jv2u8a.log konqueror 987 jplatte 14u REG 8,7 128289 97734 /tmp/kde-jplatte/khtmlcacheA7VjAa.tmp (deleted) konqueror 987 jplatte 16u REG 8,7 43334 97750 /tmp/kde-jplatte/khtmlcacheZUNlsc.tmp (deleted) konqueror 987 jplatte 22u REG 8,7 797 97751 /tmp/kde-jplatte/khtmlcache76bZYa.tmp (deleted) konqueror 987 jplatte 27u REG 8,7 102347 97735 /tmp/kde-jplatte/khtmlcacheMhlDJb.tmp (deleted) konqueror 987 jplatte 31u REG 8,7 354 97738 /tmp/kde-jplatte/khtmlcacheq21uoc.tmp (deleted) konqueror 987 jplatte 32u REG 8,7 1360 97742 /tmp/kde-jplatte/khtmlcacheo2Ms2a.tmp (deleted) konqueror 987 jplatte 34u REG 8,7 6378 97745 /tmp/kde-jplatte/khtmlcacheLETtgc.tmp (deleted) konqueror 987 jplatte 35u REG 8,7 97350 97746 /tmp/kde-jplatte/khtmlcache5jit8a.tmp (deleted) konqueror 987 jplatte 36u REG 8,7 354 97747 /tmp/kde-jplatte/khtmlcache7VBSNa.tmp (deleted) konqueror 987 jplatte 37u REG 8,7 1360 97748 /tmp/kde-jplatte/khtmlcachetSNbub.tmp (deleted) konqueror 987 jplatte 38u REG 8,7 6073 97749 /tmp/kde-jplatte/khtmlcacheKAKhuc.tmp (deleted) xfs 5503 root 3u unix 0xdfdd7e00 15884 /tmp/.font-unix/fs7100 xfs 5503 root 4u unix 0xf724ee00 18389 /tmp/.font-unix/fs7100 Xorg 6031 root 1u unix 0xf7151000 17447 /tmp/.X11-unix/X0 Xorg 6031 root 3u unix 0xf40f5540 108479 /tmp/.X11-unix/X0 Xorg 6031 root 10u unix 0xf400ec40 107863 /tmp/.X11-unix/X0 Xorg 6031 root 13u unix 0xf724ea80 18532 /tmp/.X11-unix/X0 Xorg 6031 root 14u unix 0xf6702000 18880 /tmp/.X11-unix/X0 Xorg 6031 root 15u unix 0xf5d6ac40 19298 /tmp/.X11-unix/X0 Xorg 6031 root 16u unix 0xf5297c40 25334 /tmp/.X11-unix/X0 Xorg 6031 root 17u unix 0xf410b1c0 31056 /tmp/.X11-unix/X0 Xorg 6031 root 18u unix 0xf67021c0 19002 /tmp/.X11-unix/X0 Xorg 6031 root 19u unix 0xf5c7f700 19014 /tmp/.X11-unix/X0 Xorg 6031 root 20u unix 0xf5051380 20514 /tmp/.X11-unix/X0 Xorg 6031 root 21u unix 0xf5ccc540 19100 /tmp/.X11-unix/X0 Xorg 6031 root 22u unix 0xf5c7fa80 19086 /tmp/.X11-unix/X0 Xorg 6031 root 23u unix 0xf725c380 19123 /tmp/.X11-unix/X0 Xorg 6031 root 24u unix 0xf5d29380 19147 /tmp/.X11-unix/X0 Xorg 6031 root 25u unix 0xf5174700 31277 /tmp/.X11-unix/X0 Xorg 6031 root 26u unix 0xf5db3380 19252 /tmp/.X11-unix/X0 Xorg 6031 root 27u unix 0xf51ce8c0 41610 /tmp/.X11-unix/X0 Xorg 6031 root 28u unix 0xf5de8e00 19379 /tmp/.X11-unix/X0 Xorg 6031 root 29u unix 0xf5e0b8c0 19401 /tmp/.X11-unix/X0 Xorg 6031 root 30u unix 0xf5e5c540 19419 /tmp/.X11-unix/X0 Xorg 6031 root 31u unix 0xf5e9ba80 19500 /tmp/.X11-unix/X0 Xorg 6031 root 32u unix 0xf5e0ba80 19515 /tmp/.X11-unix/X0 Xorg 6031 root 33u unix 0xf5e7fa80 19531 /tmp/.X11-unix/X0 Xorg 6031 root 34u unix 0xf5ea61c0 19982 /tmp/.X11-unix/X0 Xorg 6031 root 35u unix 0xf5ea6a80 19991 /tmp/.X11-unix/X0 Xorg 6031 root 36u unix 0xf7220000 20286 /tmp/.X11-unix/X0 Xorg 6031 root 37u unix 0xf5187700 54906 /tmp/.X11-unix/X0 Xorg 6031 root 39u unix 0xf5187e00 43577 /tmp/.X11-unix/X0 Xorg 6031 root 40u unix 0xf5ec9a80 20377 /tmp/.X11-unix/X0 Xorg 6031 root 41u unix 0xf51b2380 20809 /tmp/.X11-unix/X0 Xorg 6031 root 42u unix 0xf71921c0 20455 /tmp/.X11-unix/X0 Xorg 6031 root 43u unix 0xf528a1c0 99356 /tmp/.X11-unix/X0 Xorg 6031 root 44u unix 0xf5297540 21298 /tmp/.X11-unix/X0 Xorg 6031 root 45u unix 0xf5099c40 20549 /tmp/.X11-unix/X0 Xorg 6031 root 46u unix 0xf50ff380 20600 /tmp/.X11-unix/X0 Xorg 6031 root 47u unix 0xf510c8c0 20655 /tmp/.X11-unix/X0 Xorg 6031 root 48u unix 0xf510cc40 20657 /tmp/.X11-unix/X0 Xorg 6031 root 49u unix 0xf5138540 20663 /tmp/.X11-unix/X0 Xorg 6031 root 50u unix 0xf50cb380 20683 /tmp/.X11-unix/X0 Xorg 6031 root 51u unix 0xf51b2e00 20883 /tmp/.X11-unix/X0 Xorg 6031 root 52u unix 0xf52c3a80 21316 /tmp/.X11-unix/X0 Xorg 6031 root 53u unix 0xf52f3a80 21373 /tmp/.X11-unix/X0 Xorg 6031 root 54u unix 0xf528ac40 21871 /tmp/.X11-unix/X0 Xorg 6031 root 55u unix 0xdfd301c0 25107 /tmp/.X11-unix/X0 Xorg 6031 root 56u unix 0xf67ab380 25380 /tmp/.X11-unix/X0 Xorg 6031 root 58u unix 0xf512c700 49503 /tmp/.X11-unix/X0 Xorg 6031 root 59u unix 0xf51cee00 55093 /tmp/.X11-unix/X0 Xorg 6031 root 60u unix 0xf5187a80 54949 /tmp/.X11-unix/X0 kontact 6669 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kontact 6669 jplatte 12r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kontact 6669 jplatte 20u unix 0xf5005000 109900 /tmp/ksocket-jplatte/kontactb5GSMb.slave-socket kontact 6669 jplatte 33u unix 0xf712a700 108568 /tmp/ksocket-jplatte/kontact7somnc.slave-socket kontact 6669 jplatte 34u unix 0xf50d78c0 108570 /tmp/ksocket-jplatte/kontactQdchTb.slave-socket kontact 6669 jplatte 36u unix 0xf50d7000 108569 /tmp/ksocket-jplatte/kontactdlqZ9b.slave-socket kio_imap4 6691 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kio_imap4 6691 jplatte 10r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kio_imap4 6698 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kio_imap4 6698 jplatte 10r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kio_imap4 6699 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kio_imap4 6699 jplatte 10r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca gam_serve 6972 root cwd DIR 8,7 0 211745 /tmp/1878472609 (deleted) ssh-agent 7097 jplatte 3u unix 0xf724f1c0 18873 /tmp/ssh-gdxKKv7049/agent.7049 gpg-agent 7098 jplatte 5u unix 0xf724f540 18877 /tmp/gpg-cKJSSb/S.gpg-agent kdeinit 7159 jplatte 9u unix 0xf5c2ba80 18966 /tmp/ksocket-jplatte/kdeinit__0 kdeinit 7159 jplatte 10u unix 0xf5c2bc40 18968 /tmp/ksocket-jplatte/kdeinit- dcopserve 7162 jplatte 6u unix 0xf5c7f380 19011 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 8u unix 0xf5c891c0 18977 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 11u unix 0xf5c89e00 18993 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 12u unix 0xf5cd6700 20495 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 13u unix 0xf5c2b1c0 19092 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 14u unix 0xf5d1d540 19107 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 15u unix 0xf5dd5e00 19295 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 16u unix 0xf725c000 19118 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 17u unix 0xf725ce00 19142 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 18u unix 0xf5ee9000 108474 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 20u unix 0xf5db3000 19249 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 21u unix 0xf5db3540 19264 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 22u unix 0xf51cea80 108608 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 23u unix 0xf5de81c0 19376 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 24u unix 0xf7161700 20283 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 25u unix 0xf400e000 108491 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 26u unix 0xf5e0b540 19398 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 27u unix 0xf5e9b700 19497 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 28u unix 0xf5e5c1c0 19416 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 29u unix 0xf5de8c40 19541 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 30u unix 0xf400e700 108493 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 31u unix 0xf5ea4a80 20007 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 32u unix 0xf5e7f700 19526 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 33u unix 0xf5ea6e00 19993 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 34u unix 0xf400e380 108495 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 35u unix 0xf50ff540 108497 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 36u unix 0xf5cb0000 108499 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 37u unix 0xf5ec9380 20374 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 38u unix 0xf5cb0700 108501 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 39u unix 0xf5cb01c0 108503 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 40u unix 0xf514b8c0 20914 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 41u unix 0xf506be00 21019 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 42u unix 0xf5cd6a80 20506 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 43u unix 0xf51ce540 108610 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 44u unix 0xf5099380 20542 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 45u unix 0xf50998c0 20546 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 46u unix 0xf5138c40 20680 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 47u unix 0xdfd4fc40 99363 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 49u unix 0xf50ff000 20597 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 50u unix 0xf724f700 99470 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 51u unix 0xf50ffe00 20616 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 54u unix 0xf50fd8c0 20623 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 56u unix 0xf510c380 20649 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 58u unix 0xf6702c40 108612 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 60u unix 0xf50d7e00 20764 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 74u unix 0xf5255540 43572 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 88u unix 0xf407da80 25329 /tmp/.ICE-unix/dcop7162-1199959838 dcopserve 7162 jplatte 89u unix 0xf6702540 25375 /tmp/.ICE-unix/dcop7162-1199959838 klauncher 7164 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca klauncher 7164 jplatte 10r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca klauncher 7164 jplatte 11u unix 0xf5c89540 18999 /tmp/ksocket-jplatte/klauncherDCSyUa.slave-socket klauncher 7164 jplatte 12u unix 0xf5255a80 110308 /tmp/ksocket-jplatte/klauncherDCSyUa.slave-socket kded 7166 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kded 7166 jplatte 14r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca ksmserver 7183 jplatte 7u unix 0xf5ccc8c0 19076 /tmp/ksocket-jplatte/kdeinit__0 ksmserver 7183 jplatte 12u unix 0xf5c2b8c0 19093 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 13u unix 0xf5d1d1c0 19102 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 14u unix 0xf5d1d8c0 19110 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 15u unix 0xf725ca80 19127 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 16u unix 0xf5d298c0 19151 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 17u unix 0xf5db3a80 19256 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 18u unix 0xf5c2b000 19306 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 19u unix 0xdfdd7700 19472 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 20u unix 0xf5c7fc40 19384 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 21u unix 0xf5dddc40 19405 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 22u unix 0xf712a380 20290 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 23u unix 0xf5e5ca80 19504 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 24u unix 0xf5e7f1c0 19517 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 25u unix 0xf5e7fe00 19533 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 26u unix 0xf5ea6700 19984 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 27u unix 0xf5ea4380 19995 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 28u unix 0xf5d6aa80 43581 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 29u unix 0xf40f5700 108483 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 30u unix 0xdfd4f1c0 99358 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 31u unix 0xf5ee9e00 20381 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 32u unix 0xf5005700 20467 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 34u unix 0xf512c1c0 21389 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 35u unix 0xf50cbc40 20568 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 36u unix 0xf50e4000 20571 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 37u unix 0xf400e540 25338 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 38u unix 0xf50e4700 20575 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 39u unix 0xf50ffa80 20604 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 40u unix 0xf51381c0 20665 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 41u unix 0xf514b380 20693 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 42u unix 0xf514b540 20851 /tmp/.ICE-unix/7183 ksmserver 7183 jplatte 43u unix 0xf52c3700 25384 /tmp/.ICE-unix/7183 kicker 7192 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kicker 7192 jplatte 12r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kxkb 7215 jplatte 12r REG 8,7 11912 97730 /tmp/kde-jplatte/de..xkm konqueror 7255 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca konqueror 7255 jplatte 12r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca konqueror 7255 jplatte 13u REG 8,7 283 97732 /tmp/kde-jplatte/konqueror-crash-LPnWVb.log konqueror 7255 jplatte 18u REG 8,7 67736 97740 /tmp/kde-jplatte/khtmlcacheJfHsua.tmp (deleted) konqueror 7255 jplatte 21u REG 8,7 48368 97741 /tmp/kde-jplatte/khtmlcache8KkbLa.tmp (deleted) konqueror 7337 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca konqueror 7337 jplatte 12r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca konqueror 7337 jplatte 13u REG 8,7 102 97733 /tmp/kde-jplatte/konqueror-crash-YtoSma.log konqueror 7337 jplatte 16u REG 8,7 37907 97743 /tmp/kde-jplatte/khtmlcacheemjbkb.tmp (deleted) konqueror 7337 jplatte 22u REG 8,7 66887 97744 /tmp/kde-jplatte/khtmlcache1u4D4b.tmp (deleted) amarokapp 7351 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca amarokapp 7351 jplatte 18r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kopete 7447 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kopete 7447 jplatte 12r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kdeinit 7647 root cwd DIR 8,7 4096 130305 /tmp/1878472609 kdeinit 7647 root 7u unix 0xf51ce1c0 21255 /tmp/1878472609/.kde/socket-ibm/kdeinit__0 kdeinit 7647 root 8u unix 0xf67ab8c0 21257 /tmp/1878472609/.kde/socket-ibm/kdeinit- dcopserve 7651 root cwd DIR 8,7 4096 130305 /tmp/1878472609 dcopserve 7651 root 5u unix 0xf5297000 21279 /tmp/.ICE-unix/dcop7651-1199959912 dcopserve 7651 root 7u unix 0xf67abc40 21265 /tmp/.ICE-unix/dcop7651-1199959912 dcopserve 7651 root 10u unix 0xf52c3000 21308 /tmp/.ICE-unix/dcop7651-1199959912 dcopserve 7651 root 11u unix 0xdfd30c40 25104 /tmp/.ICE-unix/dcop7651-1199959912 klauncher 7653 root cwd DIR 8,7 4096 130305 /tmp/1878472609 klauncher 7653 root mem REG 8,7 130321 /tmp/1878472609/.kde/cache-ibm/ksycoca (path inode=130326) klauncher 7653 root 9r REG 8,7 2564654 130321 /tmp/1878472609/.kde/cache-ibm/ksycoca klauncher 7653 root 10u unix 0xf5297a80 21295 /tmp/1878472609/.kde/socket-ibm/klauncher1FOdCb.slave-socket kded 7662 root cwd DIR 8,7 4096 130305 /tmp/1878472609 kded 7662 root mem REG 8,7 2564654 130326 /tmp/1878472609/.kde/cache-ibm/ksycoca kded 7662 root 12r REG 8,7 2564654 130326 /tmp/1878472609/.kde/cache-ibm/ksycoca kio_smtp 7710 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kio_smtp 7710 jplatte 9r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kio_file 7786 jplatte mem REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca kio_file 7786 jplatte 9r REG 8,6 2590525 1336104 /var/tmp/kdecache-jplatte/ksycoca java 10922 jplatte mem REG 8,7 32768 179170 /tmp/hsperfdata_jplatte/10922 regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-11 5:29 ` Joerg Platte @ 2008-01-11 6:41 ` Joerg Platte 0 siblings, 0 replies; 59+ messages in thread From: Joerg Platte @ 2008-01-11 6:41 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel Am Freitag, 11. Januar 2008 schrieb Joerg Platte: > konqueror 987 jplatte mem REG 8,6 2590525 > 1336104 /var/tmp/kdecache-jplatte/ksycoca > konqueror 987 jplatte 12r REG 8,6 2590525 > 1336104 /var/tmp/kdecache-jplatte/ksycoca > konqueror 987 jplatte 13u REG 8,7 579 > 97731 /tmp/kde-jplatte/konqueror-crash-Jv2u8a.log > konqueror 987 jplatte 14u REG 8,7 128289 > 97734 /tmp/kde-jplatte/khtmlcacheA7VjAa.tmp (deleted) > konqueror 987 jplatte 16u REG 8,7 43334 > 97750 /tmp/kde-jplatte/khtmlcacheZUNlsc.tmp (deleted) > konqueror 987 jplatte 22u REG 8,7 797 > 97751 /tmp/kde-jplatte/khtmlcache76bZYa.tmp (deleted) > konqueror 987 jplatte 27u REG 8,7 102347 > 97735 /tmp/kde-jplatte/khtmlcacheMhlDJb.tmp (deleted) > konqueror 987 jplatte 31u REG 8,7 354 > 97738 /tmp/kde-jplatte/khtmlcacheq21uoc.tmp (deleted) > konqueror 987 jplatte 32u REG 8,7 1360 > 97742 /tmp/kde-jplatte/khtmlcacheo2Ms2a.tmp (deleted) > konqueror 987 jplatte 34u REG 8,7 6378 > 97745 /tmp/kde-jplatte/khtmlcacheLETtgc.tmp (deleted) > konqueror 987 jplatte 35u REG 8,7 97350 > 97746 /tmp/kde-jplatte/khtmlcache5jit8a.tmp (deleted) > konqueror 987 jplatte 36u REG 8,7 354 > 97747 /tmp/kde-jplatte/khtmlcache7VBSNa.tmp (deleted) > konqueror 987 jplatte 37u REG 8,7 1360 > 97748 /tmp/kde-jplatte/khtmlcachetSNbub.tmp (deleted) > konqueror 987 jplatte 38u REG 8,7 6073 > 97749 /tmp/kde-jplatte/khtmlcacheKAKhuc.tmp (deleted) Fengguang, a few minutes after killing this konqueror process the high iowait load has disappeared. Lets see if it'll come back... regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-11 4:43 ` Fengguang Wu 2008-01-11 5:29 ` Joerg Platte @ 2008-01-12 23:32 ` Joerg Platte [not found] ` <E1JDwaA-00017Q-W6@localhost.localdomain> 1 sibling, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-12 23:32 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel Am Freitag, 11. Januar 2008 schrieb Fengguang Wu: > On Thu, Jan 10, 2008 at 11:03:05AM +0100, Joerg Platte wrote: > > Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu: > > > > problem, because the iowait problem disappeared today after the > > > > regular Debian update. I'll try to install the old package versions > > > > to make it show up again. Maybe that helps to debug it. > > > > > > Thank you. I'm running sid, ext2 as rootfs now ;-) > > > > The error is back and I'm getting thousands of messages like this with > > the patched kernel: > > > > mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc > > _M > tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) > > mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc > > _M > tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) > > mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc > > _M > tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) > > Joerg, what's the output of `dumpe2fs /dev/sda7` and `lsof|grep /tmp`? After another reboot I tried to get more information about the konqueror process possibly causing the iowait load by using strace -p. Here is the output: gettimeofday({1200180588, 878508}, NULL) = 0 setitimer(ITIMER_VIRTUAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0 rt_sigaction(SIGVTALRM, {SIG_DFL}, {0xb5cffed0, [VTALRM], SA_RESTART}, 8) = 0 gettimeofday({1200180588, 879942}, NULL) = 0 time(NULL) = 1200180588 gettimeofday({1200180588, 880838}, NULL) = 0 gettimeofday({1200180588, 881284}, NULL) = 0 time(NULL) = 1200180588 gettimeofday({1200180588, 882131}, NULL) = 0 gettimeofday({1200180588, 882572}, NULL) = 0 ioctl(5, FIONREAD, [0]) = 0 gettimeofday({1200180588, 883477}, NULL) = 0 select(16, [5 6 7 9 11 14 15], [], [], {0, 150095}) = 0 (Timeout) gettimeofday({1200180589, 34269}, NULL) = 0 gettimeofday({1200180589, 34885}, NULL) = 0 time(NULL) = 1200180589 gettimeofday({1200180589, 36672}, NULL) = 0 rt_sigaction(SIGVTALRM, {0xb5cffed0, [VTALRM], SA_RESTART}, {SIG_DFL}, 8) = 0 setitimer(ITIMER_VIRTUAL, {it_interval={10, 0}, it_value={5, 0}}, {it_interval={0, 0}, it_value={0, 0}}) = 0 gettimeofday({1200180589, 38555}, NULL) = 0 time(NULL) = 1200180589 gettimeofday({1200180589, 39802}, NULL) = 0 setitimer(ITIMER_VIRTUAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0 rt_sigaction(SIGVTALRM, {SIG_DFL}, {0xb5cffed0, [VTALRM], SA_RESTART}, 8) = 0 gettimeofday({1200180589, 40912}, NULL) = 0 time(NULL) = 1200180589 gettimeofday({1200180589, 42019}, NULL) = 0 gettimeofday({1200180589, 42458}, NULL) = 0 time(NULL) = 1200180589 gettimeofday({1200180589, 43303}, NULL) = 0 gettimeofday({1200180589, 43747}, NULL) = 0 ioctl(5, FIONREAD, [0]) = 0 gettimeofday({1200180589, 45834}, NULL) = 0 select(16, [5 6 7 9 11 14 15], [], [], {0, 149913}) = 0 (Timeout) gettimeofday({1200180589, 194815}, NULL) = 0 ioctl(5, FIONREAD, [0]) = 0 gettimeofday({1200180589, 195730}, NULL) = 0 select(16, [5 6 7 9 11 14 15], [], [], {0, 17}) = 0 (Timeout) gettimeofday({1200180589, 197555}, NULL) = 0 gettimeofday({1200180589, 198020}, NULL) = 0 time(NULL) = 1200180589 gettimeofday({1200180589, 198884}, NULL) = 0 rt_sigaction(SIGVTALRM, {0xb5cffed0, [VTALRM], SA_RESTART}, {SIG_DFL}, 8) = 0 setitimer(ITIMER_VIRTUAL, {it_interval={10, 0}, it_value={5, 0}}, {it_interval={0, 0}, it_value={0, 0}}) = 0 gettimeofday({1200180589, 200702}, NULL) = 0 time(NULL) = 1200180589 gettimeofday({1200180589, 200806}, NULL) = 0 setitimer(ITIMER_VIRTUAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0 rt_sigaction(SIGVTALRM, {SIG_DFL}, {0xb5cffed0, [VTALRM], SA_RESTART}, 8) = 0 gettimeofday({1200180589, 202975}, NULL) = 0 time(NULL) = 1200180589 gettimeofday({1200180589, 203837}, NULL) = 0 gettimeofday({1200180589, 204319}, NULL) = 0 time(NULL) = 1200180589 gettimeofday({1200180589, 205169}, NULL) = 0 gettimeofday({1200180589, 205613}, NULL) = 0 ioctl(5, FIONREAD, [0]) = 0 gettimeofday({1200180589, 206515}, NULL) = 0 select(16, [5 6 7 9 11 14 15], [], [], {0, 149098} <unfinished ...> Fengguang, do you have any idea what's going wrong here? regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JDwaA-00017Q-W6@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JDwaA-00017Q-W6@localhost.localdomain> @ 2008-01-13 6:44 ` Fengguang Wu 2008-01-13 8:05 ` Joerg Platte 0 siblings, 1 reply; 59+ messages in thread From: Fengguang Wu @ 2008-01-13 6:44 UTC (permalink / raw) To: Joerg Platte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel [-- Attachment #1: Type: text/plain, Size: 2118 bytes --] On Sun, Jan 13, 2008 at 12:32:30AM +0100, Joerg Platte wrote: > Am Freitag, 11. Januar 2008 schrieb Fengguang Wu: > > On Thu, Jan 10, 2008 at 11:03:05AM +0100, Joerg Platte wrote: > > > Am Donnerstag, 10. Januar 2008 schrieb Fengguang Wu: > > > > > problem, because the iowait problem disappeared today after the > > > > > regular Debian update. I'll try to install the old package versions > > > > > to make it show up again. Maybe that helps to debug it. > > > > > > > > Thank you. I'm running sid, ext2 as rootfs now ;-) > > > > > > The error is back and I'm getting thousands of messages like this with > > > the patched kernel: > > > > > > mm/page-writeback.c 668 wb_kupdate: pdflush(146) 21115 global 3936 0 0 wc > > > _M > tw 1024 sk 0 requeue_io 301: inode 81441 size 0 at 08:07(sda7) > > > mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc > > > _M > tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) > > > mm/page-writeback.c 668 wb_kupdate: pdflush(147) 17451 global 3936 0 0 wc > > > _M > tw 1024 sk 2 requeue_io 301: inode 81441 size 0 at 08:07(sda7) > > > > Joerg, what's the output of `dumpe2fs /dev/sda7` and `lsof|grep /tmp`? > > After another reboot I tried to get more information about the konqueror > process possibly causing the iowait load by using strace -p. Here is the > output: > > gettimeofday({1200180588, 878508}, NULL) = 0 > setitimer(ITIMER_VIRTUAL, {it_interval={0, 0}, it_value={0, 0}}, NULL) = 0 > rt_sigaction(SIGVTALRM, {SIG_DFL}, {0xb5cffed0, [VTALRM], SA_RESTART}, 8) = 0 > gettimeofday({1200180588, 879942}, NULL) = 0 > time(NULL) = 1200180588 > gettimeofday({1200180588, 880838}, NULL) = 0 > gettimeofday({1200180588, 881284}, NULL) = 0 No idea yet :-/ I'm afraid I have to trouble you again - the bug just refused to appear in my system. I prepared a kernel module for you to gather more information: make && insmod ext2-writeback-debug.ko && sleep 1s && rmmod ext2-writeback-debug dmesg > ext2-writeback-debug.dmesg Please do it when 100% iowait appears, and send the dmesg file. Thank you, Fengguang [-- Attachment #2: Makefile --] [-- Type: text/plain, Size: 187 bytes --] obj-m := ext2-writeback-debug.o KDIR := /lib/modules/$(shell uname -r)/build PWD := $(shell pwd) default: $(MAKE) -C $(KDIR) SUBDIRS=$(PWD) modules clean: rm -f *.mod.c *.ko *.o [-- Attachment #3: ext2-writeback-debug.c --] [-- Type: text/x-csrc, Size: 3607 bytes --] #include <linux/kernel.h> #include <linux/module.h> #include <linux/mm.h> #include <linux/fs.h> #include <linux/writeback.h> #include <linux/kprobes.h> #include <linux/pagevec.h> void print_page(struct page *page) { printk(KERN_DEBUG "%lu\t%u\t%u\t%c%c%c%c%c\n", page->index, page_count(page), page_mapcount(page), PageUptodate(page) ? 'U' : '_', PageDirty(page) ? 'D' : '_', PageWriteback(page) ? 'W' : '_', PagePrivate(page) ? 'P' : '_', PageLocked(page) ? 'L' : '_'); } void print_writeback_control(struct writeback_control *wbc) { printk(KERN_DEBUG "global dirty %lu writeback %lu nfs %lu\n" "wbc flags %c%c towrite %ld skipped %ld\n", global_page_state(NR_FILE_DIRTY), global_page_state(NR_WRITEBACK), global_page_state(NR_UNSTABLE_NFS), wbc->encountered_congestion ? 'C':'_', wbc->more_io ? 'M':'_', wbc->nr_to_write, wbc->pages_skipped); } void print_inode_pages(struct inode *inode) { struct address_space *mapping = inode->i_mapping; struct pagevec pvec; int nr_pages; int i; pgoff_t index = 0; struct dentry *dentry; int dcount; char *dname; nr_pages = pagevec_lookup_tag(&pvec, mapping, &index, PAGECACHE_TAG_DIRTY, (pgoff_t)PAGEVEC_SIZE); if (list_empty(&inode->i_dentry)) { dname = ""; dcount = 0; } else { dentry = list_entry(inode->i_dentry.next, struct dentry, d_alias); dname = dentry->d_iname; dcount = atomic_read(&dentry->d_count); } printk(KERN_DEBUG "inode %lu(%s/%s) count %d,%d size %llu pages %lu\n", inode->i_ino, inode->i_sb->s_id, dname, atomic_read(&inode->i_count), dcount, i_size_read(inode), mapping->nrpages ); for (i = 0; i < nr_pages; i++) print_page(pvec.pages[i]); } int handler_pre(struct kprobe *p, struct pt_regs *regs) { static int count = 0; if (count++ < 10) dump_stack(); return 0; } void handler_post(struct kprobe *p, struct pt_regs *regs, unsigned long flags) { } static struct kprobe my_kprobe = { .pre_handler = handler_pre, .post_handler = handler_post, .symbol_name = "submit_bio" }; static int jdo_ext2_writepage(struct page *page, struct writeback_control *wbc) { struct inode * const inode = page->mapping->host; if (!i_size_read(inode)) { printk(KERN_DEBUG "ext2_writepage:\n"); print_page(page); print_writeback_control(wbc); } jprobe_return(); return 0; } static struct jprobe jprobe_ext2_writepage = { .entry = jdo_ext2_writepage, .kp.symbol_name = "ext2_writepage" }; static void jdo_requeue_io(struct inode *inode) { if (!i_size_read(inode)) { printk(KERN_DEBUG "requeue_io:\n"); print_inode_pages(inode); } jprobe_return(); } static struct jprobe jprobe_requeue_io = { .entry = jdo_requeue_io, .kp.symbol_name = "requeue_io" }; static int setup_kprobe(struct kprobe *kprobe) { int ret = register_kprobe(kprobe); printk("register_kprobe(%s) = %d\n", kprobe->symbol_name, ret); return ret; } static int setup_jprobe(struct jprobe *jprobe) { int ret = register_jprobe(jprobe); printk("register_jprobe(%s) = %d\n", jprobe->kp.symbol_name, ret); return ret; } static int __init jprobe_init(void) { int ret; ret = setup_jprobe(&jprobe_ext2_writepage); if (ret) return -1; ret = setup_jprobe(&jprobe_requeue_io); if (ret) return -1; ret = setup_kprobe(&my_kprobe); if (ret) return -1; return 0; } static void __exit jprobe_exit(void) { unregister_kprobe(&my_kprobe); unregister_jprobe(&jprobe_requeue_io); unregister_jprobe(&jprobe_ext2_writepage); } module_init(jprobe_init) module_exit(jprobe_exit) MODULE_LICENSE("GPL"); ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-13 6:44 ` Fengguang Wu @ 2008-01-13 8:05 ` Joerg Platte [not found] ` <E1JDy5a-0001al-Tk@localhost.localdomain> 0 siblings, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-13 8:05 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel Am Sonntag, 13. Januar 2008 schrieb Fengguang Wu: > No idea yet :-/ I'm afraid I have to trouble you again - the bug just > refused to appear in my system. I prepared a kernel module for you to > gather more information: > make && insmod ext2-writeback-debug.ko && sleep 1s && rmmod > ext2-writeback-debug dmesg > ext2-writeback-debug.dmesg Unfortunately, I unable to compile the module: make -C /lib/modules/2.6.24-rc7-ext2/build SUBDIRS=/export/src modules make[1]: Entering directory `/export/src/linux-2.6.24-rc7' CC [M] /export/src/ext2-writeback-debug.o /export/src/ext2-writeback-debug.c:89: error: variable ‘my_kprobe’ has initializer but incomplete type /export/src/ext2-writeback-debug.c:90: error: unknown field ‘pre_handler’ specified in initializer /export/src/ext2-writeback-debug.c:90: warning: excess elements in struct initializer /export/src/ext2-writeback-debug.c:90: warning: (near initialization for ‘my_kprobe’) /export/src/ext2-writeback-debug.c:91: error: unknown field ‘post_handler’ specified in initializer /export/src/ext2-writeback-debug.c:91: warning: excess elements in struct initializer /export/src/ext2-writeback-debug.c:91: warning: (near initialization for ‘my_kprobe’) /export/src/ext2-writeback-debug.c:92: error: unknown field ‘symbol_name’ specified in initializer /export/src/ext2-writeback-debug.c:93: warning: excess elements in struct initializer /export/src/ext2-writeback-debug.c:93: warning: (near initialization for ‘my_kprobe’) /export/src/ext2-writeback-debug.c:109: error: variable ‘jprobe_ext2_writepage’ has initializer but incomplete type /export/src/ext2-writeback-debug.c:110: error: unknown field ‘entry’ specified in initializer /export/src/ext2-writeback-debug.c:110: warning: excess elements in struct initializer /export/src/ext2-writeback-debug.c:110: warning: (near initialization for ‘jprobe_ext2_writepage’) /export/src/ext2-writeback-debug.c:111: error: unknown field ‘kp’ specified in initializer /export/src/ext2-writeback-debug.c:112: warning: excess elements in struct initializer /export/src/ext2-writeback-debug.c:112: warning: (near initialization for ‘jprobe_ext2_writepage’) /export/src/ext2-writeback-debug.c:124: error: variable ‘jprobe_requeue_io’ has initializer but incomplete type /export/src/ext2-writeback-debug.c:125: error: unknown field ‘entry’ specified in initializer /export/src/ext2-writeback-debug.c:125: warning: excess elements in struct initializer /export/src/ext2-writeback-debug.c:125: warning: (near initialization for ‘jprobe_requeue_io’) /export/src/ext2-writeback-debug.c:126: error: unknown field ‘kp’ specified in initializer /export/src/ext2-writeback-debug.c:127: warning: excess elements in struct initializer /export/src/ext2-writeback-debug.c:127: warning: (near initialization for ‘jprobe_requeue_io’) /export/src/ext2-writeback-debug.c: In function ‘setup_kprobe’: /export/src/ext2-writeback-debug.c:132: error: dereferencing pointer to incomplete type /export/src/ext2-writeback-debug.c: In function ‘setup_jprobe’: /export/src/ext2-writeback-debug.c:139: error: dereferencing pointer to incomplete type make[2]: *** [/export/src/ext2-writeback-debug.o] Error 1 make[1]: *** [_module_/export/src] Error 2 make[1]: Leaving directory `/export/src/linux-2.6.24-rc7' make: *** [default] Error 2 regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JDy5a-0001al-Tk@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JDy5a-0001al-Tk@localhost.localdomain> @ 2008-01-13 8:21 ` Fengguang Wu 2008-01-13 9:49 ` Joerg Platte 0 siblings, 1 reply; 59+ messages in thread From: Fengguang Wu @ 2008-01-13 8:21 UTC (permalink / raw) To: Joerg Platte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel On Sun, Jan 13, 2008 at 09:05:43AM +0100, Joerg Platte wrote: > Am Sonntag, 13. Januar 2008 schrieb Fengguang Wu: > > > No idea yet :-/ I'm afraid I have to trouble you again - the bug just > > refused to appear in my system. I prepared a kernel module for you to > > gather more information: > > > make && insmod ext2-writeback-debug.ko && sleep 1s && rmmod > > ext2-writeback-debug dmesg > ext2-writeback-debug.dmesg > > Unfortunately, I unable to compile the module: > make -C /lib/modules/2.6.24-rc7-ext2/build SUBDIRS=/export/src modules > make[1]: Entering directory `/export/src/linux-2.6.24-rc7' > CC [M] /export/src/ext2-writeback-debug.o > /export/src/ext2-writeback-debug.c:89: error: variable ‘my_kprobe’ has > initializer but incomplete type Do you have CONFIG_KPROBES=y? ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-13 8:21 ` Fengguang Wu @ 2008-01-13 9:49 ` Joerg Platte [not found] ` <E1JE1Uz-0002w5-6z@localhost.localdomain> [not found] ` <20080113115933.GA11045@mail.ustc.edu.cn> 0 siblings, 2 replies; 59+ messages in thread From: Joerg Platte @ 2008-01-13 9:49 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel [-- Attachment #1: Type: text/plain, Size: 1060 bytes --] Am Sonntag, 13. Januar 2008 schrieb Fengguang Wu: > On Sun, Jan 13, 2008 at 09:05:43AM +0100, Joerg Platte wrote: > > Am Sonntag, 13. Januar 2008 schrieb Fengguang Wu: > > > No idea yet :-/ I'm afraid I have to trouble you again - the bug just > > > refused to appear in my system. I prepared a kernel module for you to > > > gather more information: > > > > > > make && insmod ext2-writeback-debug.ko && sleep 1s && rmmod > > > ext2-writeback-debug dmesg > ext2-writeback-debug.dmesg > > > > Unfortunately, I unable to compile the module: > > > > make -C /lib/modules/2.6.24-rc7-ext2/build SUBDIRS=/export/src modules > > make[1]: Entering directory `/export/src/linux-2.6.24-rc7' > > CC [M] /export/src/ext2-writeback-debug.o > > /export/src/ext2-writeback-debug.c:89: error: variable ‘my_kprobe’ has > > initializer but incomplete type > > Do you have CONFIG_KPROBES=y? Now I have :) Here is the result. regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 [-- Attachment #2: ext2-writeback-debug.dmesg --] [-- Type: text/plain, Size: 49111 bytes --] Linux version 2.6.24-rc7-ext2 (root@ibm) (gcc version 4.2.3 20080102 (prerelease) (Debian 4.2.2-5)) #1 PREEMPT Sun Jan 13 09:48:55 CET 2008 BIOS-provided physical RAM map: BIOS-e820: 0000000000000000 - 000000000009f000 (usable) BIOS-e820: 000000000009f000 - 00000000000a0000 (reserved) BIOS-e820: 00000000000d2000 - 00000000000d4000 (reserved) BIOS-e820: 00000000000dc000 - 0000000000100000 (reserved) BIOS-e820: 0000000000100000 - 000000005ff50000 (usable) BIOS-e820: 000000005ff50000 - 000000005ff67000 (ACPI data) BIOS-e820: 000000005ff67000 - 000000005ff79000 (ACPI NVS) BIOS-e820: 000000005ff80000 - 0000000060000000 (reserved) BIOS-e820: 00000000ff800000 - 0000000100000000 (reserved) 639MB HIGHMEM available. 896MB LOWMEM available. Entering add_active_range(0, 0, 393040) 0 entries of 256 used Zone PFN ranges: DMA 0 -> 4096 Normal 4096 -> 229376 HighMem 229376 -> 393040 Movable zone start PFN for each node early_node_map[1] active PFN ranges 0: 0 -> 393040 On node 0 totalpages: 393040 DMA zone: 32 pages used for memmap DMA zone: 0 pages reserved DMA zone: 4064 pages, LIFO batch:0 Normal zone: 1760 pages used for memmap Normal zone: 223520 pages, LIFO batch:31 HighMem zone: 1278 pages used for memmap HighMem zone: 162386 pages, LIFO batch:31 Movable zone: 0 pages used for memmap DMI present. ACPI: RSDP 000F6D70, 0024 (r2 IBM ) ACPI: XSDT 5FF5A672, 004C (r1 IBM TP-1R 3230 LTP 0) ACPI: FACP 5FF5A700, 00F4 (r3 IBM TP-1R 3230 IBM 1) ACPI Warning (tbfadt-0442): Optional field "Gpe1Block" has zero address or length: 000000000000102C/0 [20070126] ACPI: DSDT 5FF5A8E7, C530 (r1 IBM TP-1R 3230 MSFT 100000E) ACPI: FACS 5FF78000, 0040 ACPI: SSDT 5FF5A8B4, 0033 (r1 IBM TP-1R 3230 MSFT 100000E) ACPI: ECDT 5FF66E17, 0052 (r1 IBM TP-1R 3230 IBM 1) ACPI: TCPA 5FF66E69, 0032 (r1 IBM TP-1R 3230 PTL 1) ACPI: BOOT 5FF66FD8, 0028 (r1 IBM TP-1R 3230 LTP 1) ACPI: PM-Timer IO Port: 0x1008 Allocating PCI resources starting at 70000000 (gap: 60000000:9f800000) swsusp: Registered nosave memory region: 000000000009f000 - 00000000000a0000 swsusp: Registered nosave memory region: 00000000000a0000 - 00000000000d2000 swsusp: Registered nosave memory region: 00000000000d2000 - 00000000000d4000 swsusp: Registered nosave memory region: 00000000000d4000 - 00000000000dc000 swsusp: Registered nosave memory region: 00000000000dc000 - 0000000000100000 Built 1 zonelists in Zone order, mobility grouping on. Total pages: 389970 Kernel command line: root=LABEL=ROOT ro resume=LABEL=SWAP panic=10 usbcore.autosuspend=1 hpet=force Unknown boot option `usbcore.autosuspend=1': ignoring Local APIC disabled by BIOS -- you can enable it with "lapic" mapped APIC to ffffb000 (01c06000) Enabling fast FPU save and restore... done. Enabling unmasked SIMD FPU exception support... done. Initializing CPU#0 CPU 0 irqstacks, hard=c0343000 soft=c0342000 PID hash table entries: 4096 (order: 12, 16384 bytes) Detected 1598.683 MHz processor. Console: colour VGA+ 80x25 console [tty0] enabled Dentry cache hash table entries: 131072 (order: 7, 524288 bytes) Inode-cache hash table entries: 65536 (order: 6, 262144 bytes) Memory: 1547564k/1572160k available (1494k kernel code, 23352k reserved, 597k data, 204k init, 654656k highmem) virtual kernel memory layout: fixmap : 0xfffa8000 - 0xfffff000 ( 348 kB) pkmap : 0xff800000 - 0xffc00000 (4096 kB) vmalloc : 0xf8800000 - 0xff7fe000 ( 111 MB) lowmem : 0xc0000000 - 0xf8000000 ( 896 MB) .init : 0xc030c000 - 0xc033f000 ( 204 kB) .data : 0xc0275a5f - 0xc030aef0 ( 597 kB) .text : 0xc0100000 - 0xc0275a5f (1494 kB) Checking if this processor honours the WP bit even in supervisor mode... Ok. SLUB: Genslabs=11, HWalign=64, Order=0-1, MinObjects=4, CPUs=1, Nodes=1 Calibrating delay using timer specific routine.. 3199.89 BogoMIPS (lpj=5331036) Security Framework initialized Capability LSM initialized Mount-cache hash table entries: 512 CPU: After generic identify, caps: a7e9f9bf 00000000 00000000 00000000 00000180 00000000 00000000 00000000 CPU: L1 I cache: 32K, L1 D cache: 32K CPU: L2 cache: 1024K CPU: After all inits, caps: a7e9f9bf 00000000 00000000 00002040 00000180 00000000 00000000 00000000 Intel machine check architecture supported. Intel machine check reporting enabled on CPU#0. Compat vDSO mapped to ffffe000. CPU: Intel(R) Pentium(R) M processor 1600MHz stepping 05 Checking 'hlt' instruction... OK. Freeing SMP alternatives: 0k freed ACPI: Core revision 20070126 ACPI: setting ELCR to 0200 (from 0800) net_namespace: 64 bytes NET: Registered protocol family 16 ACPI: bus type pci registered PCI: PCI BIOS revision 2.10 entry at 0xfd8d6, last bus=8 PCI: Using configuration type 1 Setting up standard PCI resources ACPI: EC: EC description table is found, configuring boot EC ACPI: EC: non-query interrupt received, switching to interrupt mode ACPI: Interpreter enabled ACPI: (supports S0 S3 S4 S5) ACPI: Using PIC for interrupt routing ACPI: EC: GPE = 0x1c, I/O: command/status = 0x66, data = 0x62 ACPI: EC: driver started in poll mode ACPI: PCI Root Bridge [PCI0] (0000:00) Force enabled HPET at base address 0xfed00000 PCI quirk: region 1000-107f claimed by ICH4 ACPI/GPIO/TCO PCI quirk: region 1180-11bf claimed by ICH4 GPIO PCI: Transparent bridge - 0000:00:1e.0 ACPI: PCI Interrupt Routing Table [\_SB_.PCI0._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.AGP_._PRT] ACPI: PCI Interrupt Routing Table [\_SB_.PCI0.PCI1._PRT] ACPI: PCI Interrupt Link [LNKA] (IRQs 3 4 5 6 7 9 10 *11) ACPI: PCI Interrupt Link [LNKB] (IRQs 3 4 5 6 7 9 10 *11) ACPI: PCI Interrupt Link [LNKC] (IRQs 3 4 5 6 7 9 10 *11) ACPI: PCI Interrupt Link [LNKD] (IRQs 3 4 5 6 7 9 10 *11) ACPI: PCI Interrupt Link [LNKE] (IRQs 3 4 5 6 7 9 10 11) *0, disabled. ACPI: PCI Interrupt Link [LNKF] (IRQs 3 4 5 6 7 9 10 11) *0, disabled. ACPI: PCI Interrupt Link [LNKG] (IRQs 3 4 5 6 7 9 10 11) *0, disabled. ACPI: PCI Interrupt Link [LNKH] (IRQs 3 4 5 6 7 9 10 *11) ACPI: EC: non-query interrupt received, switching to interrupt mode ACPI: Power Resource [PUBS] (on) Linux Plug and Play Support v0.97 (c) Adam Belay pnp: PnP ACPI init ACPI: bus type pnp registered pnpacpi: exceeded the max number of mem resources: 12 pnpacpi: exceeded the max number of mem resources: 12 pnpacpi: exceeded the max number of mem resources: 12 pnpacpi: exceeded the max number of mem resources: 12 pnpacpi: exceeded the max number of IO resources: 24 pnp: PnP ACPI: found 13 devices ACPI: ACPI bus type pnp unregistered PnPBIOS: Disabled by ACPI PNP PCI: Using ACPI for IRQ routing PCI: If a device doesn't work, try "pci=routeirq". If it helps, post a report NetLabel: Initializing NetLabel: domain hash size = 128 NetLabel: protocols = UNLABELED CIPSOv4 NetLabel: unlabeled traffic allowed by default hpet clockevent registered hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 hpet0: 3 64-bit timers, 14318180 Hz ACPI: RTC can wake from S4 Time: tsc clocksource has been installed. system 00:00: iomem range 0x0-0x9ffff could not be reserved system 00:00: iomem range 0xc0000-0xc3fff could not be reserved system 00:00: iomem range 0xc4000-0xc7fff could not be reserved system 00:00: iomem range 0xc8000-0xcbfff could not be reserved system 00:00: iomem range 0xcc000-0xcffff could not be reserved system 00:00: iomem range 0xd0000-0xd3fff could not be reserved system 00:00: iomem range 0x0-0x0 could not be reserved system 00:00: iomem range 0x0-0x0 could not be reserved system 00:00: iomem range 0xdc000-0xdffff has been reserved system 00:00: iomem range 0xe0000-0xe3fff could not be reserved system 00:00: iomem range 0xe4000-0xe7fff could not be reserved system 00:00: iomem range 0xe8000-0xebfff could not be reserved system 00:02: ioport range 0x1000-0x107f has been reserved system 00:02: ioport range 0x1180-0x11bf has been reserved system 00:02: ioport range 0x15e0-0x15ef has been reserved system 00:02: ioport range 0x1600-0x162f has been reserved system 00:02: ioport range 0x1632-0x167f has been reserved PCI: Bridge: 0000:00:01.0 IO window: 3000-3fff MEM window: c0100000-c01fffff PREFETCH window: e0000000-e7ffffff PCI: Bus 3, cardbus bridge: 0000:02:00.0 IO window: 00004000-000040ff IO window: 00004400-000044ff PREFETCH window: e8000000-ebffffff MEM window: c4000000-c7ffffff PCI: Bus 7, cardbus bridge: 0000:02:00.1 IO window: 00004800-000048ff IO window: 00004c00-00004cff PREFETCH window: ec000000-efffffff MEM window: c8000000-cbffffff PCI: Bridge: 0000:00:1e.0 IO window: 4000-8fff MEM window: c0200000-cfffffff PREFETCH window: e8000000-efffffff PCI: Setting latency timer of device 0000:00:1e.0 to 64 ACPI: PCI Interrupt Link [LNKA] enabled at IRQ 11 PCI: setting IRQ 11 as level-triggered ACPI: PCI Interrupt 0000:02:00.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11 ACPI: PCI Interrupt Link [LNKB] enabled at IRQ 11 ACPI: PCI Interrupt 0000:02:00.1[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11 NET: Registered protocol family 2 IP route cache hash table entries: 32768 (order: 5, 131072 bytes) TCP established hash table entries: 131072 (order: 8, 1048576 bytes) TCP bind hash table entries: 65536 (order: 6, 262144 bytes) TCP: Hash tables configured (established 131072 bind 65536) TCP reno registered checking if image is initramfs... it is Switched to high resolution mode on CPU 0 Freeing initrd memory: 6682k freed Simple Boot Flag at 0x35 set to 0x1 highmem bounce pool size: 64 pages Total HugeTLB memory allocated, 0 VFS: Disk quotas dquot_6.5.1 Dquot-cache hash table entries: 1024 (order 0, 4096 bytes) Block layer SCSI generic (bsg) driver version 0.4 loaded (major 254) io scheduler noop registered io scheduler cfq registered (default) Boot video device is 0000:01:00.0 isapnp: Scanning for PnP cards... isapnp: No Plug & Play device found Real Time Clock Driver v1.12ac RAMDISK driver initialized: 16 RAM disks of 4096K size 1024 blocksize PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 serio: i8042 KBD port at 0x60,0x64 irq 1 serio: i8042 AUX port at 0x60,0x64 irq 12 mice: PS/2 mouse device common for all mice cpuidle: using governor ladder cpuidle: using governor menu Using IPI Shortcut mode registered taskstats version 1 Freeing unused kernel memory: 204k freed input: AT Translated Set 2 keyboard as /class/input/input0 ACPI: PCI Interrupt 0000:01:00.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11 radeonfb: Retrieved PLL infos from BIOS radeonfb: Reference=27.00 MHz (RefDiv=12) Memory=252.00 Mhz, System=200.00 MHz radeonfb: PLL min 20000 max 35000 i2c-adapter i2c-1: unable to read EDID block. i2c-adapter i2c-1: unable to read EDID block. i2c-adapter i2c-1: unable to read EDID block. i2c-adapter i2c-3: unable to read EDID block. i2c-adapter i2c-3: unable to read EDID block. i2c-adapter i2c-3: unable to read EDID block. Non-DDC laptop panel detected i2c-adapter i2c-2: unable to read EDID block. i2c-adapter i2c-2: unable to read EDID block. i2c-adapter i2c-2: unable to read EDID block. i2c-adapter i2c-3: unable to read EDID block. i2c-adapter i2c-3: unable to read EDID block. i2c-adapter i2c-3: unable to read EDID block. radeonfb: Monitor 1 type LCD found radeonfb: Monitor 2 type no found radeonfb: panel ID string: SXGA+ Single (85MHz) radeonfb: detected LVDS panel size from BIOS: 1400x1050 radeondb: BIOS provided dividers will be used radeonfb: Dynamic Clock Power Management enabled radeonfb: IBM Thinkpad T40p detected, enabling workaround radeonfb (0000:01:00.0): ATI Radeon Lf Console: switching to colour frame buffer device 175x65 Non-volatile memory driver v1.2 thinkpad_acpi: ThinkPad ACPI Extras v0.17 thinkpad_acpi: http://ibm-acpi.sf.net/ thinkpad_acpi: ThinkPad BIOS 1RETDRWW (3.23 ), EC 1RHT71WW-3.04 thinkpad_acpi: IBM ThinkPad T40p input: ThinkPad Extra Buttons as /class/input/input1 NET: Registered protocol family 1 ACPI: CPU0 (power states: C1[C1] C2[C2] C3[C3]) ACPI: Processor [CPU] (supports 8 throttling states) Marking TSC unstable due to: TSC halts in idle. Time: hpet clocksource has been installed. ACPI: Thermal Zone [THM0] (52 C) usbcore: registered new interface driver usbfs usbcore: registered new interface driver hub usbcore: registered new device driver usb USB Universal Host Controller Interface driver v3.0 ACPI: PCI Interrupt 0000:00:1d.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1d.0 to 64 uhci_hcd 0000:00:1d.0: UHCI Host Controller uhci_hcd 0000:00:1d.0: new USB bus registered, assigned bus number 1 uhci_hcd 0000:00:1d.0: irq 11, io base 0x00001800 usb usb1: configuration #1 chosen from 1 choice hub 1-0:1.0: USB hub found hub 1-0:1.0: 2 ports detected ACPI: PCI Interrupt Link [LNKD] enabled at IRQ 11 ACPI: PCI Interrupt 0000:00:1d.1[B] -> Link [LNKD] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1d.1 to 64 uhci_hcd 0000:00:1d.1: UHCI Host Controller uhci_hcd 0000:00:1d.1: new USB bus registered, assigned bus number 2 uhci_hcd 0000:00:1d.1: irq 11, io base 0x00001820 usb usb2: configuration #1 chosen from 1 choice hub 2-0:1.0: USB hub found hub 2-0:1.0: 2 ports detected SCSI subsystem initialized ACPI: PCI Interrupt Link [LNKC] enabled at IRQ 11 ACPI: PCI Interrupt 0000:00:1d.2[C] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1d.2 to 64 uhci_hcd 0000:00:1d.2: UHCI Host Controller uhci_hcd 0000:00:1d.2: new USB bus registered, assigned bus number 3 uhci_hcd 0000:00:1d.2: irq 11, io base 0x00001840 usb usb3: configuration #1 chosen from 1 choice hub 3-0:1.0: USB hub found hub 3-0:1.0: 2 ports detected Intel(R) PRO/1000 Network Driver - version 7.3.20-k2 Copyright (c) 1999-2006 Intel Corporation. libata version 3.00 loaded. ACPI: PCI Interrupt Link [LNKH] enabled at IRQ 11 ACPI: PCI Interrupt 0000:00:1d.7[D] -> Link [LNKH] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1d.7 to 64 ehci_hcd 0000:00:1d.7: EHCI Host Controller ehci_hcd 0000:00:1d.7: new USB bus registered, assigned bus number 4 ehci_hcd 0000:00:1d.7: debug port 1 PCI: cache line size of 32 is not supported by device 0000:00:1d.7 ehci_hcd 0000:00:1d.7: irq 11, io mem 0xc0000000 usb 1-1: new full speed USB device using uhci_hcd and address 2 ehci_hcd 0000:00:1d.7: USB 2.0 started, EHCI 1.00, driver 10 Dec 2004 usb usb4: configuration #1 chosen from 1 choice hub 4-0:1.0: USB hub found hub 4-0:1.0: 6 ports detected Floppy drive(s): fd0 is 1.44M FDC 0 is a National Semiconductor PC87306 ACPI: PCI Interrupt 0000:02:01.0[A] -> Link [LNKA] -> GSI 11 (level, low) -> IRQ 11 usb 4-1: new high speed USB device using ehci_hcd and address 2 usb 4-1: configuration #1 chosen from 1 choice hub 4-1:1.0: USB hub found hub 4-1:1.0: 4 ports detected e1000: 0000:02:01.0: e1000_validate_option: PHY Smart Power Down Enabled e1000: 0000:02:01.0: e1000_probe: (PCI:33MHz:32-bit) 00:16:41:10:19:11 e1000: eth0: e1000_probe: Intel(R) PRO/1000 Network Connection ata_piix 0000:00:1f.1: version 2.12 PCI: Enabling device 0000:00:1f.1 (0005 -> 0007) ACPI: PCI Interrupt 0000:00:1f.1[A] -> Link [LNKC] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1f.1 to 64 scsi0 : ata_piix scsi1 : ata_piix ata1: PATA max UDMA/100 cmd 0x1f0 ctl 0x3f6 bmdma 0x1860 irq 14 ata2: PATA max UDMA/100 cmd 0x170 ctl 0x376 bmdma 0x1868 irq 15 usb 4-1.1: new high speed USB device using ehci_hcd and address 3 usb 4-1.1: configuration #1 chosen from 1 choice ata1.00: ATA-7: SAMSUNG HM160JC, AP100-16, max UDMA/100 ata1.00: 312581808 sectors, multi 16: LBA48 ata1.00: configured for UDMA/100 usb 4-1.4: new low speed USB device using ehci_hcd and address 4 usb 4-1.4: configuration #1 chosen from 1 choice usbcore: registered new interface driver hiddev input: Logitech USB-PS/2 Optical Mouse as /class/input/input2 input: USB HID v1.10 Mouse [Logitech USB-PS/2 Optical Mouse] on usb-0000:00:1d.7-1.4 usbcore: registered new interface driver usbhid drivers/hid/usbhid/hid-core.c: v2.6:USB HID core driver ata2.00: ATAPI: UJDA745 DVD/CDRW, 1.06, max UDMA/33 ata2.00: configured for UDMA/33 scsi 0:0:0:0: Direct-Access ATA SAMSUNG HM160JC AP10 PQ: 0 ANSI: 5 scsi 1:0:0:0: CD-ROM MATSHITA UJDA745 DVD/CDRW 1.06 PQ: 0 ANSI: 5 Driver 'sd' needs updating - please use bus_type methods sd 0:0:0:0: [sda] 312581808 512-byte hardware sectors (160042 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sd 0:0:0:0: [sda] 312581808 512-byte hardware sectors (160042 MB) sd 0:0:0:0: [sda] Write Protect is off sd 0:0:0:0: [sda] Mode Sense: 00 3a 00 00 sd 0:0:0:0: [sda] Write cache: enabled, read cache: enabled, doesn't support DPO or FUA sda:<4>Driver 'sr' needs updating - please use bus_type methods sda1 sda2 < sda5 sda6 sda7 sda8 > sd 0:0:0:0: [sda] Attached SCSI disk sr0: scsi3-mmc drive: 24x/24x writer cd/rw xa/form2 cdda tray Uniform CD-ROM driver Revision: 3.20 sr 1:0:0:0: Attached scsi CD-ROM sr0 sd 0:0:0:0: Attached scsi generic sg0 type 0 sr 1:0:0:0: Attached scsi generic sg1 type 5 Attempting manual resume swsusp: Marking nosave pages: 000000000009f000 - 0000000000100000 swsusp: Basic memory bitmaps created swsusp: Basic memory bitmaps freed kjournald starting. Commit interval 5 seconds EXT3-fs: mounted filesystem with ordered data mode. Linux agpgart interface v0.102 agpgart: Detected an Intel 855PM Chipset. agpgart: AGP aperture is 256M @ 0xd0000000 input: Power Button (FF) as /class/input/input3 ACPI: Power Button (FF) [PWRF] input: Lid Switch as /class/input/input4 ACPI: Lid Switch [LID] input: Sleep Button (CM) as /class/input/input5 ACPI: Sleep Button (CM) [SLPB] input: Video Bus as /class/input/input6 ACPI: Video Device [VID] (multi-head: yes rom: no post: no) ACPI: AC Adapter [AC] (on-line) ACPI: Battery Slot [BAT0] (battery present) iTCO_vendor_support: vendor-support=0 iTCO_wdt: Intel TCO WatchDog Timer Driver v1.02 (26-Jul-2007) iTCO_wdt: Found a ICH4-M TCO device (Version=1, TCOBASE=0x1060) iTCO_wdt: initialized. heartbeat=30 sec (nowayout=0) intel_rng: FWH not detected ACPI: PCI Interrupt 0000:00:1f.3[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11 DVB: registering new adapter (TerraTec/qanu USB2.0 Highspeed DVB-T Receiver) input: TerraTec/qanu USB2.0 Highspeed DVB-T Receiver remote control as /class/input/input7 usbcore: registered new interface driver cinergyT2 input: PC Speaker as /class/input/input8 Yenta: CardBus bridge found at 0000:02:00.0 [1014:0512] Yenta: Using INTVAL to route CSC interrupts to PCI Yenta: Routing CardBus interrupts to PCI Yenta TI: socket 0000:02:00.0, mfunc 0x01d21022, devctl 0x64 Serial: 8250/16550 driver $Revision: 1.90 $ 2 ports, IRQ sharing disabled serial8250: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A Yenta: ISA IRQ mask 0x04b8, PCI irq 11 Socket status: 30000006 pcmcia: parent PCI bridge I/O window: 0x4000 - 0x8fff cs: IO port probe 0x4000-0x8fff: clean. pcmcia: parent PCI bridge Memory window: 0xc0200000 - 0xcfffffff pcmcia: parent PCI bridge Memory window: 0xe8000000 - 0xefffffff 00:0a: ttyS0 at I/O 0x3f8 (irq = 4) is a NS16550A NET: Registered protocol family 23 Yenta: CardBus bridge found at 0000:02:00.1 [1014:0512] Yenta: Using INTVAL to route CSC interrupts to PCI Yenta: Routing CardBus interrupts to PCI Yenta TI: socket 0000:02:00.1, mfunc 0x01d21022, devctl 0x64 Synaptics Touchpad, model: 1, fw: 5.9, id: 0x2c6ab1, caps: 0x884793/0x0 serio: Synaptics pass-through port at isa0060/serio1/input0 input: SynPS/2 Synaptics TouchPad as /class/input/input9 Yenta: ISA IRQ mask 0x04b8, PCI irq 11 Socket status: 30000006 pcmcia: parent PCI bridge I/O window: 0x4000 - 0x8fff cs: IO port probe 0x4000-0x8fff: clean. pcmcia: parent PCI bridge Memory window: 0xc0200000 - 0xcfffffff pcmcia: parent PCI bridge Memory window: 0xe8000000 - 0xefffffff parport_pc 00:0b: reported by Plug and Play ACPI parport0: PC-style at 0x3bc (0x7bc), irq 7, dma 0 [PCSPP,TRISTATE,COMPAT,ECP,DMA] nsc-ircc 00:0c: activated nsc-ircc, chip->init nsc-ircc, Found chip at base=0x02e nsc-ircc, driver loaded (Dag Brattli) IrDA: Registered device irda0 nsc-ircc, Using dongle: IBM31T1100 or Temic TFDS6000/TFDS6500 ACPI: PCI Interrupt 0000:00:1f.6[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11 ACPI: PCI interrupt for device 0000:00:1f.6 disabled ACPI: PCI Interrupt 0000:00:1f.5[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1f.5 to 64 cs: IO port probe 0xc00-0xcff: clean. cs: IO port probe 0x800-0x8ff: clean. cs: IO port probe 0x100-0x4ff: excluding 0x4d0-0x4d7 cs: IO port probe 0xa00-0xaff: clean. cs: IO port probe 0xc00-0xcff: clean. cs: IO port probe 0x800-0x8ff: clean. cs: IO port probe 0x100-0x4ff: excluding 0x4d0-0x4d7 cs: IO port probe 0xa00-0xaff: clean. intel8x0_measure_ac97_clock: measured 53254 usecs intel8x0: measured clock 206 rejected intel8x0: clocking to 48000 ACPI: PCI Interrupt 0000:00:1f.6[B] -> Link [LNKB] -> GSI 11 (level, low) -> IRQ 11 PCI: Setting latency timer of device 0000:00:1f.6 to 64 Adding 1951856k swap on /dev/sda5. Priority:-1 extents:1 across:1951856k EXT3 FS on sda6, internal journal tun: Universal TUN/TAP device driver, 1.6 tun: (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com> IrCOMM protocol (Dag Brattli) io scheduler anticipatory registered io scheduler deadline registered Ebtables v2.0 registered Bridge firewalling registered fuse init (API version 7.9) EXT2-fs warning (device sda7): ext2_fill_super: mounting ext3 filesystem as ext2 kjournald starting. Commit interval 5 seconds EXT3 FS on sda8, internal journal EXT3-fs: mounted filesystem with ordered data mode. NTFS driver 2.1.29 [Flags: R/O MODULE]. NTFS volume version 3.1. IBM TrackPoint firmware: 0x0e, buttons: 3/3 input: TPPS/2 IBM TrackPoint as /class/input/input10 NET: Registered protocol family 15 padlock: VIA PadLock Hash Engine not detected. padlock: VIA PadLock Hash Engine not detected. padlock: VIA PadLock not detected. RPC: Registered udp transport module. RPC: Registered tcp transport module. nf_conntrack version 0.5.0 (16384 buckets, 65536 max) ip_tables: (C) 2000-2006 Netfilter Core Team Clocksource tsc unstable (delta = -197922093 ns) e1000: eth0: e1000_watchdog: NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX/TX NET: Registered protocol family 17 Installing knfsd (copyright (C) 1996 okir@monad.swb.de). NFSD: Using /var/lib/nfs/v4recovery as the NFSv4 state recovery directory NFSD: starting 90-second grace period Bluetooth: Core ver 2.11 NET: Registered protocol family 31 Bluetooth: HCI device and connection manager initialized Bluetooth: HCI socket layer initialized Bluetooth: L2CAP ver 2.9 Bluetooth: L2CAP socket layer initialized Bluetooth: RFCOMM socket layer initialized Bluetooth: RFCOMM TTY layer initialized Bluetooth: RFCOMM ver 1.8 vmmon: disagrees about version of symbol struct_module vmnet: disagrees about version of symbol struct_module [drm] Initialized drm 1.1.0 20060810 [drm] Initialized radeon 1.28.0 20060524 on minor 0 agpgart: Found an AGP 2.0 compliant device at 0000:00:00.0. agpgart: Putting AGP V2 device at 0000:00:00.0 into 4x mode agpgart: Putting AGP V2 device at 0000:01:00.0 into 4x mode [drm] Setting GART location based on new memory map [drm] Loading R200 Microcode [drm] writeback test succeeded in 2 usecs register_jprobe(ext2_writepage) = 0 register_jprobe(requeue_io) = 0 register_kprobe(submit_bio) = 0 requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 2 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 2 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 2 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 2 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 3 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 3 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 3 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 3 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 4 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 4 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 4 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 4 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 5 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 5 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 5 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 5 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 6 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 6 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 6 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 6 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 7 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 7 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 7 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 7 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 8 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 8 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 8 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 8 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 9 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 9 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 9 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 9 0 U____ Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug] [<c02737f2>] kprobe_exceptions_notify+0x151/0x368 [<c0274276>] notifier_call_chain+0x2a/0x52 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a [<c012e29a>] notify_die+0x30/0x34 [<c0272ca9>] do_int3+0x2f/0x6e [<c0272baf>] int3+0x27/0x2c [<c01acf12>] submit_bio+0x1/0xfb [<c017d77c>] submit_bh+0xb9/0xd4 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd] [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd] [<c01152a4>] update_curr+0x52/0xc8 [<c011534d>] set_next_entity+0x11/0x38 [<c02714a1>] schedule+0x2f9/0x33f [<f99e5cf0>] kjournald+0xbc/0x223 [jbd] [<c012a91b>] autoremove_wake_function+0x0/0x33 [<f99e5c34>] kjournald+0x0/0x223 [jbd] [<c012a861>] kthread+0x36/0x5d [<c012a82b>] kthread+0x0/0x5d [<c010493b>] kernel_thread_helper+0x7/0x10 ======================= Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug] [<c02737f2>] kprobe_exceptions_notify+0x151/0x368 [<c0274276>] notifier_call_chain+0x2a/0x52 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a [<c012e29a>] notify_die+0x30/0x34 [<c0272ca9>] do_int3+0x2f/0x6e [<c0272baf>] int3+0x27/0x2c [<c01acf12>] submit_bio+0x1/0xfb [<c017d77c>] submit_bh+0xb9/0xd4 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd] [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd] [<c01152a4>] update_curr+0x52/0xc8 [<c011534d>] set_next_entity+0x11/0x38 [<c02714a1>] schedule+0x2f9/0x33f [<f99e5cf0>] kjournald+0xbc/0x223 [jbd] [<c012a91b>] autoremove_wake_function+0x0/0x33 [<f99e5c34>] kjournald+0x0/0x223 [jbd] [<c012a861>] kthread+0x36/0x5d [<c012a82b>] kthread+0x0/0x5d [<c010493b>] kernel_thread_helper+0x7/0x10 ======================= Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug] [<c02737f2>] kprobe_exceptions_notify+0x151/0x368 [<c0274276>] notifier_call_chain+0x2a/0x52 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a [<c012e29a>] notify_die+0x30/0x34 [<c0272ca9>] do_int3+0x2f/0x6e [<c0272baf>] int3+0x27/0x2c [<c01acf12>] submit_bio+0x1/0xfb [<c017d77c>] submit_bh+0xb9/0xd4 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd] [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd] [<c01152a4>] update_curr+0x52/0xc8 [<c011534d>] set_next_entity+0x11/0x38 [<c02714a1>] schedule+0x2f9/0x33f [<f99e5cf0>] kjournald+0xbc/0x223 [jbd] [<c012a91b>] autoremove_wake_function+0x0/0x33 [<f99e5c34>] kjournald+0x0/0x223 [jbd] [<c012a861>] kthread+0x36/0x5d [<c012a82b>] kthread+0x0/0x5d [<c010493b>] kernel_thread_helper+0x7/0x10 ======================= Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug] [<c02737f2>] kprobe_exceptions_notify+0x151/0x368 [<c0274276>] notifier_call_chain+0x2a/0x52 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a [<c012e29a>] notify_die+0x30/0x34 [<c0272ca9>] do_int3+0x2f/0x6e [<c0272baf>] int3+0x27/0x2c [<c01acf12>] submit_bio+0x1/0xfb [<c017d77c>] submit_bh+0xb9/0xd4 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd] [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd] [<c01152a4>] update_curr+0x52/0xc8 [<c011534d>] set_next_entity+0x11/0x38 [<c02714a1>] schedule+0x2f9/0x33f [<f99e5cf0>] kjournald+0xbc/0x223 [jbd] [<c012a91b>] autoremove_wake_function+0x0/0x33 [<f99e5c34>] kjournald+0x0/0x223 [jbd] [<c012a861>] kthread+0x36/0x5d [<c012a82b>] kthread+0x0/0x5d [<c010493b>] kernel_thread_helper+0x7/0x10 ======================= requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 10 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 10 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 10 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 10 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 11 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 11 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 11 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 11 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 12 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 12 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 12 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 12 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 13 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 13 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 13 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 13 0 U____ Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug] [<c02737f2>] kprobe_exceptions_notify+0x151/0x368 [<c0274276>] notifier_call_chain+0x2a/0x52 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a [<c012e29a>] notify_die+0x30/0x34 [<c0272ca9>] do_int3+0x2f/0x6e [<c0272baf>] int3+0x27/0x2c [<c014007b>] devm_free_irq+0x4f/0x61 [<c01acf12>] submit_bio+0x1/0xfb [<c017d77c>] submit_bh+0xb9/0xd4 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd] [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd] [<c01152a4>] update_curr+0x52/0xc8 [<c011534d>] set_next_entity+0x11/0x38 [<c02714a1>] schedule+0x2f9/0x33f [<f99e5cf0>] kjournald+0xbc/0x223 [jbd] [<c012a91b>] autoremove_wake_function+0x0/0x33 [<f99e5c34>] kjournald+0x0/0x223 [jbd] [<c012a861>] kthread+0x36/0x5d [<c012a82b>] kthread+0x0/0x5d [<c010493b>] kernel_thread_helper+0x7/0x10 ======================= Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug] [<c02737f2>] kprobe_exceptions_notify+0x151/0x368 [<c0274276>] notifier_call_chain+0x2a/0x52 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a [<c012e29a>] notify_die+0x30/0x34 [<c0272ca9>] do_int3+0x2f/0x6e [<c0272baf>] int3+0x27/0x2c [<c014007b>] devm_free_irq+0x4f/0x61 [<c01acf12>] submit_bio+0x1/0xfb [<c017d77c>] submit_bh+0xb9/0xd4 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd] [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd] [<c01152a4>] update_curr+0x52/0xc8 [<c011534d>] set_next_entity+0x11/0x38 [<c02714a1>] schedule+0x2f9/0x33f [<f99e5cf0>] kjournald+0xbc/0x223 [jbd] [<c012a91b>] autoremove_wake_function+0x0/0x33 [<f99e5c34>] kjournald+0x0/0x223 [jbd] [<c012a861>] kthread+0x36/0x5d [<c012a82b>] kthread+0x0/0x5d [<c010493b>] kernel_thread_helper+0x7/0x10 ======================= Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug] [<c02737f2>] kprobe_exceptions_notify+0x151/0x368 [<c0274276>] notifier_call_chain+0x2a/0x52 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a [<c012e29a>] notify_die+0x30/0x34 [<c0272ca9>] do_int3+0x2f/0x6e [<c0272baf>] int3+0x27/0x2c [<c014007b>] devm_free_irq+0x4f/0x61 [<c01acf12>] submit_bio+0x1/0xfb [<c017d77c>] submit_bh+0xb9/0xd4 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd] [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd] [<c01152a4>] update_curr+0x52/0xc8 [<c011534d>] set_next_entity+0x11/0x38 [<c02714a1>] schedule+0x2f9/0x33f [<f99e5cf0>] kjournald+0xbc/0x223 [jbd] [<c012a91b>] autoremove_wake_function+0x0/0x33 [<f99e5c34>] kjournald+0x0/0x223 [jbd] [<c012a861>] kthread+0x36/0x5d [<c012a82b>] kthread+0x0/0x5d [<c010493b>] kernel_thread_helper+0x7/0x10 ======================= Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug] [<c02737f2>] kprobe_exceptions_notify+0x151/0x368 [<c0274276>] notifier_call_chain+0x2a/0x52 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a [<c012e29a>] notify_die+0x30/0x34 [<c0272ca9>] do_int3+0x2f/0x6e [<c0272baf>] int3+0x27/0x2c [<c014007b>] devm_free_irq+0x4f/0x61 [<c01acf12>] submit_bio+0x1/0xfb [<c017d77c>] submit_bh+0xb9/0xd4 [<f99e28d7>] journal_do_submit_data+0x23/0x2b [jbd] [<f99e2d59>] journal_commit_transaction+0x467/0xcca [jbd] [<c01152a4>] update_curr+0x52/0xc8 [<c011534d>] set_next_entity+0x11/0x38 [<c02714a1>] schedule+0x2f9/0x33f [<f99e5cf0>] kjournald+0xbc/0x223 [jbd] [<c012a91b>] autoremove_wake_function+0x0/0x33 [<f99e5c34>] kjournald+0x0/0x223 [jbd] [<c012a861>] kthread+0x36/0x5d [<c012a82b>] kthread+0x0/0x5d [<c010493b>] kernel_thread_helper+0x7/0x10 ======================= requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 14 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 14 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 14 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 14 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 15 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 15 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 15 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 15 0 U____ Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug] [<c02737f2>] kprobe_exceptions_notify+0x151/0x368 [<c0274276>] notifier_call_chain+0x2a/0x52 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a [<c012e29a>] notify_die+0x30/0x34 [<c0272ca9>] do_int3+0x2f/0x6e [<f99e1140>] __journal_file_buffer+0x8d/0x10d [jbd] [<c0272baf>] int3+0x27/0x2c [<c01acf12>] submit_bio+0x1/0xfb [<c017d77c>] submit_bh+0xb9/0xd4 [<f99e3083>] journal_commit_transaction+0x791/0xcca [jbd] [<c01152a4>] update_curr+0x52/0xc8 [<c011534d>] set_next_entity+0x11/0x38 [<c02714a1>] schedule+0x2f9/0x33f [<f99e5cf0>] kjournald+0xbc/0x223 [jbd] [<c012a91b>] autoremove_wake_function+0x0/0x33 [<f99e5c34>] kjournald+0x0/0x223 [jbd] [<c012a861>] kthread+0x36/0x5d [<c012a82b>] kthread+0x0/0x5d [<c010493b>] kernel_thread_helper+0x7/0x10 ======================= Pid: 2244, comm: kjournald Not tainted 2.6.24-rc7-ext2 #1 [<fa00716b>] handler_pre+0x18/0x1b [ext2_writeback_debug] [<c02737f2>] kprobe_exceptions_notify+0x151/0x368 [<c0274276>] notifier_call_chain+0x2a/0x52 [<c02742c1>] __atomic_notifier_call_chain+0x23/0x41 [<c02742f6>] atomic_notifier_call_chain+0x17/0x1a [<c012e29a>] notify_die+0x30/0x34 [<c0272ca9>] do_int3+0x2f/0x6e [<c0272baf>] int3+0x27/0x2c [<c01acf12>] submit_bio+0x1/0xfb [<c017d77c>] submit_bh+0xb9/0xd4 [<f99e3083>] journal_commit_transaction+0x791/0xcca [jbd] [<c01152a4>] update_curr+0x52/0xc8 [<c011534d>] set_next_entity+0x11/0x38 [<c02714a1>] schedule+0x2f9/0x33f [<f99e5cf0>] kjournald+0xbc/0x223 [jbd] [<c012a91b>] autoremove_wake_function+0x0/0x33 [<f99e5c34>] kjournald+0x0/0x223 [jbd] [<c012a861>] kthread+0x36/0x5d [<c012a82b>] kthread+0x0/0x5d [<c010493b>] kernel_thread_helper+0x7/0x10 ======================= requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 16 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 16 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 16 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 16 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 17 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 17 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 17 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 17 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 18 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 18 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 18 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 18 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 19 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 19 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 19 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 19 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 20 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 20 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 20 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 20 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 21 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 21 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 21 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 21 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 22 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 22 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 22 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 22 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 23 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 23 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 23 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 23 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 24 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 24 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 24 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 24 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 25 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 25 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 25 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 25 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 26 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 26 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 26 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 26 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 27 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 27 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 27 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 27 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 28 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 28 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 28 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 28 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 29 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 29 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 29 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 29 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 30 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 30 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 30 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 30 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 31 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 31 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 31 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 31 0 U____ register_jprobe(ext2_writepage) = 0 register_jprobe(requeue_io) = 0 register_kprobe(submit_bio) = 0 requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 32 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 32 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 32 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 32 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 33 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 33 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 33 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 33 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 34 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 34 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 34 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 34 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 35 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 35 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 35 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 35 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 36 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 36 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 36 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 36 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 37 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 37 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 37 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 37 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 38 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 38 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 38 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 38 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 39 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 39 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 39 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 39 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 40 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 40 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 40 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 40 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 41 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 41 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 41 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 41 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 42 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 42 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 42 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 42 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 43 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 43 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 43 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 43 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 44 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 44 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 44 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 44 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 45 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 45 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 45 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 45 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 46 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 46 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 46 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 46 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 47 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 47 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 47 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 47 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 48 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 48 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 48 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 48 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 49 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 49 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 49 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 49 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 50 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 50 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 50 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 50 0 U____ requeue_io: inode 114019(sda7/.kde) count 2,2 size 0 pages 1 0 51 0 U____ requeue_io: inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 0 51 0 U____ requeue_io: inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 0 51 0 U____ requeue_io: inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 0 51 0 U____ ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JE1Uz-0002w5-6z@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JE1Uz-0002w5-6z@localhost.localdomain> @ 2008-01-13 11:59 ` Fengguang Wu 0 siblings, 0 replies; 59+ messages in thread From: Fengguang Wu @ 2008-01-13 11:59 UTC (permalink / raw) To: Joerg Platte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel, linux-ext4 On Sun, Jan 13, 2008 at 10:49:31AM +0100, Joerg Platte wrote: > register_jprobe(ext2_writepage) = 0 > register_jprobe(requeue_io) = 0 > register_kprobe(submit_bio) = 0 > requeue_io: > inode 114019(sda7/.kde) count 2,2 size 0 pages 1 > 0 2 0 U____ > requeue_io: > inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 > 0 2 0 U____ > requeue_io: > inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 > 0 2 0 U____ > requeue_io: > inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 > 0 2 0 U____ It helps. Thank you, Joerg! The .kde/cache-ibm/socket-ibm/0266584877 above are directories. It's weird that dirs would have their own mappings in ext2. In particular this bug is triggered because the dir mapping page has PAGECACHE_TAG_DIRTY set and PG_dirty cleared, staying in an inconsistent state. Fengguang ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <20080113115933.GA11045@mail.ustc.edu.cn>]
[parent not found: <E1JEGPH-0001uw-Df@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JEGPH-0001uw-Df@localhost.localdomain> @ 2008-01-14 3:54 ` Fengguang Wu 0 siblings, 0 replies; 59+ messages in thread From: Fengguang Wu @ 2008-01-14 3:54 UTC (permalink / raw) To: Joerg Platte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel, linux-ext4 On Sun, Jan 13, 2008 at 07:59:33PM +0800, Fengguang Wu wrote: > On Sun, Jan 13, 2008 at 10:49:31AM +0100, Joerg Platte wrote: > > register_jprobe(ext2_writepage) = 0 > > register_jprobe(requeue_io) = 0 > > register_kprobe(submit_bio) = 0 > > requeue_io: > > inode 114019(sda7/.kde) count 2,2 size 0 pages 1 > > 0 2 0 U____ > > requeue_io: > > inode 114025(sda7/cache-ibm) count 2,1 size 0 pages 1 > > 0 2 0 U____ > > requeue_io: > > inode 114029(sda7/socket-ibm) count 2,3 size 0 pages 1 > > 0 2 0 U____ > > requeue_io: > > inode 114017(sda7/0266584877) count 3,6 size 0 pages 1 > > 0 2 0 U____ > > It helps. Thank you, Joerg! > > The .kde/cache-ibm/socket-ibm/0266584877 above are directories. > It's weird that dirs would have their own mappings in ext2. In Oh, ext2 dirs have their own mapping pages. Not the same with ext3. > particular this bug is triggered because the dir mapping page has > PAGECACHE_TAG_DIRTY set and PG_dirty cleared, staying in an > inconsistent state. Just found that a deleted dir will enter that inconsistent state when someone still have reference to it... ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <20080114035439.GA7330@mail.ustc.edu.cn>]
[parent not found: <E1JEM2I-00010S-5U@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JEM2I-00010S-5U@localhost.localdomain> @ 2008-01-14 9:55 ` Fengguang Wu 2008-01-14 11:30 ` Joerg Platte 0 siblings, 1 reply; 59+ messages in thread From: Fengguang Wu @ 2008-01-14 9:55 UTC (permalink / raw) To: Joerg Platte; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel, linux-ext4 On Mon, Jan 14, 2008 at 11:54:39AM +0800, Fengguang Wu wrote: > > particular this bug is triggered because the dir mapping page has > > PAGECACHE_TAG_DIRTY set and PG_dirty cleared, staying in an > > inconsistent state. > > Just found that a deleted dir will enter that inconsistent state when > someone still have reference to it... Joerg, this patch fixed the bug for me :-) Fengguang --- clear PAGECACHE_TAG_DIRTY for truncated page in block_write_full_page() The `truncated' page in block_write_full_page() may stick for a long time. E.g. ext2_rmdir() will set i_size to 0. The dir may still be referenced by someone, and have dirty pages in it. So clear PAGECACHE_TAG_DIRTY to prevent pdflush from retrying and iowaiting on it. Signed-off-by: Fengguang Wu <wfg@mail.ustc.edu.cn> --- fs/buffer.c | 2 ++ 1 files changed, 2 insertions(+) Index: linux/fs/buffer.c =================================================================== --- linux.orig/fs/buffer.c +++ linux/fs/buffer.c @@ -2820,7 +2820,9 @@ int block_write_full_page(struct page *p * freeable here, so the page does not leak. */ do_invalidatepage(page, 0); + set_page_writeback(page); unlock_page(page); + end_page_writeback(page); return 0; /* don't care */ } ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-14 9:55 ` Fengguang Wu @ 2008-01-14 11:30 ` Joerg Platte 2008-01-14 11:41 ` Peter Zijlstra 0 siblings, 1 reply; 59+ messages in thread From: Joerg Platte @ 2008-01-14 11:30 UTC (permalink / raw) To: Fengguang Wu; +Cc: Peter Zijlstra, Ingo Molnar, linux-kernel, linux-ext4 Am Montag, 14. Januar 2008 schrieb Fengguang Wu: > Joerg, this patch fixed the bug for me :-) Fengguang, congratulations, I can confirm that your patch fixed the bug! With previous kernels the bug showed up after each reboot. Now, when booting the patched kernel everything is fine and there is no longer any suspicious iowait! Do you have an idea why this problem appeared in 2.6.24? Did somebody change the ext2 code or is it related to the changes in the scheduler? regards, Jörg -- PGP Key: send mail with subject 'SEND PGP-KEY' PGP Key-ID: FD 4E 21 1D PGP Fingerprint: 388A872AFC5649D3 BCEC65778BE0C605 ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-14 11:30 ` Joerg Platte @ 2008-01-14 11:41 ` Peter Zijlstra [not found] ` <E1JEOmD-0001Ap-U7@localhost.localdomain> 0 siblings, 1 reply; 59+ messages in thread From: Peter Zijlstra @ 2008-01-14 11:41 UTC (permalink / raw) To: jplatte; +Cc: Fengguang Wu, Ingo Molnar, linux-kernel, linux-ext4 On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote: > Am Montag, 14. Januar 2008 schrieb Fengguang Wu: > > > Joerg, this patch fixed the bug for me :-) > > Fengguang, congratulations, I can confirm that your patch fixed the bug! With > previous kernels the bug showed up after each reboot. Now, when booting the > patched kernel everything is fine and there is no longer any suspicious > iowait! > > Do you have an idea why this problem appeared in 2.6.24? Did somebody change > the ext2 code or is it related to the changes in the scheduler? It was Fengguang who changed the inode writeback code, and I guess the new and improved code was less able do deal with these funny corner cases. But he has been very good in tracking them down and solving them, kudos to him for that work! ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JEOmD-0001Ap-U7@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JEOmD-0001Ap-U7@localhost.localdomain> @ 2008-01-14 12:50 ` Fengguang Wu 2008-01-15 21:13 ` Mike Snitzer 2008-01-15 21:42 ` Ingo Molnar 1 sibling, 1 reply; 59+ messages in thread From: Fengguang Wu @ 2008-01-14 12:50 UTC (permalink / raw) To: Peter Zijlstra Cc: jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds, Andrew Morton On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote: > > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote: > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu: > > > > > Joerg, this patch fixed the bug for me :-) > > > > Fengguang, congratulations, I can confirm that your patch fixed the bug! With > > previous kernels the bug showed up after each reboot. Now, when booting the > > patched kernel everything is fine and there is no longer any suspicious > > iowait! > > > > Do you have an idea why this problem appeared in 2.6.24? Did somebody change > > the ext2 code or is it related to the changes in the scheduler? > > It was Fengguang who changed the inode writeback code, and I guess the > new and improved code was less able do deal with these funny corner > cases. But he has been very good in tracking them down and solving them, > kudos to him for that work! Thank you. In particular the bug is triggered by the patch named: "writeback: introduce writeback_control.more_io to indicate more io" That patch means to speed up writeback, but unfortunately its aggressiveness has disclosed bugs in reiserfs, jfs and now ext2. Linus, given the number of bugs it triggered, I'd recommend revert this patch(git commit 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). Let's push it back to -mm tree for more testings? Regards, Fengguang ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-14 12:50 ` Fengguang Wu @ 2008-01-15 21:13 ` Mike Snitzer [not found] ` <E1JF0m1-000101-OK@localhost.localdomain> 0 siblings, 1 reply; 59+ messages in thread From: Mike Snitzer @ 2008-01-15 21:13 UTC (permalink / raw) To: Fengguang Wu Cc: Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds, Andrew Morton On Jan 14, 2008 7:50 AM, Fengguang Wu <wfg@mail.ustc.edu.cn> wrote: > On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote: > > > > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote: > > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu: > > > > > > > Joerg, this patch fixed the bug for me :-) > > > > > > Fengguang, congratulations, I can confirm that your patch fixed the bug! With > > > previous kernels the bug showed up after each reboot. Now, when booting the > > > patched kernel everything is fine and there is no longer any suspicious > > > iowait! > > > > > > Do you have an idea why this problem appeared in 2.6.24? Did somebody change > > > the ext2 code or is it related to the changes in the scheduler? > > > > It was Fengguang who changed the inode writeback code, and I guess the > > new and improved code was less able do deal with these funny corner > > cases. But he has been very good in tracking them down and solving them, > > kudos to him for that work! > > Thank you. > > In particular the bug is triggered by the patch named: > "writeback: introduce writeback_control.more_io to indicate more io" > That patch means to speed up writeback, but unfortunately its > aggressiveness has disclosed bugs in reiserfs, jfs and now ext2. > > Linus, given the number of bugs it triggered, I'd recommend revert > this patch(git commit 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). Let's > push it back to -mm tree for more testings? Fengguang, I'd like to better understand where your writeback work stands relative to 2.6.24-rcX and -mm. To be clear, your changes in 2.6.24-rc7 have been benchmarked to provide a ~33% sequential write performance improvement with ext3 (as compared to 2.6.22, CFS could be helping, etc but...). Very impressive! Given this improvement it is unfortunate to see your request to revert 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b but it is understandable if you're not confident in it for 2.6.24. That said, you recently posted an -mm patchset that first reverts 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b and then goes on to address the "slow writes for concurrent large and small file writes" bug: http://lkml.org/lkml/2008/1/15/132 For those interested in using your writeback improvements in production sooner rather than later (primarily with ext3); what recommendations do you have? Just heavily test our own 2.6.24 + your evolving "close, but not ready for merge" -mm writeback patchset? regards, Mike ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JF0m1-000101-OK@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JF0m1-000101-OK@localhost.localdomain> @ 2008-01-16 5:25 ` Fengguang Wu 0 siblings, 0 replies; 59+ messages in thread From: Fengguang Wu @ 2008-01-16 5:25 UTC (permalink / raw) To: Mike Snitzer Cc: Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds, Andrew Morton On Tue, Jan 15, 2008 at 04:13:22PM -0500, Mike Snitzer wrote: > On Jan 14, 2008 7:50 AM, Fengguang Wu <wfg@mail.ustc.edu.cn> wrote: > > On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote: > > > > > > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote: > > > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu: > > > > > > > > > Joerg, this patch fixed the bug for me :-) > > > > > > > > Fengguang, congratulations, I can confirm that your patch fixed the bug! With > > > > previous kernels the bug showed up after each reboot. Now, when booting the > > > > patched kernel everything is fine and there is no longer any suspicious > > > > iowait! > > > > > > > > Do you have an idea why this problem appeared in 2.6.24? Did somebody change > > > > the ext2 code or is it related to the changes in the scheduler? > > > > > > It was Fengguang who changed the inode writeback code, and I guess the > > > new and improved code was less able do deal with these funny corner > > > cases. But he has been very good in tracking them down and solving them, > > > kudos to him for that work! > > > > Thank you. > > > > In particular the bug is triggered by the patch named: > > "writeback: introduce writeback_control.more_io to indicate more io" > > That patch means to speed up writeback, but unfortunately its > > aggressiveness has disclosed bugs in reiserfs, jfs and now ext2. > > > > Linus, given the number of bugs it triggered, I'd recommend revert > > this patch(git commit 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). Let's > > push it back to -mm tree for more testings? > > Fengguang, > > I'd like to better understand where your writeback work stands > relative to 2.6.24-rcX and -mm. To be clear, your changes in > 2.6.24-rc7 have been benchmarked to provide a ~33% sequential write > performance improvement with ext3 (as compared to 2.6.22, CFS could be > helping, etc but...). Very impressive! Wow, glad to hear that. > Given this improvement it is unfortunate to see your request to revert > 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b but it is understandable if > you're not confident in it for 2.6.24. > > That said, you recently posted an -mm patchset that first reverts > 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b and then goes on to address > the "slow writes for concurrent large and small file writes" bug: > http://lkml.org/lkml/2008/1/15/132 > > For those interested in using your writeback improvements in > production sooner rather than later (primarily with ext3); what > recommendations do you have? Just heavily test our own 2.6.24 + your > evolving "close, but not ready for merge" -mm writeback patchset? It's not ready mainly because it is fresh made and need more feedbacks. It's doing OK on my desktop :-) ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JEOmD-0001Ap-U7@localhost.localdomain> 2008-01-14 12:50 ` Fengguang Wu @ 2008-01-15 21:42 ` Ingo Molnar [not found] ` <E1JF0bJ-0000zU-FG@localhost.localdomain> 1 sibling, 1 reply; 59+ messages in thread From: Ingo Molnar @ 2008-01-15 21:42 UTC (permalink / raw) To: Fengguang Wu Cc: Peter Zijlstra, jplatte, linux-kernel, linux-ext4, Linus Torvalds, Andrew Morton * Fengguang Wu <wfg@mail.ustc.edu.cn> wrote: > On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote: > > > > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote: > > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu: > > > > > > > Joerg, this patch fixed the bug for me :-) > > > > > > Fengguang, congratulations, I can confirm that your patch fixed the bug! With > > > previous kernels the bug showed up after each reboot. Now, when booting the > > > patched kernel everything is fine and there is no longer any suspicious > > > iowait! > > > > > > Do you have an idea why this problem appeared in 2.6.24? Did somebody change > > > the ext2 code or is it related to the changes in the scheduler? > > > > It was Fengguang who changed the inode writeback code, and I guess the > > new and improved code was less able do deal with these funny corner > > cases. But he has been very good in tracking them down and solving them, > > kudos to him for that work! > > Thank you. > > In particular the bug is triggered by the patch named: > "writeback: introduce writeback_control.more_io to indicate more io" > That patch means to speed up writeback, but unfortunately its > aggressiveness has disclosed bugs in reiserfs, jfs and now ext2. > > Linus, given the number of bugs it triggered, I'd recommend revert > this patch(git commit 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). Let's > push it back to -mm tree for more testings? i dont think a revert at this stage is a good idea and i'm not sure pushing it back into -mm would really expose more of these bugs. And these are real bugs in filesystems - bugs which we want to see fixed anyway. You are also tracking down those bugs very fast. [ perhaps, if it's possible technically (and if it is clean enough), you might want to offer a runtime debug tunable that can be used to switch off the new aspects of your code. That would speed up testing, in case anyone suspects the new writeback code. ] Ingo ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JF0bJ-0000zU-FG@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JF0bJ-0000zU-FG@localhost.localdomain> @ 2008-01-16 5:14 ` Fengguang Wu 0 siblings, 0 replies; 59+ messages in thread From: Fengguang Wu @ 2008-01-16 5:14 UTC (permalink / raw) To: Ingo Molnar Cc: Peter Zijlstra, jplatte, linux-kernel, linux-ext4, Linus Torvalds, Andrew Morton, Mike Snitzer On Tue, Jan 15, 2008 at 10:42:13PM +0100, Ingo Molnar wrote: > > * Fengguang Wu <wfg@mail.ustc.edu.cn> wrote: > > > On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote: > > > > > > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote: > > > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu: > > > > > > > > > Joerg, this patch fixed the bug for me :-) > > > > > > > > Fengguang, congratulations, I can confirm that your patch fixed the bug! With > > > > previous kernels the bug showed up after each reboot. Now, when booting the > > > > patched kernel everything is fine and there is no longer any suspicious > > > > iowait! > > > > > > > > Do you have an idea why this problem appeared in 2.6.24? Did somebody change > > > > the ext2 code or is it related to the changes in the scheduler? > > > > > > It was Fengguang who changed the inode writeback code, and I guess the > > > new and improved code was less able do deal with these funny corner > > > cases. But he has been very good in tracking them down and solving them, > > > kudos to him for that work! > > > > Thank you. > > > > In particular the bug is triggered by the patch named: > > "writeback: introduce writeback_control.more_io to indicate more io" > > That patch means to speed up writeback, but unfortunately its > > aggressiveness has disclosed bugs in reiserfs, jfs and now ext2. > > > > Linus, given the number of bugs it triggered, I'd recommend revert > > this patch(git commit 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). Let's > > push it back to -mm tree for more testings? > > i dont think a revert at this stage is a good idea and i'm not sure > pushing it back into -mm would really expose more of these bugs. And > these are real bugs in filesystems - bugs which we want to see fixed > anyway. You are also tracking down those bugs very fast. > > [ perhaps, if it's possible technically (and if it is clean enough), you > might want to offer a runtime debug tunable that can be used to switch > off the new aspects of your code. That would speed up testing, in case > anyone suspects the new writeback code. ] The patch is too aggressive in itself. We'd better not risk on it. The iowait is only unpleasant not destructive. But it will hurt if many users complaints. Comment says that "nfs_writepages() sometimes bales out without doing anything." However I have an improved and more safe patch now. It won't iowait when nfs_writepages() bale out without increasing pages_skipped, or even when some buggy filesystem forget to clear PAGECACHE_TAG_DIRTY. (The magic lies in the first chunk below.) Mike, you can use this one on 2.6.24. --- fs/fs-writeback.c | 17 +++++++++++++++-- include/linux/writeback.h | 1 + mm/page-writeback.c | 9 ++++++--- 3 files changed, 22 insertions(+), 5 deletions(-) --- linux.orig/fs/fs-writeback.c +++ linux/fs/fs-writeback.c @@ -284,7 +284,16 @@ __sync_single_inode(struct inode *inode, * soon as the queue becomes uncongested. */ inode->i_state |= I_DIRTY_PAGES; - requeue_io(inode); + if (wbc->nr_to_write <= 0) + /* + * slice used up: queue for next turn + */ + requeue_io(inode); + else + /* + * somehow blocked: retry later + */ + redirty_tail(inode); } else { /* * Otherwise fully redirty the inode so that @@ -479,8 +488,12 @@ sync_sb_inodes(struct super_block *sb, s iput(inode); cond_resched(); spin_lock(&inode_lock); - if (wbc->nr_to_write <= 0) + if (wbc->nr_to_write <= 0) { + wbc->more_io = 1; break; + } + if (!list_empty(&sb->s_more_io)) + wbc->more_io = 1; } return; /* Leave any unwritten inodes on s_io */ } --- linux.orig/include/linux/writeback.h +++ linux/include/linux/writeback.h @@ -62,6 +62,7 @@ struct writeback_control { unsigned for_reclaim:1; /* Invoked from the page allocator */ unsigned for_writepages:1; /* This is a writepages() call */ unsigned range_cyclic:1; /* range_start is cyclic */ + unsigned more_io:1; /* more io to be dispatched */ }; /* --- linux.orig/mm/page-writeback.c +++ linux/mm/page-writeback.c @@ -558,6 +558,7 @@ static void background_writeout(unsigned global_page_state(NR_UNSTABLE_NFS) < background_thresh && min_pages <= 0) break; + wbc.more_io = 0; wbc.encountered_congestion = 0; wbc.nr_to_write = MAX_WRITEBACK_PAGES; wbc.pages_skipped = 0; @@ -565,8 +566,9 @@ static void background_writeout(unsigned min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write; if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) { /* Wrote less than expected */ - congestion_wait(WRITE, HZ/10); - if (!wbc.encountered_congestion) + if (wbc.encountered_congestion || wbc.more_io) + congestion_wait(WRITE, HZ/10); + else break; } } @@ -631,11 +633,12 @@ static void wb_kupdate(unsigned long arg global_page_state(NR_UNSTABLE_NFS) + (inodes_stat.nr_inodes - inodes_stat.nr_unused); while (nr_to_write > 0) { + wbc.more_io = 0; wbc.encountered_congestion = 0; wbc.nr_to_write = MAX_WRITEBACK_PAGES; writeback_inodes(&wbc); if (wbc.nr_to_write > 0) { - if (wbc.encountered_congestion) + if (wbc.encountered_congestion || wbc.more_io) congestion_wait(WRITE, HZ/10); else break; /* All the old data is written */ ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX @ 2008-01-16 9:26 Martin Knoblauch [not found] ` <E1JF6w8-0000vs-HM@localhost.localdomain> 0 siblings, 1 reply; 59+ messages in thread From: Martin Knoblauch @ 2008-01-16 9:26 UTC (permalink / raw) To: Mike Snitzer, Fengguang Wu Cc: Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds ----- Original Message ---- > From: Mike Snitzer <snitzer@gmail.com> > To: Fengguang Wu <wfg@mail.ustc.edu.cn> > Cc: Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>; Andrew Morton <akpm@li> > Sent: Tuesday, January 15, 2008 10:13:22 PM > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > On Jan 14, 2008 7:50 AM, Fengguang Wu wrote: > > On Mon, Jan 14, 2008 at 12:41:26PM +0100, Peter Zijlstra wrote: > > > > > > On Mon, 2008-01-14 at 12:30 +0100, Joerg Platte wrote: > > > > Am Montag, 14. Januar 2008 schrieb Fengguang Wu: > > > > > > > > > Joerg, this patch fixed the bug for me :-) > > > > > > > > Fengguang, congratulations, I can confirm that your patch > fixed > the bug! With > > > > previous kernels the bug showed up after each reboot. Now, > when > booting the > > > > patched kernel everything is fine and there is no longer > any > suspicious > > > > iowait! > > > > > > > > Do you have an idea why this problem appeared in 2.6.24? > Did > somebody change > > > > the ext2 code or is it related to the changes in the scheduler? > > > > > > It was Fengguang who changed the inode writeback code, and I > guess > the > > > new and improved code was less able do deal with these funny corner > > > cases. But he has been very good in tracking them down and > solving > them, > > > kudos to him for that work! > > > > Thank you. > > > > In particular the bug is triggered by the patch named: > > "writeback: introduce writeback_control.more_io to > indicate > more io" > > That patch means to speed up writeback, but unfortunately its > > aggressiveness has disclosed bugs in reiserfs, jfs and now ext2. > > > > Linus, given the number of bugs it triggered, I'd recommend revert > > this patch(git commit > 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b). > Let's > > push it back to -mm tree for more testings? > > Fengguang, > > I'd like to better understand where your writeback work stands > relative to 2.6.24-rcX and -mm. To be clear, your changes in > 2.6.24-rc7 have been benchmarked to provide a ~33% sequential write > performance improvement with ext3 (as compared to 2.6.22, CFS could be > helping, etc but...). Very impressive! > > Given this improvement it is unfortunate to see your request to revert > 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b but it is understandable if > you're not confident in it for 2.6.24. > > That said, you recently posted an -mm patchset that first reverts > 2e6883bdf49abd0e7f0d9b6297fc3be7ebb2250b and then goes on to address > the "slow writes for concurrent large and small file writes" bug: > http://lkml.org/lkml/2008/1/15/132 > > For those interested in using your writeback improvements in > production sooner rather than later (primarily with ext3); what > recommendations do you have? Just heavily test our own 2.6.24 + your > evolving "close, but not ready for merge" -mm writeback patchset? > Hi Fengguang, Mike, I can add myself to Mikes question. It would be good to know a "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has been showing quite nice improvement of the overall writeback situation and it would be sad to see this [partially] gone in 2.6.24-final. Linus apparently already has reverted "...2250b". I will definitely repeat my tests with -rc8. and report. Cheers Martin ^ permalink raw reply [flat|nested] 59+ messages in thread
[parent not found: <E1JF6w8-0000vs-HM@localhost.localdomain>]
* Re: regression: 100% io-wait with 2.6.24-rcX [not found] ` <E1JF6w8-0000vs-HM@localhost.localdomain> @ 2008-01-16 12:00 ` Fengguang Wu 0 siblings, 0 replies; 59+ messages in thread From: Fengguang Wu @ 2008-01-16 12:00 UTC (permalink / raw) To: Martin Knoblauch Cc: Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote: > > For those interested in using your writeback improvements in > > production sooner rather than later (primarily with ext3); what > > recommendations do you have? Just heavily test our own 2.6.24 + your > > evolving "close, but not ready for merge" -mm writeback patchset? > > > Hi Fengguang, Mike, > > I can add myself to Mikes question. It would be good to know a "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has been showing quite nice improvement of the overall writeback situation and it would be sad to see this [partially] gone in 2.6.24-final. Linus apparently already has reverted "...2250b". I will definitely repeat my tests with -rc8. and report. Thank you, Martin. Can you help test this patch on 2.6.24-rc7? Maybe we can push it to 2.6.24 after your testing. Fengguang --- fs/fs-writeback.c | 17 +++++++++++++++-- include/linux/writeback.h | 1 + mm/page-writeback.c | 9 ++++++--- 3 files changed, 22 insertions(+), 5 deletions(-) --- linux.orig/fs/fs-writeback.c +++ linux/fs/fs-writeback.c @@ -284,7 +284,16 @@ __sync_single_inode(struct inode *inode, * soon as the queue becomes uncongested. */ inode->i_state |= I_DIRTY_PAGES; - requeue_io(inode); + if (wbc->nr_to_write <= 0) + /* + * slice used up: queue for next turn + */ + requeue_io(inode); + else + /* + * somehow blocked: retry later + */ + redirty_tail(inode); } else { /* * Otherwise fully redirty the inode so that @@ -479,8 +488,12 @@ sync_sb_inodes(struct super_block *sb, s iput(inode); cond_resched(); spin_lock(&inode_lock); - if (wbc->nr_to_write <= 0) + if (wbc->nr_to_write <= 0) { + wbc->more_io = 1; break; + } + if (!list_empty(&sb->s_more_io)) + wbc->more_io = 1; } return; /* Leave any unwritten inodes on s_io */ } --- linux.orig/include/linux/writeback.h +++ linux/include/linux/writeback.h @@ -62,6 +62,7 @@ struct writeback_control { unsigned for_reclaim:1; /* Invoked from the page allocator */ unsigned for_writepages:1; /* This is a writepages() call */ unsigned range_cyclic:1; /* range_start is cyclic */ + unsigned more_io:1; /* more io to be dispatched */ }; /* --- linux.orig/mm/page-writeback.c +++ linux/mm/page-writeback.c @@ -558,6 +558,7 @@ static void background_writeout(unsigned global_page_state(NR_UNSTABLE_NFS) < background_thresh && min_pages <= 0) break; + wbc.more_io = 0; wbc.encountered_congestion = 0; wbc.nr_to_write = MAX_WRITEBACK_PAGES; wbc.pages_skipped = 0; @@ -565,8 +566,9 @@ static void background_writeout(unsigned min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write; if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) { /* Wrote less than expected */ - congestion_wait(WRITE, HZ/10); - if (!wbc.encountered_congestion) + if (wbc.encountered_congestion || wbc.more_io) + congestion_wait(WRITE, HZ/10); + else break; } } @@ -631,11 +633,12 @@ static void wb_kupdate(unsigned long arg global_page_state(NR_UNSTABLE_NFS) + (inodes_stat.nr_inodes - inodes_stat.nr_unused); while (nr_to_write > 0) { + wbc.more_io = 0; wbc.encountered_congestion = 0; wbc.nr_to_write = MAX_WRITEBACK_PAGES; writeback_inodes(&wbc); if (wbc.nr_to_write > 0) { - if (wbc.encountered_congestion) + if (wbc.encountered_congestion || wbc.more_io) congestion_wait(WRITE, HZ/10); else break; /* All the old data is written */ ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX @ 2008-01-16 14:15 Martin Knoblauch 2008-01-16 16:27 ` Mike Snitzer 0 siblings, 1 reply; 59+ messages in thread From: Martin Knoblauch @ 2008-01-16 14:15 UTC (permalink / raw) To: Fengguang Wu Cc: Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds ----- Original Message ---- > From: Fengguang Wu <wfg@mail.ustc.edu.cn> > To: Martin Knoblauch <knobi@knobisoft.de> > Cc: Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org> > Sent: Wednesday, January 16, 2008 1:00:04 PM > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote: > > > For those interested in using your writeback improvements in > > > production sooner rather than later (primarily with ext3); what > > > recommendations do you have? Just heavily test our own 2.6.24 > + > your > > > evolving "close, but not ready for merge" -mm writeback patchset? > > > > > Hi Fengguang, Mike, > > > > I can add myself to Mikes question. It would be good to know > a > "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has > been > showing quite nice improvement of the overall writeback situation and > it > would be sad to see this [partially] gone in 2.6.24-final. > Linus > apparently already has reverted "...2250b". I will definitely repeat my > tests > with -rc8. and report. > > Thank you, Martin. Can you help test this patch on 2.6.24-rc7? > Maybe we can push it to 2.6.24 after your testing. > Will do tomorrow or friday. Actually a patch against -rc8 would be nicer for me, as I have not looked at -rc7 due to holidays and some of the reported problems with it. Cheers Martin > Fengguang > --- > fs/fs-writeback.c | 17 +++++++++++++++-- > include/linux/writeback.h | 1 + > mm/page-writeback.c | 9 ++++++--- > 3 files changed, 22 insertions(+), 5 deletions(-) > > --- linux.orig/fs/fs-writeback.c > +++ linux/fs/fs-writeback.c > @@ -284,7 +284,16 @@ __sync_single_inode(struct inode *inode, > * soon as the queue becomes uncongested. > */ > inode->i_state |= I_DIRTY_PAGES; > - requeue_io(inode); > + if (wbc->nr_to_write <= 0) > + /* > + * slice used up: queue for next turn > + */ > + requeue_io(inode); > + else > + /* > + * somehow blocked: retry later > + */ > + redirty_tail(inode); > } else { > /* > * Otherwise fully redirty the inode so that > @@ -479,8 +488,12 @@ sync_sb_inodes(struct super_block *sb, s > iput(inode); > cond_resched(); > spin_lock(&inode_lock); > - if (wbc->nr_to_write <= 0) > + if (wbc->nr_to_write <= 0) { > + wbc->more_io = 1; > break; > + } > + if (!list_empty(&sb->s_more_io)) > + wbc->more_io = 1; > } > return; /* Leave any unwritten inodes on s_io */ > } > --- linux.orig/include/linux/writeback.h > +++ linux/include/linux/writeback.h > @@ -62,6 +62,7 @@ struct writeback_control { > unsigned for_reclaim:1; /* Invoked from the page > allocator > */ > unsigned for_writepages:1; /* This is a writepages() call */ > unsigned range_cyclic:1; /* range_start is cyclic */ > + unsigned more_io:1; /* more io to be dispatched */ > }; > > /* > --- linux.orig/mm/page-writeback.c > +++ linux/mm/page-writeback.c > @@ -558,6 +558,7 @@ static void background_writeout(unsigned > global_page_state(NR_UNSTABLE_NFS) < background_thresh > && min_pages <= 0) > break; > + wbc.more_io = 0; > wbc.encountered_congestion = 0; > wbc.nr_to_write = MAX_WRITEBACK_PAGES; > wbc.pages_skipped = 0; > @@ -565,8 +566,9 @@ static void background_writeout(unsigned > min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write; > if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) { > /* Wrote less than expected */ > - congestion_wait(WRITE, HZ/10); > - if (!wbc.encountered_congestion) > + if (wbc.encountered_congestion || wbc.more_io) > + congestion_wait(WRITE, HZ/10); > + else > break; > } > } > @@ -631,11 +633,12 @@ static void wb_kupdate(unsigned long arg > global_page_state(NR_UNSTABLE_NFS) + > (inodes_stat.nr_inodes - inodes_stat.nr_unused); > while (nr_to_write > 0) { > + wbc.more_io = 0; > wbc.encountered_congestion = 0; > wbc.nr_to_write = MAX_WRITEBACK_PAGES; > writeback_inodes(&wbc); > if (wbc.nr_to_write > 0) { > - if (wbc.encountered_congestion) > + if (wbc.encountered_congestion || wbc.more_io) > congestion_wait(WRITE, HZ/10); > else > break; /* All the old data is written */ > > > ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-16 14:15 Martin Knoblauch @ 2008-01-16 16:27 ` Mike Snitzer 0 siblings, 0 replies; 59+ messages in thread From: Mike Snitzer @ 2008-01-16 16:27 UTC (permalink / raw) To: Martin Knoblauch Cc: Fengguang Wu, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds On Jan 16, 2008 9:15 AM, Martin Knoblauch <spamtrap@knobisoft.de> wrote: > ----- Original Message ---- > > From: Fengguang Wu <wfg@mail.ustc.edu.cn> > > To: Martin Knoblauch <knobi@knobisoft.de> > > Cc: Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org> > > Sent: Wednesday, January 16, 2008 1:00:04 PM > > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > > > > On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote: > > > > For those interested in using your writeback improvements in > > > > production sooner rather than later (primarily with ext3); what > > > > recommendations do you have? Just heavily test our own 2.6.24 > > + > > > your > > > > evolving "close, but not ready for merge" -mm writeback patchset? > > > > > > > Hi Fengguang, Mike, > > > > > > I can add myself to Mikes question. It would be good to know > > a > > > "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has > > been > > > showing quite nice improvement of the overall writeback situation and > > it > > > would be sad to see this [partially] gone in 2.6.24-final. > > Linus > > > apparently already has reverted "...2250b". I will definitely repeat my > > tests > > > with -rc8. and report. > > > > Thank you, Martin. Can you help test this patch on 2.6.24-rc7? > > Maybe we can push it to 2.6.24 after your testing. > > > > Will do tomorrow or friday. Actually a patch against -rc8 would be nicer for me, as I have not looked at -rc7 due to holidays and some of the reported problems with it. Fengguang's latest writeback patch applies cleanly, builds, boots on 2.6.24-rc8. I'll be able to share ext3 performance results (relative to 2.6.24-rc7) shortly. Mike > > > > Fengguang > > --- > > fs/fs-writeback.c | 17 +++++++++++++++-- > > include/linux/writeback.h | 1 + > > mm/page-writeback.c | 9 ++++++--- > > 3 files changed, 22 insertions(+), 5 deletions(-) > > > > --- linux.orig/fs/fs-writeback.c > > +++ linux/fs/fs-writeback.c > > @@ -284,7 +284,16 @@ __sync_single_inode(struct inode *inode, > > * soon as the queue becomes uncongested. > > */ > > inode->i_state |= I_DIRTY_PAGES; > > - requeue_io(inode); > > + if (wbc->nr_to_write <= 0) > > + /* > > + * slice used up: queue for next turn > > + */ > > + requeue_io(inode); > > + else > > + /* > > + * somehow blocked: retry later > > + */ > > + redirty_tail(inode); > > } else { > > /* > > * Otherwise fully redirty the inode so that > > @@ -479,8 +488,12 @@ sync_sb_inodes(struct super_block *sb, s > > iput(inode); > > cond_resched(); > > spin_lock(&inode_lock); > > - if (wbc->nr_to_write <= 0) > > + if (wbc->nr_to_write <= 0) { > > + wbc->more_io = 1; > > break; > > + } > > + if (!list_empty(&sb->s_more_io)) > > + wbc->more_io = 1; > > } > > return; /* Leave any unwritten inodes on s_io */ > > } > > --- linux.orig/include/linux/writeback.h > > +++ linux/include/linux/writeback.h > > @@ -62,6 +62,7 @@ struct writeback_control { > > unsigned for_reclaim:1; /* Invoked from the page > > allocator > > > */ > > unsigned for_writepages:1; /* This is a writepages() call */ > > unsigned range_cyclic:1; /* range_start is cyclic */ > > + unsigned more_io:1; /* more io to be dispatched */ > > }; > > > > /* > > --- linux.orig/mm/page-writeback.c > > +++ linux/mm/page-writeback.c > > @@ -558,6 +558,7 @@ static void background_writeout(unsigned > > global_page_state(NR_UNSTABLE_NFS) < background_thresh > > && min_pages <= 0) > > break; > > + wbc.more_io = 0; > > wbc.encountered_congestion = 0; > > wbc.nr_to_write = MAX_WRITEBACK_PAGES; > > wbc.pages_skipped = 0; > > @@ -565,8 +566,9 @@ static void background_writeout(unsigned > > min_pages -= MAX_WRITEBACK_PAGES - wbc.nr_to_write; > > if (wbc.nr_to_write > 0 || wbc.pages_skipped > 0) { > > /* Wrote less than expected */ > > - congestion_wait(WRITE, HZ/10); > > - if (!wbc.encountered_congestion) > > + if (wbc.encountered_congestion || wbc.more_io) > > + congestion_wait(WRITE, HZ/10); > > + else > > break; > > } > > } > > @@ -631,11 +633,12 @@ static void wb_kupdate(unsigned long arg > > global_page_state(NR_UNSTABLE_NFS) + > > (inodes_stat.nr_inodes - inodes_stat.nr_unused); > > while (nr_to_write > 0) { > > + wbc.more_io = 0; > > wbc.encountered_congestion = 0; > > wbc.nr_to_write = MAX_WRITEBACK_PAGES; > > writeback_inodes(&wbc); > > if (wbc.nr_to_write > 0) { > > - if (wbc.encountered_congestion) > > + if (wbc.encountered_congestion || wbc.more_io) > > congestion_wait(WRITE, HZ/10); > > else > > break; /* All the old data is written */ > > > > > > > > > ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX @ 2008-01-17 13:52 Martin Knoblauch 2008-01-17 16:11 ` Mike Snitzer 0 siblings, 1 reply; 59+ messages in thread From: Martin Knoblauch @ 2008-01-17 13:52 UTC (permalink / raw) To: Fengguang Wu Cc: Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds ----- Original Message ---- > From: Fengguang Wu <wfg@mail.ustc.edu.cn> > To: Martin Knoblauch <knobi@knobisoft.de> > Cc: Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org> > Sent: Wednesday, January 16, 2008 1:00:04 PM > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote: > > > For those interested in using your writeback improvements in > > > production sooner rather than later (primarily with ext3); what > > > recommendations do you have? Just heavily test our own 2.6.24 > + > your > > > evolving "close, but not ready for merge" -mm writeback patchset? > > > > > Hi Fengguang, Mike, > > > > I can add myself to Mikes question. It would be good to know > a > "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has > been > showing quite nice improvement of the overall writeback situation and > it > would be sad to see this [partially] gone in 2.6.24-final. > Linus > apparently already has reverted "...2250b". I will definitely repeat my > tests > with -rc8. and report. > > Thank you, Martin. Can you help test this patch on 2.6.24-rc7? > Maybe we can push it to 2.6.24 after your testing. > Hi Fengguang, something really bad has happened between -rc3 and -rc6. Embarrassingly I did not catch that earlier :-( Compared to the numbers I posted in http://lkml.org/lkml/2007/10/26/208 , dd1 is now at 60 MB/sec (slight plus), while dd2/dd3 suck the same way as in pre 2.6.24. The only test that is still good is mix3, which I attribute to the per-BDI stuff. At the moment I am frantically trying to find when things went down. I did run -rc8 and rc8+yourpatch. No difference to what I see with -rc6. Sorry that I cannot provide any input to your patch. Depressed Martin ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-17 13:52 Martin Knoblauch @ 2008-01-17 16:11 ` Mike Snitzer 0 siblings, 0 replies; 59+ messages in thread From: Mike Snitzer @ 2008-01-17 16:11 UTC (permalink / raw) To: Martin Knoblauch Cc: Fengguang Wu, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds On Jan 17, 2008 8:52 AM, Martin Knoblauch <spamtrap@knobisoft.de> wrote: > > ----- Original Message ---- > > From: Fengguang Wu <wfg@mail.ustc.edu.cn> > > To: Martin Knoblauch <knobi@knobisoft.de> > > Cc: Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org> > > Sent: Wednesday, January 16, 2008 1:00:04 PM > > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > > > On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote: > > > > For those interested in using your writeback improvements in > > > > production sooner rather than later (primarily with ext3); what > > > > recommendations do you have? Just heavily test our own 2.6.24 > > + > > > your > > > > evolving "close, but not ready for merge" -mm writeback patchset? > > > > > > > Hi Fengguang, Mike, > > > > > > I can add myself to Mikes question. It would be good to know > > a > > > "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has > > been > > > showing quite nice improvement of the overall writeback situation and > > it > > > would be sad to see this [partially] gone in 2.6.24-final. > > Linus > > > apparently already has reverted "...2250b". I will definitely repeat my > > tests > > > with -rc8. and report. > > > > Thank you, Martin. Can you help test this patch on 2.6.24-rc7? > > Maybe we can push it to 2.6.24 after your testing. > > > Hi Fengguang, > > something really bad has happened between -rc3 and -rc6. Embarrassingly I did not catch that earlier :-( > > Compared to the numbers I posted in http://lkml.org/lkml/2007/10/26/208 , dd1 is now at 60 MB/sec (slight plus), while dd2/dd3 suck the same way as in pre 2.6.24. The only test that is still good is mix3, which I attribute to the per-BDI stuff. > > At the moment I am frantically trying to find when things went down. I did run -rc8 and rc8+yourpatch. No difference to what I see with -rc6. Sorry that I cannot provide any input to your patch. > > Depressed > Martin Martin, I've backported Peter's perbdi patchset to 2.6.22.x. I can share it with anyone who might be interested. As expected, it has yielded 2.6.24-rcX level scaling. Given the test result matrix you previously posted, 2.6.22.x+perbdi might give you what you're looking for (sans improved writeback that 2.6.24 was thought to be providing). That is, much improved scaling with better O_DIRECT and network throughput. Just a thought... Unfortunately, my priorities (and computing resources) have shifted and I won't be able to thoroughly test Fengguang's new writeback patch on 2.6.24-rc8... whereby missing out on providing justification/testing to others on _some_ improved writeback being included in 2.6.24 final. Not to mention the window for writeback improvement is all but closed considering the 2.6.24-rc8 announcement's 2.6.24 final release timetable. regards, Mike ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX @ 2008-01-17 17:44 Martin Knoblauch 2008-01-17 20:23 ` Mel Gorman 0 siblings, 1 reply; 59+ messages in thread From: Martin Knoblauch @ 2008-01-17 17:44 UTC (permalink / raw) To: Fengguang Wu Cc: Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds ----- Original Message ---- > From: Martin Knoblauch <spamtrap@knobisoft.de> > To: Fengguang Wu <wfg@mail.ustc.edu.cn> > Cc: Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org> > Sent: Thursday, January 17, 2008 2:52:58 PM > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > ----- Original Message ---- > > From: Fengguang Wu > > To: Martin Knoblauch > > Cc: Mike Snitzer ; Peter > Zijlstra > ; jplatte@naasa.net; Ingo Molnar > ; > linux-kernel@vger.kernel.org; > "linux-ext4@vger.kernel.org" > ; Linus > Torvalds > > > Sent: Wednesday, January 16, 2008 1:00:04 PM > > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > > > On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote: > > > > For those interested in using your writeback improvements in > > > > production sooner rather than later (primarily with ext3); what > > > > recommendations do you have? Just heavily test our own 2.6.24 > > + > > > your > > > > evolving "close, but not ready for merge" -mm writeback patchset? > > > > > > > Hi Fengguang, Mike, > > > > > > I can add myself to Mikes question. It would be good to know > > a > > > "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has > > been > > > showing quite nice improvement of the overall writeback situation and > > it > > > would be sad to see this [partially] gone in 2.6.24-final. > > Linus > > > apparently already has reverted "...2250b". I will definitely > repeat > my > > tests > > > with -rc8. and report. > > > > Thank you, Martin. Can you help test this patch on 2.6.24-rc7? > > Maybe we can push it to 2.6.24 after your testing. > > > Hi Fengguang, > > something really bad has happened between -rc3 and > -rc6. > Embarrassingly I did not catch that earlier :-( > > Compared to the numbers I posted > in > http://lkml.org/lkml/2007/10/26/208 , dd1 is now at 60 MB/sec > (slight > plus), while dd2/dd3 suck the same way as in pre 2.6.24. The only > test > that is still good is mix3, which I attribute to the per-BDI stuff. > > At the moment I am frantically trying to find when things went down. > I > did run -rc8 and rc8+yourpatch. No difference to what I see with > -rc6. > Sorry that I cannot provide any input to your patch. > OK, the change happened between rc5 and rc6. Just following a gut feeling, I reverted #commit 81eabcbe0b991ddef5216f30ae91c4b226d54b6d #Author: Mel Gorman <mel@csn.ul.ie> #Date: Mon Dec 17 16:20:05 2007 -0800 # # mm: fix page allocation for larger I/O segments # # In some cases the IO subsystem is able to merge requests if the pages are # adjacent in physical memory. This was achieved in the allocator by having # expand() return pages in physically contiguous order in situations were a # large buddy was split. However, list-based anti-fragmentation changed the # order pages were returned in to avoid searching in buffered_rmqueue() for a # page of the appropriate migrate type. # # This patch restores behaviour of rmqueue_bulk() preserving the physical # order of pages returned by the allocator without incurring increased search # costs for anti-fragmentation. # # Signed-off-by: Mel Gorman <mel@csn.ul.ie> # Cc: James Bottomley <James.Bottomley@steeleye.com> # Cc: Jens Axboe <jens.axboe@oracle.com> # Cc: Mark Lord <mlord@pobox.com # Signed-off-by: Andrew Morton <akpm@linux-foundation.org> # Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> diff -urN linux-2.6.24-rc5/mm/page_alloc.c linux-2.6.24-rc6/mm/page_alloc.c --- linux-2.6.24-rc5/mm/page_alloc.c 2007-12-21 04:14:11.305633890 +0000 +++ linux-2.6.24-rc6/mm/page_alloc.c 2007-12-21 04:14:17.746985697 +0000 @@ -847,8 +847,19 @@ struct page *page = __rmqueue(zone, order, migratetype); if (unlikely(page == NULL)) break; + + /* + * Split buddy pages returned by expand() are received here + * in physical page order. The page is added to the callers and + * list and the list head then moves forward. From the callers + * perspective, the linked list is ordered by page number in + * some conditions. This is useful for IO devices that can + * merge IO requests if the physical pages are ordered + * properly. + */ list_add(&page->lru, list); set_page_private(page, migratetype); + list = &page->lru; } spin_unlock(&zone->lock); return i; This has brought back the good results I observed and reported. I do not know what to make out of this. At least on the systems I care about (HP/DL380g4, dual CPUs, HT-enabled, 8 GB Memory, SmartaArray6i controller with 4x72GB SCSI disks as RAID5 (battery protected writeback cache enabled) and gigabit networking (tg3)) this optimisation is a dissaster. On the other hand, it is not a regression against 2.6.22/23. Those had bad IO scaling to. It would just be a shame to loose an apparently great performance win. is Cheers Martin ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-17 17:44 Martin Knoblauch @ 2008-01-17 20:23 ` Mel Gorman 0 siblings, 0 replies; 59+ messages in thread From: Mel Gorman @ 2008-01-17 20:23 UTC (permalink / raw) To: Martin Knoblauch Cc: Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds, James.Bottomley On (17/01/08 09:44), Martin Knoblauch didst pronounce: > > > > > > > On Wed, Jan 16, 2008 at 01:26:41AM -0800, Martin Knoblauch wrote: > > > > > > For those interested in using your writeback improvements in > > > > > > production sooner rather than later (primarily with ext3); what > > > > > > recommendations do you have? Just heavily test our own 2.6.24 > > > > > > evolving "close, but not ready for merge" -mm writeback patchset? > > > > > > > > > > > > > > > > I can add myself to Mikes question. It would be good to know a > > > > > > > > "roadmap" for the writeback changes. Testing 2.6.24-rcX so far has > > > > been showing quite nice improvement of the overall writeback situation and > > > > it would be sad to see this [partially] gone in 2.6.24-final. > > > > Linus apparently already has reverted "...2250b". I will definitely > > > > repeat my tests with -rc8. and report. > > > > > > > Thank you, Martin. Can you help test this patch on 2.6.24-rc7? > > > Maybe we can push it to 2.6.24 after your testing. > > > > > Hi Fengguang, > > > > something really bad has happened between -rc3 and -rc6. > > Embarrassingly I did not catch that earlier :-( > > Compared to the numbers I posted in > > http://lkml.org/lkml/2007/10/26/208 , dd1 is now at 60 MB/sec > > (slight plus), while dd2/dd3 suck the same way as in pre 2.6.24. > > The only test that is still good is mix3, which I attribute to > > the per-BDI stuff. I suspect that the IO hardware you have is very sensitive to the color of the physical page. I wonder, do you boot the system cleanly and then run these tests? If so, it would be interesting to know what happens if you stress the system first (many kernel compiles for example, basically anything that would use a lot of memory in different ways for some time) to randomise the free lists a bit and then run your test. You'd need to run the test three times for 2.6.23, 2.6.24-rc8 and 2.6.24-rc8 with the patch you identified reverted. > > At the moment I am frantically trying to find when things went down. I > > did run -rc8 and rc8+yourpatch. No difference to what I see with -rc6. > > > > Sorry that I cannot provide any input to your patch. > > OK, the change happened between rc5 and rc6. Just following a gut feeling, I reverted > > #commit 81eabcbe0b991ddef5216f30ae91c4b226d54b6d > #Author: Mel Gorman <mel@csn.ul.ie> > #Date: Mon Dec 17 16:20:05 2007 -0800 > # > # mm: fix page allocation for larger I/O segments > # > # In some cases the IO subsystem is able to merge requests if the pages are > # adjacent in physical memory. This was achieved in the allocator by having > # expand() return pages in physically contiguous order in situations were a > # large buddy was split. However, list-based anti-fragmentation changed the > # order pages were returned in to avoid searching in buffered_rmqueue() for a > # page of the appropriate migrate type. > # > # This patch restores behaviour of rmqueue_bulk() preserving the physical > # order of pages returned by the allocator without incurring increased search > # costs for anti-fragmentation. > # > # Signed-off-by: Mel Gorman <mel@csn.ul.ie> > # Cc: James Bottomley <James.Bottomley@steeleye.com> > # Cc: Jens Axboe <jens.axboe@oracle.com> > # Cc: Mark Lord <mlord@pobox.com > # Signed-off-by: Andrew Morton <akpm@linux-foundation.org> > # Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org> > diff -urN linux-2.6.24-rc5/mm/page_alloc.c linux-2.6.24-rc6/mm/page_alloc.c > --- linux-2.6.24-rc5/mm/page_alloc.c 2007-12-21 04:14:11.305633890 +0000 > +++ linux-2.6.24-rc6/mm/page_alloc.c 2007-12-21 04:14:17.746985697 +0000 > @@ -847,8 +847,19 @@ > struct page *page = __rmqueue(zone, order, migratetype); > if (unlikely(page == NULL)) > break; > + > + /* > + * Split buddy pages returned by expand() are received here > + * in physical page order. The page is added to the callers and > + * list and the list head then moves forward. From the callers > + * perspective, the linked list is ordered by page number in > + * some conditions. This is useful for IO devices that can > + * merge IO requests if the physical pages are ordered > + * properly. > + */ > list_add(&page->lru, list); > set_page_private(page, migratetype); > + list = &page->lru; > } > spin_unlock(&zone->lock); > return i; > > This has brought back the good results I observed and reported. > I do not know what to make out of this. At least on the systems I care > about (HP/DL380g4, dual CPUs, HT-enabled, 8 GB Memory, SmartaArray6i > controller with 4x72GB SCSI disks as RAID5 (battery protected writeback > cache enabled) and gigabit networking (tg3)) this optimisation is a dissaster. > That patch was not an optimisation, it was a regression fix against 2.6.23 and I don't believe reverting it is an option. Other IO hardware benefits from having the allocator supply pages in PFN order. Your controller would seem to suffer when presented with the same situation but I don't know why that is. I've added James to the cc in case he has seen this sort of situation before. > On the other hand, it is not a regression against 2.6.22/23. Those had > bad IO scaling to. It would just be a shame to loose an apparently great > performance win. > is Could you try running your tests again when the system has been stressed with some other workload first? -- Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX
@ 2008-01-17 17:51 Martin Knoblauch
0 siblings, 0 replies; 59+ messages in thread
From: Martin Knoblauch @ 2008-01-17 17:51 UTC (permalink / raw)
To: Mike Snitzer
Cc: Fengguang Wu, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel,
linux-ext4, Linus Torvalds
----- Original Message ----
> From: Mike Snitzer <snitzer@gmail.com>
> To: Martin Knoblauch <spamtrap@knobisoft.de>
> Cc: Fengguang Wu <wfg@mail.ustc.edu.cn>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>
> Sent: Thursday, January 17, 2008 5:11:50 PM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
>
>
> I've backported Peter's perbdi patchset to 2.6.22.x. I can share it
> with anyone who might be interested.
>
> As expected, it has yielded 2.6.24-rcX level scaling. Given the test
> result matrix you previously posted, 2.6.22.x+perbdi might give you
> what you're looking for (sans improved writeback that 2.6.24 was
> thought to be providing). That is, much improved scaling with better
> O_DIRECT and network throughput. Just a thought...
>
> Unfortunately, my priorities (and computing resources) have shifted
> and I won't be able to thoroughly test Fengguang's new writeback patch
> on 2.6.24-rc8... whereby missing out on providing
> justification/testing to others on _some_ improved writeback being
> included in 2.6.24 final.
>
> Not to mention the window for writeback improvement is all but closed
> considering the 2.6.24-rc8 announcement's 2.6.24 final release
> timetable.
>
Mike,
thanks for the offer, but the improved throughput is my #1 priority nowadays.
And while the better scaling for different targets is nothing to frown upon, the
much better scaling when writing to the same target would have been the big
winner for me.
Anyway, I located the "offending" commit. Lets see what the experts say.
Cheers
Martin
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX @ 2008-01-17 21:50 Martin Knoblauch 2008-01-17 22:12 ` Mel Gorman 0 siblings, 1 reply; 59+ messages in thread From: Martin Knoblauch @ 2008-01-17 21:50 UTC (permalink / raw) To: Mel Gorman Cc: Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds, James.Bottomley ----- Original Message ---- > From: Mel Gorman <mel@csn.ul.ie> > To: Martin Knoblauch <spamtrap@knobisoft.de> > Cc: Fengguang Wu <wfg@mail.ustc.edu.cn>; Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>; James.Bottomley@steeleye.com > Sent: Thursday, January 17, 2008 9:23:57 PM > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > On (17/01/08 09:44), Martin Knoblauch didst pronounce: > > > > > > > > On Wed, Jan 16, 2008 at 01:26:41AM -0800, > Martin > Knoblauch wrote: > > > > > > > For those interested in using your writeback > improvements > in > > > > > > > production sooner rather than later (primarily with > ext3); > what > > > > > > > recommendations do you have? Just heavily test our > own > 2.6.24 > > > > > > > evolving "close, but not ready for merge" -mm > writeback > patchset? > > > > > > > > > > > > > > > > > > > I can add myself to Mikes question. It would be good to > know > a > > > > > > > > > > "roadmap" for the writeback changes. Testing 2.6.24-rcX so > far > has > > > > > been showing quite nice improvement of the overall > writeback > situation and > > > > > it would be sad to see this [partially] gone in 2.6.24-final. > > > > > Linus apparently already has reverted "...2250b". I > will > definitely > > > > > repeat my tests with -rc8. and report. > > > > > > > > > Thank you, Martin. Can you help test this patch on 2.6.24-rc7? > > > > Maybe we can push it to 2.6.24 after your testing. > > > > > > > Hi Fengguang, > > > > > > something really bad has happened between -rc3 and -rc6. > > > Embarrassingly I did not catch that earlier :-( > > > Compared to the numbers I posted in > > > http://lkml.org/lkml/2007/10/26/208 , dd1 is now at 60 MB/sec > > > (slight plus), while dd2/dd3 suck the same way as in pre 2.6.24. > > > The only test that is still good is mix3, which I attribute to > > > the per-BDI stuff. > > I suspect that the IO hardware you have is very sensitive to the > color of the physical page. I wonder, do you boot the system cleanly > and then run these tests? If so, it would be interesting to know what > happens if you stress the system first (many kernel compiles for example, > basically anything that would use a lot of memory in different ways for some > time) to randomise the free lists a bit and then run your test. You'd need to run > the test three times for 2.6.23, 2.6.24-rc8 and 2.6.24-rc8 with the patch you > identified reverted. > The effect is defintely depending on the IO hardware. I performed the same tests on a different box with an AACRAID controller and there things look different. Basically the "offending" commit helps seingle stream performance on that box, while dual/triple stream are not affected. So I suspect that the CCISS is just not behaving well. And yes, the tests are usually done on a freshly booted box. Of course, I repeat them a few times. On the CCISS box the numbers are very constant. On the AACRAID box they vary quite a bit. I can certainly stress the box before doing the tests. Please define "many" for the kernel compiles :-) > > > > OK, the change happened between rc5 and rc6. Just following a > > gut feeling, I reverted > > > > #commit 81eabcbe0b991ddef5216f30ae91c4b226d54b6d > > #Author: Mel Gorman > > #Date: Mon Dec 17 16:20:05 2007 -0800 > > # > > > > This has brought back the good results I observed and reported. > > I do not know what to make out of this. At least on the systems > > I care about (HP/DL380g4, dual CPUs, HT-enabled, 8 GB Memory, > > SmartaArray6i controller with 4x72GB SCSI disks as RAID5 (battery > > protected writeback cache enabled) and gigabit networking (tg3)) this > > optimisation is a dissaster. > > > > That patch was not an optimisation, it was a regression fix > against 2.6.23 and I don't believe reverting it is an option. Other IO > hardware benefits from having the allocator supply pages in PFN order. I think this late in the 2.6.24 game we just should leave things as they are. But we should try to find a way to make CCISS faster, as it apparently can be faster. > Your controller would seem to suffer when presented with the same situation > but I don't know why that is. I've added James to the cc in case he has seen this > sort of situation before. > > > On the other hand, it is not a regression against 2.6.22/23. Those > > had bad IO scaling to. It would just be a shame to loose an apparently > > great performance win. > > Could you try running your tests again when the system has been > stressed with some other workload first? > Will do. Cheers Martin ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-17 21:50 Martin Knoblauch @ 2008-01-17 22:12 ` Mel Gorman 0 siblings, 0 replies; 59+ messages in thread From: Mel Gorman @ 2008-01-17 22:12 UTC (permalink / raw) To: Martin Knoblauch Cc: Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds, James.Bottomley On (17/01/08 13:50), Martin Knoblauch didst pronounce: > > <mail manglement snipped> > > The effect is defintely depending on the IO hardware. I performed the same tests > on a different box with an AACRAID controller and there things look different. I take it different also means it does not show this odd performance behaviour and is similar whether the patch is applied or not? > Basically > the "offending" commit helps seingle stream performance on that box, while dual/triple > stream are not affected. So I suspect that the CCISS is just not behaving well. > > And yes, the tests are usually done on a freshly booted box. Of course, I repeat them > a few times. On the CCISS box the numbers are very constant. On the AACRAID box > they vary quite a bit. > > I can certainly stress the box before doing the tests. Please define "many" for the kernel > compiles :-) > With 8GiB of RAM, try making 24 copies of the kernel and compiling them all simultaneously. Running that for for 20-30 minutes should be enough to randomise the freelists affecting what color of page is used for the dd test. > > > OK, the change happened between rc5 and rc6. Just following a > > > gut feeling, I reverted > > > > > > #commit 81eabcbe0b991ddef5216f30ae91c4b226d54b6d > > > #Author: Mel Gorman > > > #Date: Mon Dec 17 16:20:05 2007 -0800 > > > # > > > > > > > This has brought back the good results I observed and reported. > > > I do not know what to make out of this. At least on the systems > > > I care about (HP/DL380g4, dual CPUs, HT-enabled, 8 GB Memory, > > > SmartaArray6i controller with 4x72GB SCSI disks as RAID5 (battery > > > protected writeback cache enabled) and gigabit networking (tg3)) this > > > optimisation is a dissaster. > > > > > > > That patch was not an optimisation, it was a regression fix > > against 2.6.23 and I don't believe reverting it is an option. Other IO > > hardware benefits from having the allocator supply pages in PFN order. > > I think this late in the 2.6.24 game we just should leave things as they are. But > we should try to find a way to make CCISS faster, as it apparently can be faster. > > > Your controller would seem to suffer when presented with the same situation > > but I don't know why that is. I've added James to the cc in case he has seen this > > sort of situation before. > > > > > On the other hand, it is not a regression against 2.6.22/23. Those > > > > had bad IO scaling to. It would just be a shame to loose an apparently > > > > great performance win. > > > > Could you try running your tests again when the system has been > > stressed with some other workload first? > > > > Will do. > Thanks -- Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX @ 2008-01-18 8:19 Martin Knoblauch 2008-01-18 16:01 ` Mel Gorman 0 siblings, 1 reply; 59+ messages in thread From: Martin Knoblauch @ 2008-01-18 8:19 UTC (permalink / raw) To: Mel Gorman Cc: Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds, James.Bottomley ----- Original Message ---- > From: Mel Gorman <mel@csn.ul.ie> > To: Martin Knoblauch <spamtrap@knobisoft.de> > Cc: Fengguang Wu <wfg@mail.ustc.edu.cn>; Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; Linus Torvalds <torvalds@linux-foundation.org>; James.Bottomley@steeleye.com > Sent: Thursday, January 17, 2008 11:12:21 PM > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > On (17/01/08 13:50), Martin Knoblauch didst pronounce: > > > > > > > The effect is defintely depending on the IO hardware. > > performed the same tests > > on a different box with an AACRAID controller and there things > > look different. > > I take it different also means it does not show this odd performance > behaviour and is similar whether the patch is applied or not? > Here are the numbers (MB/s) from the AACRAID box, after a fresh boot: Test 2.6.19.2 2.6.24-rc6 2.6.24-rc6-81eabcbe0b991ddef5216f30ae91c4b226d54b6d dd1 325 350 290 dd1-dir 180 160 160 dd2 2x90 2x113 2x110 dd2-dir 2x120 2x92 2x93 dd3 3x54 3x70 3x70 dd3-dir 3x83 3x64 3x64 mix3 55,2x30 400,2x25 310,2x25 What we are seing here is that: a) DIRECT IO takes a much bigger hit (2.6.19 vs. 2.6.24) on this IO system compared to the CCISS box b) Reverting your patch hurts single stream c) dual/triple stream are not affected by your patch and are improved over 2.6.19 d) the mix3 performance is improved compared to 2.6.19. d1) reverting your patch hurts the local-disk part of mix3 e) the AACRAID setup is definitely faster than the CCISS. So, on this box your patch is definitely needed to get the pre-2.6.24 performance when writing a single big file. Actually things on the CCISS box might be even more complicated. I forgot the fact that on that box we have ext2/LVM/DM/Hardware, while on the AACRAID box we have ext2/Hardware. Do you think that the LVM/MD are sensitive to the page order/coloring? Anyway: does your patch only address this performance issue, or are there also data integrity concerns without it? I may consider reverting the patch for my production environment. It really helps two thirds of my boxes big time, while it does not hurt the other third that much :-) > > > > I can certainly stress the box before doing the tests. Please > > define "many" for the kernel compiles :-) > > > > With 8GiB of RAM, try making 24 copies of the kernel and compiling them > all simultaneously. Running that for for 20-30 minutes should be enough > to randomise the freelists affecting what color of page is used for the > dd test. > ouch :-) OK, I will try that. Martin ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-18 8:19 Martin Knoblauch @ 2008-01-18 16:01 ` Mel Gorman 2008-01-18 17:46 ` Linus Torvalds 0 siblings, 1 reply; 59+ messages in thread From: Mel Gorman @ 2008-01-18 16:01 UTC (permalink / raw) To: Martin Knoblauch Cc: Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, Linus Torvalds, James.Bottomley On (18/01/08 00:19), Martin Knoblauch didst pronounce: > > > The effect is defintely depending on the IO hardware. > > > performed the same tests > > > on a different box with an AACRAID controller and there things > > > look different. > > > > I take it different also means it does not show this odd performance > > behaviour and is similar whether the patch is applied or not? > > > > Here are the numbers (MB/s) from the AACRAID box, after a fresh boot: > > Test 2.6.19.2 2.6.24-rc6 2.6.24-rc6-81eabcbe0b991ddef5216f30ae91c4b226d54b6d > dd1 325 350 290 > dd1-dir 180 160 160 > dd2 2x 90 2x113 2x110 > dd2-dir 2x120 2x 92 2x 93 > dd3 3x 54 3x 70 3x 70 > dd3-dir 3x 83 3x 64 3x 64 > mix3 55,2x 30 400,2x 25 310,2x 25 > > What we are seing here is that: > > a) DIRECT IO takes a much bigger hit (2.6.19 vs. 2.6.24) on this IO system compared to the CCISS box > b) Reverting your patch hurts single stream Right, and this is consistent with other complaints about the PFN of the page mattering to some hardware. > c) dual/triple stream are not affected by your patch and are improved over 2.6.19 I am not very surprised. The callers to the page allocator are probably making no special effort to get a batch of pages in PFN-order. They are just assuming that subsequent calls give contiguous pages. With two or more threads involved, there will not be a correlation between physical pages and what is on disk any more. > d) the mix3 performance is improved compared to 2.6.19. > d1) reverting your patch hurts the local-disk part of mix3 > e) the AACRAID setup is definitely faster than the CCISS. > > So, on this box your patch is definitely needed to get the pre-2.6.24 performance > when writing a single big file. > > Actually things on the CCISS box might be even more complicated. I forgot the fact > that on that box we have ext2/LVM/DM/Hardware, while on the AACRAID box we have > ext2/Hardware. Do you think that the LVM/MD are sensitive to the page order/coloring? > I don't have enough experience with LVM setups to make an intelligent guess. > Anyway: does your patch only address this performance issue, or are there also > data integrity concerns without it? Performance issue only. There are no data integrity concerns with that patch. > I may consider reverting the patch for my > production environment. It really helps two thirds of my boxes big time, while it does > not hurt the other third that much :-) > That is certainly an option. > > > > > > I can certainly stress the box before doing the tests. Please > > > define "many" for the kernel compiles :-) > > > > > > > With 8GiB of RAM, try making 24 copies of the kernel and compiling them > > all simultaneously. Running that for for 20-30 minutes should be enough > > > to randomise the freelists affecting what color of page is used for the > > dd test. > > > > ouch :-) OK, I will try that. > Thanks. -- Mel Gorman Part-time Phd Student Linux Technology Center University of Limerick IBM Dublin Software Lab ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-18 16:01 ` Mel Gorman @ 2008-01-18 17:46 ` Linus Torvalds 2008-01-18 19:01 ` Martin Knoblauch 2008-01-18 20:00 ` Mike Snitzer 0 siblings, 2 replies; 59+ messages in thread From: Linus Torvalds @ 2008-01-18 17:46 UTC (permalink / raw) To: Mel Gorman Cc: Martin Knoblauch, Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, James.Bottomley On Fri, 18 Jan 2008, Mel Gorman wrote: > > Right, and this is consistent with other complaints about the PFN of the > page mattering to some hardware. I don't think it's actually the PFN per se. I think it's simply that some controllers (quite probably affected by both driver and hardware limits) have some subtle interactions with the size of the IO commands. For example, let's say that you have a controller that has some limit X on the size of IO in flight (whether due to hardware or driver issues doesn't really matter) in addition to a limit on the size of the scatter-gather size. They all tend to have limits, and they differ. Now, the PFN doesn't matter per se, but the allocation pattern definitely matters for whether the IO's are physically contiguous, and thus matters for the size of the scatter-gather thing. Now, generally the rule-of-thumb is that you want big commands, so physical merging is good for you, but I could well imagine that the IO limits interact, and end up hurting each other. Let's say that a better allocation order allows for bigger contiguous physical areas, and thus fewer scatter-gather entries. What does that result in? The obvious answer is "Better performance obviously, because the controller needs to do fewer scatter-gather lookups, and the requests are bigger, because there are fewer IO's that hit scatter-gather limits!" Agreed? Except maybe the *real* answer for some controllers end up being "Worse performance, because individual commands grow because they don't hit the per-command limits, but now we hit the global size-in-flight limits and have many fewer of these good commands in flight. And while the commands are larger, it means that there are fewer outstanding commands, which can mean that the disk cannot scheduling things as well, or makes high latency of command generation by the controller much more visible because there aren't enough concurrent requests queued up to hide it" Is this the reason? I have no idea. But somebody who knows the AACRAID hardware and driver limits might think about interactions like that. Sometimes you actually might want to have smaller individual commands if there is some other limit that means that it can be more advantageous to have many small requests over a few big onees. RAID might well make it worse. Maybe small requests work better because they are simpler to schedule because they only hit one disk (eg if you have simple striping)! So that's another reason why one *large* request may actually be slower than two requests half the size, even if it's against the "normal rule". And it may be that that AACRAID box takes a big hit on DIO exactly because DIO has been optimized almost purely for making one command as big as possible. Just a theory. Linus ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-18 17:46 ` Linus Torvalds @ 2008-01-18 19:01 ` Martin Knoblauch 2008-01-18 19:23 ` Linus Torvalds 2008-01-22 14:39 ` Alasdair G Kergon 2008-01-18 20:00 ` Mike Snitzer 1 sibling, 2 replies; 59+ messages in thread From: Martin Knoblauch @ 2008-01-18 19:01 UTC (permalink / raw) To: Linus Torvalds, Mel Gorman Cc: Martin Knoblauch, Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, James.Bottomley --- Linus Torvalds <torvalds@linux-foundation.org> wrote: > > > On Fri, 18 Jan 2008, Mel Gorman wrote: > > > > Right, and this is consistent with other complaints about the PFN > > of the page mattering to some hardware. > > I don't think it's actually the PFN per se. > > I think it's simply that some controllers (quite probably affected by > both driver and hardware limits) have some subtle interactions with > the size of the IO commands. > > For example, let's say that you have a controller that has some limit > X on the size of IO in flight (whether due to hardware or driver > issues doesn't really matter) in addition to a limit on the size > of the scatter-gather size. They all tend to have limits, and > they differ. > > Now, the PFN doesn't matter per se, but the allocation pattern > definitely matters for whether the IO's are physically > contiguous, and thus matters for the size of the scatter-gather > thing. > > Now, generally the rule-of-thumb is that you want big commands, so > physical merging is good for you, but I could well imagine that the > IO limits interact, and end up hurting each other. Let's say that a > better allocation order allows for bigger contiguous physical areas, > and thus fewer scatter-gather entries. > > What does that result in? The obvious answer is > > "Better performance obviously, because the controller needs to do > fewer scatter-gather lookups, and the requests are bigger, because > there are fewer IO's that hit scatter-gather limits!" > > Agreed? > > Except maybe the *real* answer for some controllers end up being > > "Worse performance, because individual commands grow because they > don't hit the per-command limits, but now we hit the global > size-in-flight limits and have many fewer of these good commands in > flight. And while the commands are larger, it means that there > are fewer outstanding commands, which can mean that the disk > cannot scheduling things as well, or makes high latency of command > generation by the controller much more visible because there aren't > enough concurrent requests queued up to hide it" > > Is this the reason? I have no idea. But somebody who knows the > AACRAID hardware and driver limits might think about interactions > like that. Sometimes you actually might want to have smaller > individual commands if there is some other limit that means that > it can be more advantageous to have many small requests over a > few big onees. > > RAID might well make it worse. Maybe small requests work better > because they are simpler to schedule because they only hit one > disk (eg if you have simple striping)! So that's another reason > why one *large* request may actually be slower than two requests > half the size, even if it's against the "normal rule". > > And it may be that that AACRAID box takes a big hit on DIO > exactly because DIO has been optimized almost purely for making > one command as big as possible. > > Just a theory. > > Linus just to make one thing clear - I am not so much concerned about the performance of AACRAID. It is OK with or without Mel's patch. It is better with Mel's patch. The regression in DIO compared to 2.6.19.2 is completely independent of Mel's stuff. What interests me much more is the behaviour of the CCISS+LVM based system. Here I see a huge benefit of reverting Mel's patch. I dirtied the system after reboot as Mel suggested (24 parallel kernel build) and repeated the tests. The dirtying did not make any difference. Here are the results: Test -rc8 -rc8-without-Mels-Patch dd1 57 94 dd1-dir 87 86 dd2 2x8.5 2x45 dd2-dir 2x43 2x43 dd3 3x7 3x30 dd3-dir 3x28.5 3x28.5 mix3 59,2x25 98,2x24 The big IO size with Mel's patch really has a devastating effect on the parallel write. Nowhere near the value one would expect, while the numbers are perfect without Mel's patch as in rc1-rc5. To bad I did not see this earlier. Maybe we could have found a solution for .24. At least, rc1-rc5 have shown that the CCISS system can do well. Now the question is which part of the system does not cope well with the larger IO sizes? Is it the CCISS controller, LVM or both. I am open to suggestions on how to debug that. Cheers Martin ------------------------------------------------------ Martin Knoblauch email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-18 19:01 ` Martin Knoblauch @ 2008-01-18 19:23 ` Linus Torvalds 2008-01-22 14:39 ` Alasdair G Kergon 1 sibling, 0 replies; 59+ messages in thread From: Linus Torvalds @ 2008-01-18 19:23 UTC (permalink / raw) To: Martin Knoblauch Cc: Mel Gorman, Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, James.Bottomley On Fri, 18 Jan 2008, Martin Knoblauch wrote: > > just to make one thing clear - I am not so much concerned about the > performance of AACRAID. It is OK with or without Mel's patch. It is > better with Mel's patch. The regression in DIO compared to 2.6.19.2 is > completely independent of Mel's stuff. > > What interests me much more is the behaviour of the CCISS+LVM based > system. Here I see a huge benefit of reverting Mel's patch. Ok, I just got your usage cases confused. The argument stays the same: some controllers/drivers may have subtle behavioural differences that come from the IO limits themselves. So it wasn't AACRAID, it was CCISS+LVM. The argument is the same: it may well be that the *bigger* IO sizes are actually what hurts, even if the conventional wisdom is traditionally that bigger submissions are better. > At least, rc1-rc5 have shown that the CCISS system can do well. Now > the question is which part of the system does not cope well with the > larger IO sizes? Is it the CCISS controller, LVM or both. I am open to > suggestions on how to debug that. I think you need to ask the MD/DM people for suggestions.. Who aren't cc'd here. Linus ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-18 19:01 ` Martin Knoblauch 2008-01-18 19:23 ` Linus Torvalds @ 2008-01-22 14:39 ` Alasdair G Kergon 1 sibling, 0 replies; 59+ messages in thread From: Alasdair G Kergon @ 2008-01-22 14:39 UTC (permalink / raw) To: Martin Knoblauch Cc: Linus Torvalds, Mel Gorman, Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, James.Bottomley, Jens Axboe, Milan Broz, Neil Brown On Fri, Jan 18, 2008 at 11:01:11AM -0800, Martin Knoblauch wrote: > At least, rc1-rc5 have shown that the CCISS system can do well. Now > the question is which part of the system does not cope well with the > larger IO sizes? Is it the CCISS controller, LVM or both. I am open to > suggestions on how to debug that. What is your LVM device configuration? E.g. 'dmsetup table' and 'dmsetup info -c' output. Some configurations lead to large IOs getting split up on the way through device-mapper. See if these patches make any difference: http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/ dm-md-merge_bvec_fn-with-separate-bdev-and-sector.patch dm-introduce-merge_bvec_fn.patch dm-linear-add-merge.patch dm-table-remove-merge_bvec-sector-restriction.patch Alasdair -- agk@redhat.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-18 17:46 ` Linus Torvalds 2008-01-18 19:01 ` Martin Knoblauch @ 2008-01-18 20:00 ` Mike Snitzer 2008-01-18 22:47 ` Mike Snitzer 1 sibling, 1 reply; 59+ messages in thread From: Mike Snitzer @ 2008-01-18 20:00 UTC (permalink / raw) To: Linus Torvalds Cc: Mel Gorman, Martin Knoblauch, Fengguang Wu, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, James.Bottomley On Jan 18, 2008 12:46 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote: > > > On Fri, 18 Jan 2008, Mel Gorman wrote: > > > > Right, and this is consistent with other complaints about the PFN of the > > page mattering to some hardware. > > I don't think it's actually the PFN per se. > > I think it's simply that some controllers (quite probably affected by both > driver and hardware limits) have some subtle interactions with the size of > the IO commands. > > For example, let's say that you have a controller that has some limit X on > the size of IO in flight (whether due to hardware or driver issues doesn't > really matter) in addition to a limit on the size of the scatter-gather > size. They all tend to have limits, and they differ. > > Now, the PFN doesn't matter per se, but the allocation pattern definitely > matters for whether the IO's are physically contiguous, and thus matters > for the size of the scatter-gather thing. > > Now, generally the rule-of-thumb is that you want big commands, so > physical merging is good for you, but I could well imagine that the IO > limits interact, and end up hurting each other. Let's say that a better > allocation order allows for bigger contiguous physical areas, and thus > fewer scatter-gather entries. > > What does that result in? The obvious answer is > > "Better performance obviously, because the controller needs to do fewer > scatter-gather lookups, and the requests are bigger, because there are > fewer IO's that hit scatter-gather limits!" > > Agreed? > > Except maybe the *real* answer for some controllers end up being > > "Worse performance, because individual commands grow because they don't > hit the per-command limits, but now we hit the global size-in-flight > limits and have many fewer of these good commands in flight. And while > the commands are larger, it means that there are fewer outstanding > commands, which can mean that the disk cannot scheduling things > as well, or makes high latency of command generation by the controller > much more visible because there aren't enough concurrent requests > queued up to hide it" > > Is this the reason? I have no idea. But somebody who knows the AACRAID > hardware and driver limits might think about interactions like that. > Sometimes you actually might want to have smaller individual commands if > there is some other limit that means that it can be more advantageous to > have many small requests over a few big onees. > > RAID might well make it worse. Maybe small requests work better because > they are simpler to schedule because they only hit one disk (eg if you > have simple striping)! So that's another reason why one *large* request > may actually be slower than two requests half the size, even if it's > against the "normal rule". > > And it may be that that AACRAID box takes a big hit on DIO exactly because > DIO has been optimized almost purely for making one command as big as > possible. > > Just a theory. Oddly enough, I'm seeing the opposite here with 2.6.22.16 w/ AACRAID configured with 5 LUNS (each 2disk HW RAID0, 1024k stripesz). That is, with dd the avgrqsiz (from iostat) shows DIO to be ~130k whereas non-DIO is a mere ~13k! (NOTE: with aacraid, max_hw_sectors_kb=192) DIO cmdline: dd if=/dev/zero of=/dev/sdX bs=8192k count=1k non-DIO cmdline: dd if=/dev/zero of=/dev/sdX bs=8192k count=1k DIO is ~80MB/s on all 5 LUNs for a total of ~400MB/s non-DIO is only ~12MB on all 5 LUNs for a mere ~70MB/s aggregate (deadline w/ nr_requests=32) Calls into question the theory of small requests being beneficial for AACRAID. Martin, what are you seeing for the avg request size when you're conducting your AACRAID tests? I can fire up 2.6.24-rc8 in short order to see if things are vastly improved (as Martin seems to indicate that he is happy with AACRAID on 2.6.24-rc8). Although even Martin's AACRAID numbers from 2.6.19.2 are still quite good (relative to mine). Martin can you share any tuning you may have done to get AACRAID to where it is for you right now? regards, Mike ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-18 20:00 ` Mike Snitzer @ 2008-01-18 22:47 ` Mike Snitzer 0 siblings, 0 replies; 59+ messages in thread From: Mike Snitzer @ 2008-01-18 22:47 UTC (permalink / raw) To: Linus Torvalds Cc: Mel Gorman, Martin Knoblauch, Fengguang Wu, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, James.Bottomley On Jan 18, 2008 3:00 PM, Mike Snitzer <snitzer@gmail.com> wrote: > > On Jan 18, 2008 12:46 PM, Linus Torvalds <torvalds@linux-foundation.org> wrote: > > > > > > On Fri, 18 Jan 2008, Mel Gorman wrote: > > > > > > Right, and this is consistent with other complaints about the PFN of the > > > page mattering to some hardware. > > > > I don't think it's actually the PFN per se. > > > > I think it's simply that some controllers (quite probably affected by both > > driver and hardware limits) have some subtle interactions with the size of > > the IO commands. > > > > For example, let's say that you have a controller that has some limit X on > > the size of IO in flight (whether due to hardware or driver issues doesn't > > really matter) in addition to a limit on the size of the scatter-gather > > size. They all tend to have limits, and they differ. > > > > Now, the PFN doesn't matter per se, but the allocation pattern definitely > > matters for whether the IO's are physically contiguous, and thus matters > > for the size of the scatter-gather thing. > > > > Now, generally the rule-of-thumb is that you want big commands, so > > physical merging is good for you, but I could well imagine that the IO > > limits interact, and end up hurting each other. Let's say that a better > > allocation order allows for bigger contiguous physical areas, and thus > > fewer scatter-gather entries. > > > > What does that result in? The obvious answer is > > > > "Better performance obviously, because the controller needs to do fewer > > scatter-gather lookups, and the requests are bigger, because there are > > fewer IO's that hit scatter-gather limits!" > > > > Agreed? > > > > Except maybe the *real* answer for some controllers end up being > > > > "Worse performance, because individual commands grow because they don't > > hit the per-command limits, but now we hit the global size-in-flight > > limits and have many fewer of these good commands in flight. And while > > the commands are larger, it means that there are fewer outstanding > > commands, which can mean that the disk cannot scheduling things > > as well, or makes high latency of command generation by the controller > > much more visible because there aren't enough concurrent requests > > queued up to hide it" > > > > Is this the reason? I have no idea. But somebody who knows the AACRAID > > hardware and driver limits might think about interactions like that. > > Sometimes you actually might want to have smaller individual commands if > > there is some other limit that means that it can be more advantageous to > > have many small requests over a few big onees. > > > > RAID might well make it worse. Maybe small requests work better because > > they are simpler to schedule because they only hit one disk (eg if you > > have simple striping)! So that's another reason why one *large* request > > may actually be slower than two requests half the size, even if it's > > against the "normal rule". > > > > And it may be that that AACRAID box takes a big hit on DIO exactly because > > DIO has been optimized almost purely for making one command as big as > > possible. > > > > Just a theory. > > Oddly enough, I'm seeing the opposite here with 2.6.22.16 w/ AACRAID > configured with 5 LUNS (each 2disk HW RAID0, 1024k stripesz). That > is, with dd the avgrqsiz (from iostat) shows DIO to be ~130k whereas > non-DIO is a mere ~13k! (NOTE: with aacraid, max_hw_sectors_kb=192) ... > I can fire up 2.6.24-rc8 in short order to see if things are vastly > improved (as Martin seems to indicate that he is happy with AACRAID on > 2.6.24-rc8). Although even Martin's AACRAID numbers from 2.6.19.2 are > still quite good (relative to mine). Martin can you share any tuning > you may have done to get AACRAID to where it is for you right now? I can confirm 2.6.24-rc8 behaves like Martin has posted for the AACRAID. Slower DIO with smaller avgreqsiz. Much faster buffered IO (for my config anyway) with a much larger avgreqsiz (180K). I have no idea why 2.6.22.16's request size on non-DIO is _so_ small... Mike ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX @ 2008-01-19 10:24 Martin Knoblauch 0 siblings, 0 replies; 59+ messages in thread From: Martin Knoblauch @ 2008-01-19 10:24 UTC (permalink / raw) To: Mike Snitzer, Linus Torvalds Cc: Mel Gorman, Fengguang Wu, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, James.Bottomley ----- Original Message ---- > From: Mike Snitzer <snitzer@gmail.com> > To: Linus Torvalds <torvalds@linux-foundation.org> > Cc: Mel Gorman <mel@csn.ul.ie>; Martin Knoblauch <spamtrap@knobisoft.de>; Fengguang Wu <wfg@mail.ustc.edu.cn>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; James.Bottomley@steeleye.com > Sent: Friday, January 18, 2008 11:47:02 PM > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > > I can fire up 2.6.24-rc8 in short order to see if things are vastly > > improved (as Martin seems to indicate that he is happy with > > AACRAID on 2.6.24-rc8). Although even Martin's AACRAID > > numbers from 2.6.19.2 > > are still quite good (relative to mine). Martin can you share any tuning > > you may have done to get AACRAID to where it is for you right now? Mike, I have always been happy with the AACRAID box compared to the CCISS system. Even with the "regression" in 2.6.24-rc1..rc5 it was more than acceptable to me. For me the differences between 2.6.19 and 2.6.24-rc8 on the AACRAID setup are: - 11% (single stream) to 25% (dual/triple stream) regression in DIO. Something I do not care much about. I just measure it for reference. + the very nice behaviour when writing to different targets (mix3), which I attribute to Peter's per-dbi stuff. And until -rc6 I was extremely pleased with the cool speedup I saw on my CCISS boxes. This would have been the next "production" kernel for me. But lets discuss this under a seperate topic. It has nothing to do with the original wait-io issue. Oh, before I forget. There has been no tuning for the AACRAID. The system is an IBM x3650 with built in AACRAID and battery backed write cache. The disks are 6x142GB/15krpm in a RAID5 setup. I see one big difference between your an my tests. I do 1MB writes to simulate the behaviour of the real applications, while yours seem to be much smaller. Cheers Martin ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX @ 2008-01-22 15:25 Martin Knoblauch 2008-01-22 23:40 ` Alasdair G Kergon 0 siblings, 1 reply; 59+ messages in thread From: Martin Knoblauch @ 2008-01-22 15:25 UTC (permalink / raw) To: Alasdair G Kergon Cc: Linus Torvalds, Mel Gorman, Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, James.Bottomley, Jens Axboe, Milan Broz, Neil Brown ------------------------------------------------------ Martin Knoblauch email: k n o b i AT knobisoft DOT de www: http://www.knobisoft.de ----- Original Message ---- > From: Alasdair G Kergon <agk@redhat.com> > To: Martin Knoblauch <spamtrap@knobisoft.de> > Cc: Linus Torvalds <torvalds@linux-foundation.org>; Mel Gorman <mel@csn.ul.ie>; Fengguang Wu <wfg@mail.ustc.edu.cn>; Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; James.Bottomley@steeleye.com; Jens Axboe <jens.axboe@oracle.com>; Milan Broz <mbroz@redhat.com>; Neil Brown <neilb@suse.de> > Sent: Tuesday, January 22, 2008 3:39:33 PM > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > On Fri, Jan 18, 2008 at 11:01:11AM -0800, Martin Knoblauch wrote: > > At least, rc1-rc5 have shown that the CCISS system can do well. Now > > the question is which part of the system does not cope well with the > > larger IO sizes? Is it the CCISS controller, LVM or both. I am > open > to > > suggestions on how to debug that. > > What is your LVM device configuration? > E.g. 'dmsetup table' and 'dmsetup info -c' output. > Some configurations lead to large IOs getting split up on the > way > through > device-mapper. > Hi Alastair, here is the output, the filesystem in question is on LogVol02: [root@lpsdm52 ~]# dmsetup table VolGroup00-LogVol02: 0 350945280 linear 104:2 67109248 VolGroup00-LogVol01: 0 8388608 linear 104:2 418054528 VolGroup00-LogVol00: 0 67108864 linear 104:2 384 [root@lpsdm52 ~]# dmsetup info -c Name Maj Min Stat Open Targ Event UUID VolGroup00-LogVol02 253 1 L--w 1 1 0 LVM-IV4PeE8cdxA3piC1qk79GY9PE9OC4OgmOZ4OzOgGQIdF3qDx6fJmlZukXXLIy39R VolGroup00-LogVol01 253 2 L--w 1 1 0 LVM-IV4PeE8cdxA3piC1qk79GY9PE9OC4Ogmfn2CcAd2Fh7i48twe8PZc2XK5bSOe1Fq VolGroup00-LogVol00 253 0 L--w 1 1 0 LVM-IV4PeE8cdxA3piC1qk79GY9PE9OC4OgmfYjxQKFP3zw2fGsezJN7ypSrfmP7oSvE > See if these patches make any difference: > http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/ > > dm-md-merge_bvec_fn-with-separate-bdev-and-sector.patch > dm-introduce-merge_bvec_fn.patch > dm-linear-add-merge.patch > dm-table-remove-merge_bvec-sector-restriction.patch > thanks for the suggestion. Are they supposed to apply to mainline? Cheers Martin ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX 2008-01-22 15:25 Martin Knoblauch @ 2008-01-22 23:40 ` Alasdair G Kergon 0 siblings, 0 replies; 59+ messages in thread From: Alasdair G Kergon @ 2008-01-22 23:40 UTC (permalink / raw) To: Martin Knoblauch Cc: Linus Torvalds, Mel Gorman, Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, James.Bottomley, Jens Axboe, Milan Broz, Neil Brown On Tue, Jan 22, 2008 at 07:25:15AM -0800, Martin Knoblauch wrote: > [root@lpsdm52 ~]# dmsetup table > VolGroup00-LogVol02: 0 350945280 linear 104:2 67109248 > VolGroup00-LogVol01: 0 8388608 linear 104:2 418054528 > VolGroup00-LogVol00: 0 67108864 linear 104:2 384 The IO should pass straight through simple linear targets like that without needing to get broken up, so I wouldn't expect those patches to make any difference in this particular case. Alasdair -- agk@redhat.com ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX @ 2008-01-22 18:51 Martin Knoblauch 0 siblings, 0 replies; 59+ messages in thread From: Martin Knoblauch @ 2008-01-22 18:51 UTC (permalink / raw) To: Alasdair G Kergon Cc: Linus Torvalds, Mel Gorman, Fengguang Wu, Mike Snitzer, Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4, James.Bottomley, Jens Axboe, Milan Broz, Neil Brown ----- Original Message ---- > From: Alasdair G Kergon <agk@redhat.com> > To: Martin Knoblauch <spamtrap@knobisoft.de> > Cc: Linus Torvalds <torvalds@linux-foundation.org>; Mel Gorman <mel@csn.ul.ie>; Fengguang Wu <wfg@mail.ustc.edu.cn>; Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; James.Bottomley@steeleye.com; Jens Axboe <jens.axboe@oracle.com>; Milan Broz <mbroz@redhat.com>; Neil Brown <neilb@suse.de> > Sent: Tuesday, January 22, 2008 3:39:33 PM > Subject: Re: regression: 100% io-wait with 2.6.24-rcX > > > See if these patches make any difference: > http://www.kernel.org/pub/linux/kernel/people/agk/patches/2.6/editing/ > > dm-md-merge_bvec_fn-with-separate-bdev-and-sector.patch > dm-introduce-merge_bvec_fn.patch > dm-linear-add-merge.patch > dm-table-remove-merge_bvec-sector-restriction.patch > nope. Exactely the same poor results. To rule out LVM/DM I really have to see what happens if I setup a system with filesystems directly on partitions. Might take some time though. Cheers Martin ^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: regression: 100% io-wait with 2.6.24-rcX
@ 2008-01-23 11:12 Martin Knoblauch
0 siblings, 0 replies; 59+ messages in thread
From: Martin Knoblauch @ 2008-01-23 11:12 UTC (permalink / raw)
To: Alasdair G Kergon
Cc: Linus Torvalds, Mel Gorman, Fengguang Wu, Mike Snitzer,
Peter Zijlstra, jplatte, Ingo Molnar, linux-kernel, linux-ext4,
James.Bottomley, Jens Axboe, Milan Broz, Neil Brown
----- Original Message ----
> From: Alasdair G Kergon <agk@redhat.com>
> To: Martin Knoblauch <spamtrap@knobisoft.de>
> Cc: Linus Torvalds <torvalds@linux-foundation.org>; Mel Gorman <mel@csn.ul.ie>; Fengguang Wu <wfg@mail.ustc.edu.cn>; Mike Snitzer <snitzer@gmail.com>; Peter Zijlstra <peterz@infradead.org>; jplatte@naasa.net; Ingo Molnar <mingo@elte.hu>; linux-kernel@vger.kernel.org; "linux-ext4@vger.kernel.org" <linux-ext4@vger.kernel.org>; James.Bottomley@steeleye.com; Jens Axboe <jens.axboe@oracle.com>; Milan Broz <mbroz@redhat.com>; Neil Brown <neilb@suse.de>
> Sent: Wednesday, January 23, 2008 12:40:52 AM
> Subject: Re: regression: 100% io-wait with 2.6.24-rcX
>
> On Tue, Jan 22, 2008 at 07:25:15AM -0800, Martin Knoblauch wrote:
> > [root@lpsdm52 ~]# dmsetup table
> > VolGroup00-LogVol02: 0 350945280 linear 104:2 67109248
> > VolGroup00-LogVol01: 0 8388608 linear 104:2 418054528
> > VolGroup00-LogVol00: 0 67108864 linear 104:2 384
>
> The IO should pass straight through simple linear targets like
> that without needing to get broken up, so I wouldn't expect those patches to
> make any difference in this particular case.
>
Alasdair,
LVM/DM are off the hook :-) I converted one box to direct using partitions and the performance is the same disappointment as with LVM/DM. Thanks anyway for looking at my problem.
I will move the discussion now to a new thread, targetting CCISS directly.
Cheers
Martin
^ permalink raw reply [flat|nested] 59+ messages in thread
end of thread, other threads:[~2008-01-23 11:12 UTC | newest] Thread overview: 59+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2008-01-07 10:51 regression: 100% io-wait with 2.6.24-rcX Joerg Platte 2008-01-07 11:19 ` Ingo Molnar 2008-01-07 13:24 ` Joerg Platte 2008-01-07 13:32 ` Peter Zijlstra 2008-01-07 13:40 ` Joerg Platte [not found] ` <E1JCRbA-0002bh-3c@localhost.localdomain> 2008-01-09 3:27 ` Fengguang Wu 2008-01-09 6:13 ` Joerg Platte [not found] ` <E1JCZg2-0001DE-RP@localhost.localdomain> 2008-01-09 12:04 ` Fengguang Wu 2008-01-09 12:22 ` Joerg Platte [not found] ` <E1JCaUd-0001Ko-Tt@localhost.localdomain> 2008-01-09 12:57 ` Fengguang Wu 2008-01-09 13:04 ` Joerg Platte [not found] ` <E1JCrMj-0001HR-SZ@localhost.localdomain> 2008-01-10 6:58 ` Fengguang Wu [not found] ` <E1JCrsE-0000v4-Dz@localhost.localdomain> 2008-01-10 7:30 ` Fengguang Wu [not found] ` <20080110073046.GA3432@mail.ustc.edu.cn> [not found] ` <E1JCsDr-0002cl-0e@localhost.localdomain> 2008-01-10 7:53 ` Fengguang Wu 2008-01-10 8:37 ` Joerg Platte [not found] ` <E1JCt0n-00048n-AD@localhost.localdomain> 2008-01-10 8:43 ` Fengguang Wu 2008-01-10 10:03 ` Joerg Platte [not found] ` <E1JDBk4-0000UF-03@localhost.localdomain> 2008-01-11 4:43 ` Fengguang Wu 2008-01-11 5:29 ` Joerg Platte 2008-01-11 6:41 ` Joerg Platte 2008-01-12 23:32 ` Joerg Platte [not found] ` <E1JDwaA-00017Q-W6@localhost.localdomain> 2008-01-13 6:44 ` Fengguang Wu 2008-01-13 8:05 ` Joerg Platte [not found] ` <E1JDy5a-0001al-Tk@localhost.localdomain> 2008-01-13 8:21 ` Fengguang Wu 2008-01-13 9:49 ` Joerg Platte [not found] ` <E1JE1Uz-0002w5-6z@localhost.localdomain> 2008-01-13 11:59 ` Fengguang Wu [not found] ` <20080113115933.GA11045@mail.ustc.edu.cn> [not found] ` <E1JEGPH-0001uw-Df@localhost.localdomain> 2008-01-14 3:54 ` Fengguang Wu [not found] ` <20080114035439.GA7330@mail.ustc.edu.cn> [not found] ` <E1JEM2I-00010S-5U@localhost.localdomain> 2008-01-14 9:55 ` Fengguang Wu 2008-01-14 11:30 ` Joerg Platte 2008-01-14 11:41 ` Peter Zijlstra [not found] ` <E1JEOmD-0001Ap-U7@localhost.localdomain> 2008-01-14 12:50 ` Fengguang Wu 2008-01-15 21:13 ` Mike Snitzer [not found] ` <E1JF0m1-000101-OK@localhost.localdomain> 2008-01-16 5:25 ` Fengguang Wu 2008-01-15 21:42 ` Ingo Molnar [not found] ` <E1JF0bJ-0000zU-FG@localhost.localdomain> 2008-01-16 5:14 ` Fengguang Wu 2008-01-16 9:26 Martin Knoblauch [not found] ` <E1JF6w8-0000vs-HM@localhost.localdomain> 2008-01-16 12:00 ` Fengguang Wu 2008-01-16 14:15 Martin Knoblauch 2008-01-16 16:27 ` Mike Snitzer 2008-01-17 13:52 Martin Knoblauch 2008-01-17 16:11 ` Mike Snitzer 2008-01-17 17:44 Martin Knoblauch 2008-01-17 20:23 ` Mel Gorman 2008-01-17 17:51 Martin Knoblauch 2008-01-17 21:50 Martin Knoblauch 2008-01-17 22:12 ` Mel Gorman 2008-01-18 8:19 Martin Knoblauch 2008-01-18 16:01 ` Mel Gorman 2008-01-18 17:46 ` Linus Torvalds 2008-01-18 19:01 ` Martin Knoblauch 2008-01-18 19:23 ` Linus Torvalds 2008-01-22 14:39 ` Alasdair G Kergon 2008-01-18 20:00 ` Mike Snitzer 2008-01-18 22:47 ` Mike Snitzer 2008-01-19 10:24 Martin Knoblauch 2008-01-22 15:25 Martin Knoblauch 2008-01-22 23:40 ` Alasdair G Kergon 2008-01-22 18:51 Martin Knoblauch 2008-01-23 11:12 Martin Knoblauch
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).