From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jeremy Fitzhardinge Subject: Re: Crash on blktap shutdown Date: Wed, 24 Feb 2010 15:26:05 -0800 Message-ID: <4B85B58D.20204@goop.org> References: <4B85AE5C.8050603@goop.org> <1267053644.5962.409.camel@agari.van.xensource.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Return-path: In-Reply-To: <1267053644.5962.409.camel@agari.van.xensource.com> List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xensource.com Errors-To: xen-devel-bounces@lists.xensource.com To: Daniel Stodden Cc: Xen-devel , Jake Wires List-Id: xen-devel@lists.xenproject.org On 02/24/2010 03:20 PM, Daniel Stodden wrote: > Jake, any immediate ideas? > Just got another one on domain shutdown. The crashing instruction is: 0xffffffff8104a3f2 : mov 0x28(%r12),%r14 r12 = 6b6b6b6b6b6b6c8b 0x6b is the use-after-free poison value. So I think a use-after-free. 0xffffffff8104a3f2 is in lock_timer_base (/home/jeremy/git/linux/kernel/timer.c:620). 615 __acquires(timer->base->lock) 616 { 617 struct tvec_base *base; 618 619 for (;;) { 620 struct tvec_base *prelock_base = timer->base; 621 base = tbase_get_base(prelock_base); 622 if (likely(base != NULL)) { 623 spin_lock_irqsave(&base->lock, *flags); 624 if (likely(prelock_base == timer->base)) general protection fault: 0000 [#1] SMP DEBUG_PAGEALLOC last sysfs file: /sys/devices/virtual/blktap2/blktap0/remove CPU 1 Modules linked in: e1000 evdev ahci dm_mod sd_mod mptspi mptscsih mptbase scsi_] Pid: 6533, comm: xend Not tainted 2.6.32.9 #356 PowerEdge 1850 RIP: e030:[] [] lock_timer_base+0x11/0x4d RSP: e02b:ffff880021a73ce8 EFLAGS: 00010286 RAX: ffff88001d858f40 RBX: 6b6b6b6b6b6b6c8b RCX: 0000000000000000 RDX: ffffffff8104abda RSI: ffff880021a73d20 RDI: 6b6b6b6b6b6b6c8b RBP: ffff880021a73d08 R08: 0000000000000000 R09: 0000000000000001 R10: ffffffff8104abda R11: ffff880003cd1810 R12: 6b6b6b6b6b6b6c8b R13: ffff880021a73d20 R14: 000000000000011e R15: ffff880021a73e20 FS: 00007f164dffb910(0000) GS:ffff8800028fb000(0000) knlGS:0000000000000000 CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000001d62140 CR3: 000000002eac8000 CR4: 0000000000000660 DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 Process xend (pid: 6533, threadinfo ffff880021a72000, task ffff88001d858f40) Stack: 6b6b6b6b6b6b6c8b 00000000ffffffff ffff88002f2802e8 000000000000011e <0> ffff880021a73d38 ffffffff8104a7b5 0000000000000001 ffffffff8104abda <0> 6b6b6b6b6b6b6c8b 6b6b6b6b6b6b6cbb ffff880021a73d78 ffffffff8104ac68 Call Trace: [] try_to_del_timer_sync+0x1b/0x81 [] ? del_timer_sync+0x0/0xa1 [] del_timer_sync+0x8e/0xa1 [] ? del_timer_sync+0x0/0xa1 [] ? kobject_release+0x0/0x66 [] blk_sync_queue+0x18/0x34 [] blk_cleanup_queue+0xf/0x4b [] blktap_device_destroy+0xad/0xd7 [] blktap_control_destroy_device+0x55/0x154 [] ? mutex_lock_nested+0x2a5/0x2b4 [] blktap_sysfs_remove_device+0x39/0x49 [] dev_attr_store+0x1b/0x1d [] sysfs_write_file+0xf6/0x132 [] vfs_write+0xad/0x14e [] ? fget_light+0x52/0xeb [] ? __up_read+0x1c/0xa2 [] sys_write+0x45/0x6c [] system_call_fastpath+0x16/0x1b Code: 55 31 d2 48 89 e5 31 f6 65 48 8b 3c 25 c0 cb 00 00 e8 95 77 00 00 c9 48 9 RIP [] lock_timer_base+0x11/0x4d RSP ---[ end trace 767ddf28dd1b4a3e ]--- > Daniel > > On Wed, 2010-02-24 at 17:55 -0500, Jeremy Fitzhardinge wrote: > >> When rebooting the machine, I got this crash from blktap. The rip maps to line 262 in >> 0xffffffff812548a1 is in blktap_request_pool_free (/home/jeremy/git/linux/drivers/xen/blktap/request.c:262). >> 257 spin_lock_irqsave(&pool.lock, flags); >> 258 >> 259 pool.status = BLKTAP_POOL_CLOSING; >> 260 while (atomic_read(&pool.reqs_in_use)) { >> 261 spin_unlock_irqrestore(&pool.lock, flags); >> 262 wait_event(pool.wait_queue, !atomic_read(&pool.reqs_in_use)); >> 263 spin_lock_irqsave(&pool.lock, flags); >> 264 } >> 265 >> 266 for (i = 0; i< MAX_BUCKETS; i++) { >> >> >> blktap_ring_vm_close: unmapping ring 6 >> blktap_ring_release: freeing device 6 >> general protection fault: 0000 [#2] SMP DEBUG_PAGEALLOC >> last sysfs file: /sys/devices/virtual/net/eth0/address >> CPU 1 >> Modules linked in: e1000 evdev ahci dm_mod sd_mod mptspi mptscsih mptbase scsi_] >> Pid: 993, comm: tapdisk2 Tainted: G D 2.6.32.8 #355 PowerEdge 1850 >> RIP: e030:[] [] blktap_device_restart+0x7a8 >> RSP: e02b:ffff88002d767be8 EFLAGS: 00010092 >> RAX: ffff88002ea06b08 RBX: ffff88002f319090 RCX: 0000000000000000 >> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6b6b >> RBP: ffff88002d767bf8 R08: 0000000000000002 R09: 0000000000000001 >> R10: ffffffff8125412d R11: ffffffff811eaa4a R12: ffff88002f319330 >> R13: ffff88002f3191b8 R14: ffff8800242a3a50 R15: 0000000000000001 >> FS: 00007f7e3234a740(0000) GS:ffff8800028fb000(0000) knlGS:0000000000000000 >> CS: e033 DS: 0000 ES: 0000 CR0: 000000008005003b >> CR2: 00000036a05a8d84 CR3: 000000002d364000 CR4: 0000000000000660 >> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 >> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400 >> Process tapdisk2 (pid: 993, threadinfo ffff88002d766000, task ffff8800242c3d00) >> Stack: >> ffff88002f319090 ffff88002f319238 ffff88002d767c28 ffffffff81251b3b >> <0> ffff8800242a3a50 ffff88002f2c2870 ffff880002909820 ffff88002400ad60 >> <0> ffff88002d767c48 ffffffff8109aead ffff8800242a3a50 ffff88002400ad00 >> Call Trace: >> [] blktap_ring_vm_close+0x39/0x12d >> [] remove_vma+0x3b/0x71 >> [] exit_mmap+0x153/0x175 >> [] mmput+0x3e/0xd9 >> [] exit_mm+0x100/0x10b >> [] do_exit+0x1b9/0x638 >> [] ? get_signal_to_deliver+0x2dd/0x36e >> [] ? xen_restore_fl_direct_end+0x0/0x1 >> [] do_group_exit+0x73/0x9c >> [] get_signal_to_deliver+0x34f/0x36e >> [] do_signal+0x6d/0x6b0 >> [] ? sys_getsid+0x88/0xaf >> [] ? poll_select_copy_remaining+0xc9/0xee >> [] do_notify_resume+0x27/0x47 >> [] ? trace_hardirqs_on_thunk+0x3a/0x3f >> [] ? remove_wait_queue+0x12/0x45 >> [] int_signal+0x12/0x17 >> Code: a8 01 74 0a 48 89 df e8 24 e6 ff ff eb 46 4c 8d a3 a0 02 00 00 4c 89 e7 e >> RIP [] blktap_device_restart+0x7a/0xa8 >> RSP >> ---[ end trace 1b88501e9b8effb5 ]--- >> >> J >> >> > >