* fs: use after free in /proc/pid/mountinfo @ 2014-07-02 11:55 Sasha Levin 2014-07-03 3:15 ` Sasha Levin 0 siblings, 1 reply; 19+ messages in thread From: Sasha Levin @ 2014-07-02 11:55 UTC (permalink / raw) To: linux-fsdevel; +Cc: Al Viro, Jan Kara, Dave Jones, LKML Hi all, While fuzzing with trinity inside a KVM tools guest running the latest -next kernel I've stumbled on the following spew: [ 3569.869749] general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 3569.869769] Dumping ftrace buffer: [ 3569.869879] (ftrace buffer empty) [ 3569.869894] Modules linked in: [ 3569.869900] CPU: 7 PID: 10239 Comm: trinity-c86 Tainted: G W 3.16.0-rc3-next-20140701-sasha-00023-g4eb2544-dirty #759 [ 3569.869906] task: ffff88039e873000 ti: ffff880393f8c000 task.ti: ffff880393f8c000 [ 3569.869932] RIP: show_mountinfo (fs/proc_namespace.c:127) [ 3569.869938] RSP: 0018:ffff880393f8fd08 EFLAGS: 00010282 [ 3569.869943] RAX: ffff880200c6d208 RBX: 6b6b6b6b6b6b6b03 RCX: 0000000000000074 [ 3569.869946] RDX: 0000000000000000 RSI: 6b6b6b6b6b6b6b03 RDI: ffff880200c6d208 [ 3569.869953] RBP: ffff880393f8fd58 R08: 0000000000000001 R09: ffff88039e873d38 [ 3569.869957] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880200c6d208 [ 3569.869962] R13: 6b6b6b6b6b6b6b6b R14: ffff880393f8ff50 R15: ffff880200c6d208 [ 3569.869966] FS: 00007f6f20db9700(0000) GS:ffff8801ece00000(0000) knlGS:0000000000000000 [ 3569.869969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 3569.869974] CR2: 00007f6f20bee000 CR3: 00000003a0775000 CR4: 00000000000006a0 [ 3569.869992] Stack: [ 3569.870011] ffff880393f8ff50 ffff880200c6d208 ffff880393f8fd48 ffffffff984e10ea [ 3569.870025] ffffffff9532c7fe ffff8802069a3b80 0000000000000065 6b6b6b6b6b6b6b6b [ 3569.870036] ffff880393f8ff50 ffff880200c6d208 ffff880393f8fd68 ffffffff9532c756 [ 3569.870038] Call Trace: [ 3569.870051] ? down_read (kernel/locking/rwsem.c:45 (discriminator 2)) [ 3569.870059] ? m_start (kernel/module.c:3634) [ 3569.870069] m_show (kernel/module.c:3649) [ 3569.870076] seq_read (fs/seq_file.c:235) [ 3569.870081] ? single_open_size (fs/seq_file.c:166) [ 3569.870092] do_loop_readv_writev (fs/read_write.c:708) [ 3569.870098] ? single_open_size (fs/seq_file.c:166) [ 3569.870098] do_readv_writev (fs/read_write.c:840) [ 3569.870098] ? get_parent_ip (kernel/sched/core.c:2550) [ 3569.870098] ? mutex_lock_nested (./arch/x86/include/asm/preempt.h:98 kernel/locking/mutex.c:570 kernel/locking/mutex.c:587) [ 3569.870098] ? __fdget_pos (fs/file.c:714) [ 3569.870098] ? __fdget_pos (fs/file.c:714) [ 3569.870098] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2557 kernel/locking/lockdep.c:2599) [ 3569.870098] ? __fget_light (fs/file.c:685) [ 3569.870098] vfs_readv (fs/read_write.c:866) [ 3569.870098] SyS_readv (fs/read_write.c:892 fs/read_write.c:884) [ 3569.870098] tracesys (arch/x86/kernel/entry_64.S:542) [ 3569.870098] Code: 00 00 00 00 66 66 66 66 90 55 48 89 e5 48 83 ec 50 48 89 5d d8 48 89 f3 4c 89 65 e0 49 89 fc 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 <48> 8b 06 48 89 75 b0 4c 8b 76 08 8b 96 ec 00 00 00 48 89 45 b8 All code ======== 0: 00 00 add %al,(%rax) 2: 00 00 add %al,(%rax) 4: 66 66 66 66 90 data32 data32 data32 xchg %ax,%ax 9: 55 push %rbp a: 48 89 e5 mov %rsp,%rbp d: 48 83 ec 50 sub $0x50,%rsp 11: 48 89 5d d8 mov %rbx,-0x28(%rbp) 15: 48 89 f3 mov %rsi,%rbx 18: 4c 89 65 e0 mov %r12,-0x20(%rbp) 1c: 49 89 fc mov %rdi,%r12 1f: 4c 89 6d e8 mov %r13,-0x18(%rbp) 23: 4c 89 75 f0 mov %r14,-0x10(%rbp) 27: 4c 89 7d f8 mov %r15,-0x8(%rbp) 2b:* 48 8b 06 mov (%rsi),%rax <-- trapping instruction 2e: 48 89 75 b0 mov %rsi,-0x50(%rbp) 32: 4c 8b 76 08 mov 0x8(%rsi),%r14 36: 8b 96 ec 00 00 00 mov 0xec(%rsi),%edx 3c: 48 89 45 b8 mov %rax,-0x48(%rbp) ... Code starting with the faulting instruction =========================================== 0: 48 8b 06 mov (%rsi),%rax 3: 48 89 75 b0 mov %rsi,-0x50(%rbp) 7: 4c 8b 76 08 mov 0x8(%rsi),%r14 b: 8b 96 ec 00 00 00 mov 0xec(%rsi),%edx 11: 48 89 45 b8 mov %rax,-0x48(%rbp) ... [ 3569.870098] RIP show_mountinfo (fs/proc_namespace.c:127) [ 3569.870098] RSP <ffff880393f8fd08> Thanks, Sasha ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-02 11:55 fs: use after free in /proc/pid/mountinfo Sasha Levin @ 2014-07-03 3:15 ` Sasha Levin 2014-07-03 21:37 ` David Rientjes 0 siblings, 1 reply; 19+ messages in thread From: Sasha Levin @ 2014-07-03 3:15 UTC (permalink / raw) To: linux-fsdevel; +Cc: Al Viro, Jan Kara, Dave Jones, LKML On 07/02/2014 07:55 AM, Sasha Levin wrote: > Hi all, > > While fuzzing with trinity inside a KVM tools guest running the latest -next > kernel I've stumbled on the following spew: > > [ 3569.869749] general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > [ 3569.869769] Dumping ftrace buffer: > [ 3569.869879] (ftrace buffer empty) > [ 3569.869894] Modules linked in: > [ 3569.869900] CPU: 7 PID: 10239 Comm: trinity-c86 Tainted: G W 3.16.0-rc3-next-20140701-sasha-00023-g4eb2544-dirty #759 > [ 3569.869906] task: ffff88039e873000 ti: ffff880393f8c000 task.ti: ffff880393f8c000 > [ 3569.869932] RIP: show_mountinfo (fs/proc_namespace.c:127) > [ 3569.869938] RSP: 0018:ffff880393f8fd08 EFLAGS: 00010282 > [ 3569.869943] RAX: ffff880200c6d208 RBX: 6b6b6b6b6b6b6b03 RCX: 0000000000000074 > [ 3569.869946] RDX: 0000000000000000 RSI: 6b6b6b6b6b6b6b03 RDI: ffff880200c6d208 > [ 3569.869953] RBP: ffff880393f8fd58 R08: 0000000000000001 R09: ffff88039e873d38 > [ 3569.869957] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880200c6d208 > [ 3569.869962] R13: 6b6b6b6b6b6b6b6b R14: ffff880393f8ff50 R15: ffff880200c6d208 > [ 3569.869966] FS: 00007f6f20db9700(0000) GS:ffff8801ece00000(0000) knlGS:0000000000000000 > [ 3569.869969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > [ 3569.869974] CR2: 00007f6f20bee000 CR3: 00000003a0775000 CR4: 00000000000006a0 > [ 3569.869992] Stack: > [ 3569.870011] ffff880393f8ff50 ffff880200c6d208 ffff880393f8fd48 ffffffff984e10ea > [ 3569.870025] ffffffff9532c7fe ffff8802069a3b80 0000000000000065 6b6b6b6b6b6b6b6b > [ 3569.870036] ffff880393f8ff50 ffff880200c6d208 ffff880393f8fd68 ffffffff9532c756 > [ 3569.870038] Call Trace: > [ 3569.870051] ? down_read (kernel/locking/rwsem.c:45 (discriminator 2)) > [ 3569.870059] ? m_start (kernel/module.c:3634) > [ 3569.870069] m_show (kernel/module.c:3649) > [ 3569.870076] seq_read (fs/seq_file.c:235) > [ 3569.870081] ? single_open_size (fs/seq_file.c:166) > [ 3569.870092] do_loop_readv_writev (fs/read_write.c:708) > [ 3569.870098] ? single_open_size (fs/seq_file.c:166) > [ 3569.870098] do_readv_writev (fs/read_write.c:840) > [ 3569.870098] ? get_parent_ip (kernel/sched/core.c:2550) > [ 3569.870098] ? mutex_lock_nested (./arch/x86/include/asm/preempt.h:98 kernel/locking/mutex.c:570 kernel/locking/mutex.c:587) > [ 3569.870098] ? __fdget_pos (fs/file.c:714) > [ 3569.870098] ? __fdget_pos (fs/file.c:714) > [ 3569.870098] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2557 kernel/locking/lockdep.c:2599) > [ 3569.870098] ? __fget_light (fs/file.c:685) > [ 3569.870098] vfs_readv (fs/read_write.c:866) > [ 3569.870098] SyS_readv (fs/read_write.c:892 fs/read_write.c:884) > [ 3569.870098] tracesys (arch/x86/kernel/entry_64.S:542) > [ 3569.870098] Code: 00 00 00 00 66 66 66 66 90 55 48 89 e5 48 83 ec 50 48 89 5d d8 48 89 f3 4c 89 65 e0 49 89 fc 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 <48> 8b 06 48 89 75 b0 4c 8b 76 08 8b 96 ec 00 00 00 48 89 45 b8 > All code > ======== > 0: 00 00 add %al,(%rax) > 2: 00 00 add %al,(%rax) > 4: 66 66 66 66 90 data32 data32 data32 xchg %ax,%ax > 9: 55 push %rbp > a: 48 89 e5 mov %rsp,%rbp > d: 48 83 ec 50 sub $0x50,%rsp > 11: 48 89 5d d8 mov %rbx,-0x28(%rbp) > 15: 48 89 f3 mov %rsi,%rbx > 18: 4c 89 65 e0 mov %r12,-0x20(%rbp) > 1c: 49 89 fc mov %rdi,%r12 > 1f: 4c 89 6d e8 mov %r13,-0x18(%rbp) > 23: 4c 89 75 f0 mov %r14,-0x10(%rbp) > 27: 4c 89 7d f8 mov %r15,-0x8(%rbp) > 2b:* 48 8b 06 mov (%rsi),%rax <-- trapping instruction > 2e: 48 89 75 b0 mov %rsi,-0x50(%rbp) > 32: 4c 8b 76 08 mov 0x8(%rsi),%r14 > 36: 8b 96 ec 00 00 00 mov 0xec(%rsi),%edx > 3c: 48 89 45 b8 mov %rax,-0x48(%rbp) > ... > > Code starting with the faulting instruction > =========================================== > 0: 48 8b 06 mov (%rsi),%rax > 3: 48 89 75 b0 mov %rsi,-0x50(%rbp) > 7: 4c 8b 76 08 mov 0x8(%rsi),%r14 > b: 8b 96 ec 00 00 00 mov 0xec(%rsi),%edx > 11: 48 89 45 b8 mov %rax,-0x48(%rbp) > ... > [ 3569.870098] RIP show_mountinfo (fs/proc_namespace.c:127) > [ 3569.870098] RSP <ffff880393f8fd08> > > > Thanks, > Sasha > This one seems related: [ 4292.460172] kernel tried to execute NX-protected page - exploit attempt? (uid: 84) [ 4292.462109] BUG: unable to handle kernel paging request at ffff8804650c3ec8 [ 4292.464167] IP: 0xffff8804650c3ec8 (??:?) [ 4292.465775] PGD 24b7d067 PUD 703f42067 PMD 703e19067 PTE 80000004650c3163 [ 4292.467882] Oops: 0011 [#1] PREEMPT SMP DEBUG_PAGEALLOC [ 4292.469606] Dumping ftrace buffer: [ 4292.470032] (ftrace buffer empty) [ 4292.470032] Modules linked in: [ 4292.470032] CPU: 9 PID: 16397 Comm: trinity-c103 Tainted: G W 3.16.0-rc3-next-20140702-sasha-00023-ga35ffce-dirty #760 [ 4292.470032] task: ffff8801b479b000 ti: ffff8804650c0000 task.ti: ffff8804650c0000 [ 4292.470032] RIP: 0xffff8804650c3ec8 (??:?) [ 4292.470032] RSP: 0000:ffff8804650c3e60 EFLAGS: 00010293 [ 4292.470032] RAX: ffff8803d5e586a0 RBX: ffff88007bf85b00 RCX: ffff8800a5ab69a8 [ 4292.470032] RDX: 0000000000000000 RSI: ffffffffa31352f8 RDI: ffff8803d5e5b6a8 [ 4292.470032] RBP: ffff8804650c3e58 R08: ffff8803d5e5b6a8 R09: 0000000000000009 [ 4292.470032] R10: 0000000000000000 R11: 0000000000000000 R12: 0000000000000000 [ 4292.490142] R13: ffff8803d5e5b6a8 R14: ffff8804650c3f50 R15: ffff88008082e900 [ 4292.490142] FS: 00007ff380e2e700(0000) GS:ffff88025ce00000(0000) knlGS:0000000000000000 [ 4292.490142] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4292.490142] CR2: ffff8804650c3ec8 CR3: 000000017f20e000 CR4: 00000000000006a0 [ 4292.490142] Stack: [ 4292.490142] ffffffff9d332972 0000000000000009 0000000081000000 ffff88008082e940 [ 4292.490142] 0000000000000e5f 0000000000000000 0000000000000001 ffff8804650c3ec8 [ 4292.490142] ffff8803d69b1ff0 ffff88007bf85b00 0000000081000000 ffff8804650c3f50 [ 4292.490142] Call Trace: [ 4292.490142] seq_read (fs/seq_file.c:265) [ 4292.490142] ? proc_reg_read (fs/proc/inode.c:211) [ 4292.508936] ? vfs_read (fs/read_write.c:430) [ 4292.508936] ? SyS_read (fs/read_write.c:568 fs/read_write.c:560) [ 4292.508936] ? ia32_do_call (arch/x86/ia32/ia32entry.S:430) [ 4292.508936] Code: 88 ff ff f0 1f 9b d6 03 88 ff ff 00 5b f8 7b 00 88 ff ff 00 00 00 81 00 00 00 00 50 3f 0c 65 04 88 ff ff 03 00 00 00 00 00 00 00 <f8> 3e 0c 65 04 88 ff ff 42 ae 37 9d ff ff ff ff 08 3f 0c 65 04 All code ======== 0: 88 ff mov %bh,%bh 2: ff f0 push %rax 4: 1f (bad) 5: 9b fwait 6: d6 (bad) 7: 03 88 ff ff 00 5b add 0x5b00ffff(%rax),%ecx d: f8 clc e: 7b 00 jnp 0x10 10: 88 ff mov %bh,%bh 12: ff 00 incl (%rax) 14: 00 00 add %al,(%rax) 16: 81 00 00 00 00 50 addl $0x50000000,(%rax) 1c: 3f (bad) 1d: 0c 65 or $0x65,%al 1f: 04 88 add $0x88,%al 21: ff (bad) 22: ff 03 incl (%rbx) 24: 00 00 add %al,(%rax) 26: 00 00 add %al,(%rax) 28: 00 00 add %al,(%rax) 2a:* 00 f8 add %bh,%al <-- trapping instruction 2c: 3e ds 2d: 0c 65 or $0x65,%al 2f: 04 88 add $0x88,%al 31: ff (bad) 32: ff 42 ae incl -0x52(%rdx) 35: 37 (bad) 36: 9d popfq 37: ff (bad) 38: ff (bad) 39: ff (bad) 3a: ff 08 decl (%rax) 3c: 3f (bad) 3d: 0c 65 or $0x65,%al 3f: 04 00 add $0x0,%al Code starting with the faulting instruction =========================================== 0: f8 clc 1: 3e ds 2: 0c 65 or $0x65,%al 4: 04 88 add $0x88,%al 6: ff (bad) 7: ff 42 ae incl -0x52(%rdx) a: 37 (bad) b: 9d popfq c: ff (bad) d: ff (bad) e: ff (bad) f: ff 08 decl (%rax) 11: 3f (bad) 12: 0c 65 or $0x65,%al 14: 04 00 add $0x0,%al [ 4292.508936] RIP 0xffff8804650c3ec8 (??:?) [ 4292.508936] RSP <ffff8804650c3e60> [ 4292.508936] CR2: ffff8804650c3ec8 Thanks, Sasha ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-03 3:15 ` Sasha Levin @ 2014-07-03 21:37 ` David Rientjes 2014-07-04 14:55 ` Sasha Levin 0 siblings, 1 reply; 19+ messages in thread From: David Rientjes @ 2014-07-03 21:37 UTC (permalink / raw) To: Sasha Levin Cc: linux-fsdevel, Al Viro, Jan Kara, Dave Jones, LKML, Heiko Carstens On Wed, 2 Jul 2014, Sasha Levin wrote: > > Hi all, > > > > While fuzzing with trinity inside a KVM tools guest running the latest -next > > kernel I've stumbled on the following spew: > > > > [ 3569.869749] general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > > [ 3569.869769] Dumping ftrace buffer: > > [ 3569.869879] (ftrace buffer empty) > > [ 3569.869894] Modules linked in: > > [ 3569.869900] CPU: 7 PID: 10239 Comm: trinity-c86 Tainted: G W 3.16.0-rc3-next-20140701-sasha-00023-g4eb2544-dirty #759 > > [ 3569.869906] task: ffff88039e873000 ti: ffff880393f8c000 task.ti: ffff880393f8c000 > > [ 3569.869932] RIP: show_mountinfo (fs/proc_namespace.c:127) > > [ 3569.869938] RSP: 0018:ffff880393f8fd08 EFLAGS: 00010282 > > [ 3569.869943] RAX: ffff880200c6d208 RBX: 6b6b6b6b6b6b6b03 RCX: 0000000000000074 > > [ 3569.869946] RDX: 0000000000000000 RSI: 6b6b6b6b6b6b6b03 RDI: ffff880200c6d208 > > [ 3569.869953] RBP: ffff880393f8fd58 R08: 0000000000000001 R09: ffff88039e873d38 > > [ 3569.869957] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880200c6d208 > > [ 3569.869962] R13: 6b6b6b6b6b6b6b6b R14: ffff880393f8ff50 R15: ffff880200c6d208 > > [ 3569.869966] FS: 00007f6f20db9700(0000) GS:ffff8801ece00000(0000) knlGS:0000000000000000 > > [ 3569.869969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 > > [ 3569.869974] CR2: 00007f6f20bee000 CR3: 00000003a0775000 CR4: 00000000000006a0 > > [ 3569.869992] Stack: > > [ 3569.870011] ffff880393f8ff50 ffff880200c6d208 ffff880393f8fd48 ffffffff984e10ea > > [ 3569.870025] ffffffff9532c7fe ffff8802069a3b80 0000000000000065 6b6b6b6b6b6b6b6b > > [ 3569.870036] ffff880393f8ff50 ffff880200c6d208 ffff880393f8fd68 ffffffff9532c756 > > [ 3569.870038] Call Trace: > > [ 3569.870051] ? down_read (kernel/locking/rwsem.c:45 (discriminator 2)) > > [ 3569.870059] ? m_start (kernel/module.c:3634) > > [ 3569.870069] m_show (kernel/module.c:3649) > > [ 3569.870076] seq_read (fs/seq_file.c:235) > > [ 3569.870081] ? single_open_size (fs/seq_file.c:166) > > [ 3569.870092] do_loop_readv_writev (fs/read_write.c:708) > > [ 3569.870098] ? single_open_size (fs/seq_file.c:166) > > [ 3569.870098] do_readv_writev (fs/read_write.c:840) > > [ 3569.870098] ? get_parent_ip (kernel/sched/core.c:2550) > > [ 3569.870098] ? mutex_lock_nested (./arch/x86/include/asm/preempt.h:98 kernel/locking/mutex.c:570 kernel/locking/mutex.c:587) > > [ 3569.870098] ? __fdget_pos (fs/file.c:714) > > [ 3569.870098] ? __fdget_pos (fs/file.c:714) > > [ 3569.870098] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2557 kernel/locking/lockdep.c:2599) > > [ 3569.870098] ? __fget_light (fs/file.c:685) > > [ 3569.870098] vfs_readv (fs/read_write.c:866) > > [ 3569.870098] SyS_readv (fs/read_write.c:892 fs/read_write.c:884) > > [ 3569.870098] tracesys (arch/x86/kernel/entry_64.S:542) > > [ 3569.870098] Code: 00 00 00 00 66 66 66 66 90 55 48 89 e5 48 83 ec 50 48 89 5d d8 48 89 f3 4c 89 65 e0 49 89 fc 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 <48> 8b 06 48 89 75 b0 4c 8b 76 08 8b 96 ec 00 00 00 48 89 45 b8 > > All code > > ======== > > 0: 00 00 add %al,(%rax) > > 2: 00 00 add %al,(%rax) > > 4: 66 66 66 66 90 data32 data32 data32 xchg %ax,%ax > > 9: 55 push %rbp > > a: 48 89 e5 mov %rsp,%rbp > > d: 48 83 ec 50 sub $0x50,%rsp > > 11: 48 89 5d d8 mov %rbx,-0x28(%rbp) > > 15: 48 89 f3 mov %rsi,%rbx > > 18: 4c 89 65 e0 mov %r12,-0x20(%rbp) > > 1c: 49 89 fc mov %rdi,%r12 > > 1f: 4c 89 6d e8 mov %r13,-0x18(%rbp) > > 23: 4c 89 75 f0 mov %r14,-0x10(%rbp) > > 27: 4c 89 7d f8 mov %r15,-0x8(%rbp) > > 2b:* 48 8b 06 mov (%rsi),%rax <-- trapping instruction > > 2e: 48 89 75 b0 mov %rsi,-0x50(%rbp) > > 32: 4c 8b 76 08 mov 0x8(%rsi),%r14 > > 36: 8b 96 ec 00 00 00 mov 0xec(%rsi),%edx > > 3c: 48 89 45 b8 mov %rax,-0x48(%rbp) Does this now reproduce on Linus's tree? If so, does reverting commit 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this issue? This is a use-after-free since the poison value is 0x6b and I'm presuming that your /proc/self/mountinfo may be larger than PAGE_SIZE in your testing environment. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-03 21:37 ` David Rientjes @ 2014-07-04 14:55 ` Sasha Levin 2014-07-04 22:35 ` David Rientjes 2014-07-06 10:04 ` fs: use after free in /proc/pid/mountinfo Heiko Carstens 0 siblings, 2 replies; 19+ messages in thread From: Sasha Levin @ 2014-07-04 14:55 UTC (permalink / raw) To: David Rientjes Cc: linux-fsdevel, Al Viro, Jan Kara, Dave Jones, LKML, Heiko Carstens On 07/03/2014 05:37 PM, David Rientjes wrote: > On Wed, 2 Jul 2014, Sasha Levin wrote: > >>> Hi all, >>> >>> While fuzzing with trinity inside a KVM tools guest running the latest -next >>> kernel I've stumbled on the following spew: >>> >>> [ 3569.869749] general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC >>> [ 3569.869769] Dumping ftrace buffer: >>> [ 3569.869879] (ftrace buffer empty) >>> [ 3569.869894] Modules linked in: >>> [ 3569.869900] CPU: 7 PID: 10239 Comm: trinity-c86 Tainted: G W 3.16.0-rc3-next-20140701-sasha-00023-g4eb2544-dirty #759 >>> [ 3569.869906] task: ffff88039e873000 ti: ffff880393f8c000 task.ti: ffff880393f8c000 >>> [ 3569.869932] RIP: show_mountinfo (fs/proc_namespace.c:127) >>> [ 3569.869938] RSP: 0018:ffff880393f8fd08 EFLAGS: 00010282 >>> [ 3569.869943] RAX: ffff880200c6d208 RBX: 6b6b6b6b6b6b6b03 RCX: 0000000000000074 >>> [ 3569.869946] RDX: 0000000000000000 RSI: 6b6b6b6b6b6b6b03 RDI: ffff880200c6d208 >>> [ 3569.869953] RBP: ffff880393f8fd58 R08: 0000000000000001 R09: ffff88039e873d38 >>> [ 3569.869957] R10: 0000000000000001 R11: 0000000000000000 R12: ffff880200c6d208 >>> [ 3569.869962] R13: 6b6b6b6b6b6b6b6b R14: ffff880393f8ff50 R15: ffff880200c6d208 >>> [ 3569.869966] FS: 00007f6f20db9700(0000) GS:ffff8801ece00000(0000) knlGS:0000000000000000 >>> [ 3569.869969] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 >>> [ 3569.869974] CR2: 00007f6f20bee000 CR3: 00000003a0775000 CR4: 00000000000006a0 >>> [ 3569.869992] Stack: >>> [ 3569.870011] ffff880393f8ff50 ffff880200c6d208 ffff880393f8fd48 ffffffff984e10ea >>> [ 3569.870025] ffffffff9532c7fe ffff8802069a3b80 0000000000000065 6b6b6b6b6b6b6b6b >>> [ 3569.870036] ffff880393f8ff50 ffff880200c6d208 ffff880393f8fd68 ffffffff9532c756 >>> [ 3569.870038] Call Trace: >>> [ 3569.870051] ? down_read (kernel/locking/rwsem.c:45 (discriminator 2)) >>> [ 3569.870059] ? m_start (kernel/module.c:3634) >>> [ 3569.870069] m_show (kernel/module.c:3649) >>> [ 3569.870076] seq_read (fs/seq_file.c:235) >>> [ 3569.870081] ? single_open_size (fs/seq_file.c:166) >>> [ 3569.870092] do_loop_readv_writev (fs/read_write.c:708) >>> [ 3569.870098] ? single_open_size (fs/seq_file.c:166) >>> [ 3569.870098] do_readv_writev (fs/read_write.c:840) >>> [ 3569.870098] ? get_parent_ip (kernel/sched/core.c:2550) >>> [ 3569.870098] ? mutex_lock_nested (./arch/x86/include/asm/preempt.h:98 kernel/locking/mutex.c:570 kernel/locking/mutex.c:587) >>> [ 3569.870098] ? __fdget_pos (fs/file.c:714) >>> [ 3569.870098] ? __fdget_pos (fs/file.c:714) >>> [ 3569.870098] ? trace_hardirqs_on_caller (kernel/locking/lockdep.c:2557 kernel/locking/lockdep.c:2599) >>> [ 3569.870098] ? __fget_light (fs/file.c:685) >>> [ 3569.870098] vfs_readv (fs/read_write.c:866) >>> [ 3569.870098] SyS_readv (fs/read_write.c:892 fs/read_write.c:884) >>> [ 3569.870098] tracesys (arch/x86/kernel/entry_64.S:542) >>> [ 3569.870098] Code: 00 00 00 00 66 66 66 66 90 55 48 89 e5 48 83 ec 50 48 89 5d d8 48 89 f3 4c 89 65 e0 49 89 fc 4c 89 6d e8 4c 89 75 f0 4c 89 7d f8 <48> 8b 06 48 89 75 b0 4c 8b 76 08 8b 96 ec 00 00 00 48 89 45 b8 >>> All code >>> ======== >>> 0: 00 00 add %al,(%rax) >>> 2: 00 00 add %al,(%rax) >>> 4: 66 66 66 66 90 data32 data32 data32 xchg %ax,%ax >>> 9: 55 push %rbp >>> a: 48 89 e5 mov %rsp,%rbp >>> d: 48 83 ec 50 sub $0x50,%rsp >>> 11: 48 89 5d d8 mov %rbx,-0x28(%rbp) >>> 15: 48 89 f3 mov %rsi,%rbx >>> 18: 4c 89 65 e0 mov %r12,-0x20(%rbp) >>> 1c: 49 89 fc mov %rdi,%r12 >>> 1f: 4c 89 6d e8 mov %r13,-0x18(%rbp) >>> 23: 4c 89 75 f0 mov %r14,-0x10(%rbp) >>> 27: 4c 89 7d f8 mov %r15,-0x8(%rbp) >>> 2b:* 48 8b 06 mov (%rsi),%rax <-- trapping instruction >>> 2e: 48 89 75 b0 mov %rsi,-0x50(%rbp) >>> 32: 4c 8b 76 08 mov 0x8(%rsi),%r14 >>> 36: 8b 96 ec 00 00 00 mov 0xec(%rsi),%edx >>> 3c: 48 89 45 b8 mov %rax,-0x48(%rbp) > > Does this now reproduce on Linus's tree? If so, does reverting commit > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this > issue? > > This is a use-after-free since the poison value is 0x6b and I'm presuming > that your /proc/self/mountinfo may be larger than PAGE_SIZE in your > testing environment. > Good call, reverting that patch made both issues go away. Thanks, Sasha ^ permalink raw reply [flat|nested] 19+ messages in thread
* [stable] please do not merge 058504edd026 (was Re: fs: use after free in /proc/pid/mountinfo) 2014-07-04 14:55 ` Sasha Levin @ 2014-07-04 22:35 ` David Rientjes 2014-07-06 10:04 ` fs: use after free in /proc/pid/mountinfo Heiko Carstens 1 sibling, 0 replies; 19+ messages in thread From: David Rientjes @ 2014-07-04 22:35 UTC (permalink / raw) To: Sasha Levin, stable Cc: Al Viro, Jan Kara, Dave Jones, Heiko Carstens, Andrew Morton, Linus Torvalds, linux-kernel, linux-fsdevel On Fri, 4 Jul 2014, Sasha Levin wrote: > > Does this now reproduce on Linus's tree? If so, does reverting commit > > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this > > issue? > > > > This is a use-after-free since the poison value is 0x6b and I'm presuming > > that your /proc/self/mountinfo may be larger than PAGE_SIZE in your > > testing environment. > > > > Good call, reverting that patch made both issues go away. > Thanks for checking, Sasha. Stable maintainers, please do not merge commit 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") that is annotated with a cc for stable@vger.kernel.org in Linus's tree into stable kernels. Although we're at 3.16-rc3, I'm hoping that we can get a fix for the use-after-free in the next couple days before asking for a revert. Sasha confirms[*] this commit causes the bug. Thanks. [*] http://marc.info/?l=linux-kernel&m=140448573612154 ^ permalink raw reply [flat|nested] 19+ messages in thread
* please do not merge 058504edd026 (was Re: fs: use after free in /proc/pid/mountinfo) @ 2014-07-04 22:35 ` David Rientjes 0 siblings, 0 replies; 19+ messages in thread From: David Rientjes @ 2014-07-04 22:35 UTC (permalink / raw) To: Sasha Levin, stable Cc: Al Viro, Jan Kara, Dave Jones, Heiko Carstens, Andrew Morton, Linus Torvalds, linux-kernel, linux-fsdevel On Fri, 4 Jul 2014, Sasha Levin wrote: > > Does this now reproduce on Linus's tree? If so, does reverting commit > > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this > > issue? > > > > This is a use-after-free since the poison value is 0x6b and I'm presuming > > that your /proc/self/mountinfo may be larger than PAGE_SIZE in your > > testing environment. > > > > Good call, reverting that patch made both issues go away. > Thanks for checking, Sasha. Stable maintainers, please do not merge commit 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") that is annotated with a cc for stable@vger.kernel.org in Linus's tree into stable kernels. Although we're at 3.16-rc3, I'm hoping that we can get a fix for the use-after-free in the next couple days before asking for a revert. Sasha confirms[*] this commit causes the bug. Thanks. [*] http://marc.info/?l=linux-kernel&m=140448573612154 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [stable] please do not merge 058504edd026 (was Re: fs: use after free in /proc/pid/mountinfo) 2014-07-04 22:35 ` David Rientjes @ 2014-07-07 23:05 ` Greg KH -1 siblings, 0 replies; 19+ messages in thread From: Greg KH @ 2014-07-07 23:05 UTC (permalink / raw) To: David Rientjes Cc: Sasha Levin, stable, Al Viro, Jan Kara, Dave Jones, Heiko Carstens, Andrew Morton, Linus Torvalds, linux-kernel, linux-fsdevel On Fri, Jul 04, 2014 at 03:35:54PM -0700, David Rientjes wrote: > On Fri, 4 Jul 2014, Sasha Levin wrote: > > > > Does this now reproduce on Linus's tree? If so, does reverting commit > > > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this > > > issue? > > > > > > This is a use-after-free since the poison value is 0x6b and I'm presuming > > > that your /proc/self/mountinfo may be larger than PAGE_SIZE in your > > > testing environment. > > > > > > > Good call, reverting that patch made both issues go away. > > > > Thanks for checking, Sasha. > > Stable maintainers, please do not merge commit 058504edd026 ("fs/seq_file: > fallback to vmalloc allocation") that is annotated with a cc for > stable@vger.kernel.org in Linus's tree into stable kernels. > > Although we're at 3.16-rc3, I'm hoping that we can get a fix for the > use-after-free in the next couple days before asking for a revert. Sasha > confirms[*] this commit causes the bug. I should also drop "Subject: /proc/stat: convert to single_open_size()" from the -stable tree, right? thanks, greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: please do not merge 058504edd026 (was Re: fs: use after free in /proc/pid/mountinfo) @ 2014-07-07 23:05 ` Greg KH 0 siblings, 0 replies; 19+ messages in thread From: Greg KH @ 2014-07-07 23:05 UTC (permalink / raw) To: David Rientjes Cc: Sasha Levin, stable, Al Viro, Jan Kara, Dave Jones, Heiko Carstens, Andrew Morton, Linus Torvalds, linux-kernel, linux-fsdevel On Fri, Jul 04, 2014 at 03:35:54PM -0700, David Rientjes wrote: > On Fri, 4 Jul 2014, Sasha Levin wrote: > > > > Does this now reproduce on Linus's tree? If so, does reverting commit > > > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this > > > issue? > > > > > > This is a use-after-free since the poison value is 0x6b and I'm presuming > > > that your /proc/self/mountinfo may be larger than PAGE_SIZE in your > > > testing environment. > > > > > > > Good call, reverting that patch made both issues go away. > > > > Thanks for checking, Sasha. > > Stable maintainers, please do not merge commit 058504edd026 ("fs/seq_file: > fallback to vmalloc allocation") that is annotated with a cc for > stable@vger.kernel.org in Linus's tree into stable kernels. > > Although we're at 3.16-rc3, I'm hoping that we can get a fix for the > use-after-free in the next couple days before asking for a revert. Sasha > confirms[*] this commit causes the bug. I should also drop "Subject: /proc/stat: convert to single_open_size()" from the -stable tree, right? thanks, greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [stable] please do not merge 058504edd026 (was Re: fs: use after free in /proc/pid/mountinfo) 2014-07-07 23:05 ` Greg KH (?) @ 2014-07-07 23:06 ` Andrew Morton 2014-07-07 23:19 ` Greg KH -1 siblings, 1 reply; 19+ messages in thread From: Andrew Morton @ 2014-07-07 23:06 UTC (permalink / raw) To: Greg KH Cc: David Rientjes, Sasha Levin, stable, Al Viro, Jan Kara, Dave Jones, Heiko Carstens, Linus Torvalds, linux-kernel, linux-fsdevel On Mon, 7 Jul 2014 16:05:42 -0700 Greg KH <greg@kroah.com> wrote: > On Fri, Jul 04, 2014 at 03:35:54PM -0700, David Rientjes wrote: > > On Fri, 4 Jul 2014, Sasha Levin wrote: > > > > > > Does this now reproduce on Linus's tree? If so, does reverting commit > > > > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this > > > > issue? > > > > > > > > This is a use-after-free since the poison value is 0x6b and I'm presuming > > > > that your /proc/self/mountinfo may be larger than PAGE_SIZE in your > > > > testing environment. > > > > > > > > > > Good call, reverting that patch made both issues go away. > > > > > > > Thanks for checking, Sasha. > > > > Stable maintainers, please do not merge commit 058504edd026 ("fs/seq_file: > > fallback to vmalloc allocation") that is annotated with a cc for > > stable@vger.kernel.org in Linus's tree into stable kernels. > > > > Although we're at 3.16-rc3, I'm hoping that we can get a fix for the > > use-after-free in the next couple days before asking for a revert. Sasha > > confirms[*] this commit causes the bug. > > I should also drop "Subject: /proc/stat: convert to single_open_size()" > from the -stable tree, right? > That would be best. I can't see how "fs: use after free in /proc/pid/mountinfo" can cause a use-after-free so perhaps the bug lies elsewhere and was hidden by luck (slab buffering or slab rcu-freeing or something). In which case "fs/seq_file: fallback to vmalloc allocation" might be the patch which added the bug. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [stable] please do not merge 058504edd026 (was Re: fs: use after free in /proc/pid/mountinfo) 2014-07-07 23:06 ` [stable] " Andrew Morton @ 2014-07-07 23:19 ` Greg KH 0 siblings, 0 replies; 19+ messages in thread From: Greg KH @ 2014-07-07 23:19 UTC (permalink / raw) To: Andrew Morton Cc: David Rientjes, Sasha Levin, stable, Al Viro, Jan Kara, Dave Jones, Heiko Carstens, Linus Torvalds, linux-kernel, linux-fsdevel On Mon, Jul 07, 2014 at 04:06:27PM -0700, Andrew Morton wrote: > On Mon, 7 Jul 2014 16:05:42 -0700 Greg KH <greg@kroah.com> wrote: > > > On Fri, Jul 04, 2014 at 03:35:54PM -0700, David Rientjes wrote: > > > On Fri, 4 Jul 2014, Sasha Levin wrote: > > > > > > > > Does this now reproduce on Linus's tree? If so, does reverting commit > > > > > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this > > > > > issue? > > > > > > > > > > This is a use-after-free since the poison value is 0x6b and I'm presuming > > > > > that your /proc/self/mountinfo may be larger than PAGE_SIZE in your > > > > > testing environment. > > > > > > > > > > > > > Good call, reverting that patch made both issues go away. > > > > > > > > > > Thanks for checking, Sasha. > > > > > > Stable maintainers, please do not merge commit 058504edd026 ("fs/seq_file: > > > fallback to vmalloc allocation") that is annotated with a cc for > > > stable@vger.kernel.org in Linus's tree into stable kernels. > > > > > > Although we're at 3.16-rc3, I'm hoping that we can get a fix for the > > > use-after-free in the next couple days before asking for a revert. Sasha > > > confirms[*] this commit causes the bug. > > > > I should also drop "Subject: /proc/stat: convert to single_open_size()" > > from the -stable tree, right? > > > > That would be best. Ok, now dropped, thanks. greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: please do not merge 058504edd026 (was Re: fs: use after free in /proc/pid/mountinfo) @ 2014-07-07 23:19 ` Greg KH 0 siblings, 0 replies; 19+ messages in thread From: Greg KH @ 2014-07-07 23:19 UTC (permalink / raw) To: Andrew Morton Cc: David Rientjes, Sasha Levin, stable, Al Viro, Jan Kara, Dave Jones, Heiko Carstens, Linus Torvalds, linux-kernel, linux-fsdevel On Mon, Jul 07, 2014 at 04:06:27PM -0700, Andrew Morton wrote: > On Mon, 7 Jul 2014 16:05:42 -0700 Greg KH <greg@kroah.com> wrote: > > > On Fri, Jul 04, 2014 at 03:35:54PM -0700, David Rientjes wrote: > > > On Fri, 4 Jul 2014, Sasha Levin wrote: > > > > > > > > Does this now reproduce on Linus's tree? If so, does reverting commit > > > > > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this > > > > > issue? > > > > > > > > > > This is a use-after-free since the poison value is 0x6b and I'm presuming > > > > > that your /proc/self/mountinfo may be larger than PAGE_SIZE in your > > > > > testing environment. > > > > > > > > > > > > > Good call, reverting that patch made both issues go away. > > > > > > > > > > Thanks for checking, Sasha. > > > > > > Stable maintainers, please do not merge commit 058504edd026 ("fs/seq_file: > > > fallback to vmalloc allocation") that is annotated with a cc for > > > stable@vger.kernel.org in Linus's tree into stable kernels. > > > > > > Although we're at 3.16-rc3, I'm hoping that we can get a fix for the > > > use-after-free in the next couple days before asking for a revert. Sasha > > > confirms[*] this commit causes the bug. > > > > I should also drop "Subject: /proc/stat: convert to single_open_size()" > > from the -stable tree, right? > > > > That would be best. Ok, now dropped, thanks. greg k-h ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-04 14:55 ` Sasha Levin 2014-07-04 22:35 ` David Rientjes @ 2014-07-06 10:04 ` Heiko Carstens 2014-07-09 14:24 ` Heiko Carstens 1 sibling, 1 reply; 19+ messages in thread From: Heiko Carstens @ 2014-07-06 10:04 UTC (permalink / raw) To: Sasha Levin Cc: David Rientjes, linux-fsdevel, Al Viro, Jan Kara, Dave Jones, LKML On Fri, Jul 04, 2014 at 10:55:13AM -0400, Sasha Levin wrote: > On 07/03/2014 05:37 PM, David Rientjes wrote: > > On Wed, 2 Jul 2014, Sasha Levin wrote: > > > >>> Hi all, > >>> > >>> While fuzzing with trinity inside a KVM tools guest running the latest -next > >>> kernel I've stumbled on the following spew: > >>> > >>> [ 3569.869749] general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > >>> [ 3569.869769] Dumping ftrace buffer: > >>> [ 3569.869879] (ftrace buffer empty) > >>> [ 3569.869894] Modules linked in: > >>> [ 3569.869900] CPU: 7 PID: 10239 Comm: trinity-c86 Tainted: G W 3.16.0-rc3-next-20140701-sasha-00023-g4eb2544-dirty #759 > >>> [ 3569.869906] task: ffff88039e873000 ti: ffff880393f8c000 task.ti: ffff880393f8c000 > >>> [ 3569.869932] RIP: show_mountinfo (fs/proc_namespace.c:127) So that would be this line then: static int show_mountinfo(struct seq_file *m, struct vfsmount *mnt) { struct proc_mounts *p = proc_mounts(m); struct mount *r = real_mount(mnt); struct super_block *sb = mnt->mnt_sb; struct path mnt_path = { .dentry = mnt->mnt_root, .mnt = mnt }; <--- if I understand the output correctly? > >>> All code > >>> ======== > >>> 0: 00 00 add %al,(%rax) > >>> 2: 00 00 add %al,(%rax) > >>> 4: 66 66 66 66 90 data32 data32 data32 xchg %ax,%ax > >>> 9: 55 push %rbp > >>> a: 48 89 e5 mov %rsp,%rbp > >>> d: 48 83 ec 50 sub $0x50,%rsp > >>> 11: 48 89 5d d8 mov %rbx,-0x28(%rbp) > >>> 15: 48 89 f3 mov %rsi,%rbx > >>> 18: 4c 89 65 e0 mov %r12,-0x20(%rbp) > >>> 1c: 49 89 fc mov %rdi,%r12 > >>> 1f: 4c 89 6d e8 mov %r13,-0x18(%rbp) > >>> 23: 4c 89 75 f0 mov %r14,-0x10(%rbp) > >>> 27: 4c 89 7d f8 mov %r15,-0x8(%rbp) > >>> 2b:* 48 8b 06 mov (%rsi),%rax <-- trapping instruction > >>> 2e: 48 89 75 b0 mov %rsi,-0x50(%rbp) > >>> 32: 4c 8b 76 08 mov 0x8(%rsi),%r14 > >>> 36: 8b 96 ec 00 00 00 mov 0xec(%rsi),%edx > >>> 3c: 48 89 45 b8 mov %rax,-0x48(%rbp) Does that fit to this asm code? Sorry.. I'm not very familar with x86 asm. > > Does this now reproduce on Linus's tree? If so, does reverting commit > > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this > > issue? > > > > This is a use-after-free since the poison value is 0x6b and I'm presuming > > that your /proc/self/mountinfo may be larger than PAGE_SIZE in your > > testing environment. > > Good call, reverting that patch made both issues go away. Ok, does that mean you have a test case so you can reproduce the crashes? Just wondering, since trinity is usually quite random in what it does. I tried to reproduce a crash on one of my systems by enforcing several seq_files to grow beyond PAGE_SIZE, but couldn't reproduce anything yet. Could you try the patch below please? It basically reverts my patch and just leaves the kfree->kvfree conversion in. This is just a shot in the dark, since I can't make any sense of this ...yet :) diff --git a/fs/seq_file.c b/fs/seq_file.c index 3857b720cb1b..c1cf494cc238 100644 --- a/fs/seq_file.c +++ b/fs/seq_file.c @@ -34,12 +34,7 @@ static void seq_set_overflow(struct seq_file *m) static void *seq_buf_alloc(unsigned long size) { - void *buf; - - buf = kmalloc(size, GFP_KERNEL | __GFP_NOWARN); - if (!buf && size > PAGE_SIZE) - buf = vmalloc(size); - return buf; + return kmalloc(size, GFP_KERNEL); } /** ^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-06 10:04 ` fs: use after free in /proc/pid/mountinfo Heiko Carstens @ 2014-07-09 14:24 ` Heiko Carstens 2014-07-09 20:31 ` Andrew Morton 0 siblings, 1 reply; 19+ messages in thread From: Heiko Carstens @ 2014-07-09 14:24 UTC (permalink / raw) To: Sasha Levin, David Rientjes, linux-fsdevel, Al Viro, Jan Kara, Dave Jones, LKML Cc: Andrew Morton, Linus Torvalds, linux-kernel [full quote of my mail below] 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") is suspected to cause a crash. Actually I can't reprocude the crash nor would I be able to tell how the commit could cause the crash. Anyway, I'll be offline for the next 2.5 weeks. So if Sasha could confirm that reverting the patch actually does fix the crash, please revert the commit, unless somebody else can make sense of the report of course. I'm still wondering how Sasha could reproduce the crash. Thanks, Heiko On Sun, Jul 06, 2014 at 12:04:20PM +0200, Heiko Carstens wrote: > On Fri, Jul 04, 2014 at 10:55:13AM -0400, Sasha Levin wrote: > > On 07/03/2014 05:37 PM, David Rientjes wrote: > > > On Wed, 2 Jul 2014, Sasha Levin wrote: > > > > > >>> Hi all, > > >>> > > >>> While fuzzing with trinity inside a KVM tools guest running the latest -next > > >>> kernel I've stumbled on the following spew: > > >>> > > >>> [ 3569.869749] general protection fault: 0000 [#1] PREEMPT SMP DEBUG_PAGEALLOC > > >>> [ 3569.869769] Dumping ftrace buffer: > > >>> [ 3569.869879] (ftrace buffer empty) > > >>> [ 3569.869894] Modules linked in: > > >>> [ 3569.869900] CPU: 7 PID: 10239 Comm: trinity-c86 Tainted: G W 3.16.0-rc3-next-20140701-sasha-00023-g4eb2544-dirty #759 > > >>> [ 3569.869906] task: ffff88039e873000 ti: ffff880393f8c000 task.ti: ffff880393f8c000 > > >>> [ 3569.869932] RIP: show_mountinfo (fs/proc_namespace.c:127) > > So that would be this line then: > > static int show_mountinfo(struct seq_file *m, struct vfsmount *mnt) > { > struct proc_mounts *p = proc_mounts(m); > struct mount *r = real_mount(mnt); > struct super_block *sb = mnt->mnt_sb; > struct path mnt_path = { .dentry = mnt->mnt_root, .mnt = mnt }; <--- > > if I understand the output correctly? > > > >>> All code > > >>> ======== > > >>> 0: 00 00 add %al,(%rax) > > >>> 2: 00 00 add %al,(%rax) > > >>> 4: 66 66 66 66 90 data32 data32 data32 xchg %ax,%ax > > >>> 9: 55 push %rbp > > >>> a: 48 89 e5 mov %rsp,%rbp > > >>> d: 48 83 ec 50 sub $0x50,%rsp > > >>> 11: 48 89 5d d8 mov %rbx,-0x28(%rbp) > > >>> 15: 48 89 f3 mov %rsi,%rbx > > >>> 18: 4c 89 65 e0 mov %r12,-0x20(%rbp) > > >>> 1c: 49 89 fc mov %rdi,%r12 > > >>> 1f: 4c 89 6d e8 mov %r13,-0x18(%rbp) > > >>> 23: 4c 89 75 f0 mov %r14,-0x10(%rbp) > > >>> 27: 4c 89 7d f8 mov %r15,-0x8(%rbp) > > >>> 2b:* 48 8b 06 mov (%rsi),%rax <-- trapping instruction > > >>> 2e: 48 89 75 b0 mov %rsi,-0x50(%rbp) > > >>> 32: 4c 8b 76 08 mov 0x8(%rsi),%r14 > > >>> 36: 8b 96 ec 00 00 00 mov 0xec(%rsi),%edx > > >>> 3c: 48 89 45 b8 mov %rax,-0x48(%rbp) > > Does that fit to this asm code? Sorry.. I'm not very familar with x86 asm. > > > > Does this now reproduce on Linus's tree? If so, does reverting commit > > > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") prevent this > > > issue? > > > > > > This is a use-after-free since the poison value is 0x6b and I'm presuming > > > that your /proc/self/mountinfo may be larger than PAGE_SIZE in your > > > testing environment. > > > > Good call, reverting that patch made both issues go away. > > Ok, does that mean you have a test case so you can reproduce the crashes? > Just wondering, since trinity is usually quite random in what it does. > > I tried to reproduce a crash on one of my systems by enforcing several > seq_files to grow beyond PAGE_SIZE, but couldn't reproduce anything yet. > > Could you try the patch below please? It basically reverts my patch and > just leaves the kfree->kvfree conversion in. This is just a shot in the > dark, since I can't make any sense of this ...yet :) > > diff --git a/fs/seq_file.c b/fs/seq_file.c > index 3857b720cb1b..c1cf494cc238 100644 > --- a/fs/seq_file.c > +++ b/fs/seq_file.c > @@ -34,12 +34,7 @@ static void seq_set_overflow(struct seq_file *m) > > static void *seq_buf_alloc(unsigned long size) > { > - void *buf; > - > - buf = kmalloc(size, GFP_KERNEL | __GFP_NOWARN); > - if (!buf && size > PAGE_SIZE) > - buf = vmalloc(size); > - return buf; > + return kmalloc(size, GFP_KERNEL); > } > > /** > > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-09 14:24 ` Heiko Carstens @ 2014-07-09 20:31 ` Andrew Morton 2014-07-09 22:10 ` Sasha Levin 0 siblings, 1 reply; 19+ messages in thread From: Andrew Morton @ 2014-07-09 20:31 UTC (permalink / raw) To: Heiko Carstens Cc: Sasha Levin, David Rientjes, linux-fsdevel, Al Viro, Jan Kara, Dave Jones, LKML, Linus Torvalds On Wed, 9 Jul 2014 16:24:06 +0200 Heiko Carstens <heiko.carstens@de.ibm.com> wrote: > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") is suspected to > cause a crash. Actually I can't reprocude the crash nor would I be able to > tell how the commit could cause the crash. > Anyway, I'll be offline for the next 2.5 weeks. So if Sasha could confirm > that reverting the patch actually does fix the crash, please revert the > commit, unless somebody else can make sense of the report of course. > > I'm still wondering how Sasha could reproduce the crash. This really sucks :( I suppose I'll queue up a revert of 058504edd026 as a for-3.16 bandaid. Sasha, it would be great if you could delve a bit further into this, see if we can identify a way for others to reproduce the bug. Did you test that little patchlet which Heiko sent? --- a/fs/seq_file.c +++ b/fs/seq_file.c @@ -34,12 +34,7 @@ static void seq_set_overflow(struct seq_file *m) static void *seq_buf_alloc(unsigned long size) { - void *buf; - - buf = kmalloc(size, GFP_KERNEL | __GFP_NOWARN); - if (!buf && size > PAGE_SIZE) - buf = vmalloc(size); - return buf; + return kmalloc(size, GFP_KERNEL); } /** It would also be interesting to try the opposite: always use vmalloc(). That may make it easier to trigger the bug. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-09 20:31 ` Andrew Morton @ 2014-07-09 22:10 ` Sasha Levin 2014-07-09 22:59 ` Andrew Morton 0 siblings, 1 reply; 19+ messages in thread From: Sasha Levin @ 2014-07-09 22:10 UTC (permalink / raw) To: Andrew Morton, Heiko Carstens Cc: David Rientjes, linux-fsdevel, Al Viro, Jan Kara, Dave Jones, LKML, Linus Torvalds On 07/09/2014 04:31 PM, Andrew Morton wrote: > On Wed, 9 Jul 2014 16:24:06 +0200 Heiko Carstens <heiko.carstens@de.ibm.com> wrote: > >> > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") is suspected to >> > cause a crash. Actually I can't reprocude the crash nor would I be able to >> > tell how the commit could cause the crash. >> > Anyway, I'll be offline for the next 2.5 weeks. So if Sasha could confirm >> > that reverting the patch actually does fix the crash, please revert the >> > commit, unless somebody else can make sense of the report of course. >> > >> > I'm still wondering how Sasha could reproduce the crash. > This really sucks :( > > I suppose I'll queue up a revert of 058504edd026 as a for-3.16 bandaid. > > Sasha, it would be great if you could delve a bit further into this, > see if we can identify a way for others to reproduce the bug. > > Did you test that little patchlet which Heiko sent? I've been trying, but as there isn't an easy way to reproduce it, breakage elsewhere in -next (there is a long list) is slowing me down. I'll revert back to the -next where I've initially observed this issue and will retry. Thanks, Sasha ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-09 22:10 ` Sasha Levin @ 2014-07-09 22:59 ` Andrew Morton 2014-07-15 13:52 ` Sasha Levin 0 siblings, 1 reply; 19+ messages in thread From: Andrew Morton @ 2014-07-09 22:59 UTC (permalink / raw) To: Sasha Levin Cc: Heiko Carstens, David Rientjes, linux-fsdevel, Al Viro, Jan Kara, Dave Jones, LKML, Linus Torvalds On Wed, 09 Jul 2014 18:10:47 -0400 Sasha Levin <sasha.levin@oracle.com> wrote: > On 07/09/2014 04:31 PM, Andrew Morton wrote: > > On Wed, 9 Jul 2014 16:24:06 +0200 Heiko Carstens <heiko.carstens@de.ibm.com> wrote: > > > >> > 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") is suspected to > >> > cause a crash. Actually I can't reprocude the crash nor would I be able to > >> > tell how the commit could cause the crash. > >> > Anyway, I'll be offline for the next 2.5 weeks. So if Sasha could confirm > >> > that reverting the patch actually does fix the crash, please revert the > >> > commit, unless somebody else can make sense of the report of course. > >> > > >> > I'm still wondering how Sasha could reproduce the crash. > > This really sucks :( > > > > I suppose I'll queue up a revert of 058504edd026 as a for-3.16 bandaid. > > > > Sasha, it would be great if you could delve a bit further into this, > > see if we can identify a way for others to reproduce the bug. > > > > Did you test that little patchlet which Heiko sent? > > I've been trying, but as there isn't an easy way to reproduce it, breakage > elsewhere in -next (there is a long list) is slowing me down. I'll revert > back to the -next where I've initially observed this issue and will retry. > Thanks. The bug should be present in current mainline. ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-09 22:59 ` Andrew Morton @ 2014-07-15 13:52 ` Sasha Levin 2014-07-15 23:01 ` David Rientjes 0 siblings, 1 reply; 19+ messages in thread From: Sasha Levin @ 2014-07-15 13:52 UTC (permalink / raw) To: Andrew Morton Cc: Heiko Carstens, David Rientjes, linux-fsdevel, Al Viro, Jan Kara, Dave Jones, LKML, Linus Torvalds On 07/09/2014 06:59 PM, Andrew Morton wrote: > On Wed, 09 Jul 2014 18:10:47 -0400 Sasha Levin <sasha.levin@oracle.com> wrote: > >> On 07/09/2014 04:31 PM, Andrew Morton wrote: >>> On Wed, 9 Jul 2014 16:24:06 +0200 Heiko Carstens <heiko.carstens@de.ibm.com> wrote: >>> >>>>> 058504edd026 ("fs/seq_file: fallback to vmalloc allocation") is suspected to >>>>> cause a crash. Actually I can't reprocude the crash nor would I be able to >>>>> tell how the commit could cause the crash. >>>>> Anyway, I'll be offline for the next 2.5 weeks. So if Sasha could confirm >>>>> that reverting the patch actually does fix the crash, please revert the >>>>> commit, unless somebody else can make sense of the report of course. >>>>> >>>>> I'm still wondering how Sasha could reproduce the crash. >>> This really sucks :( >>> >>> I suppose I'll queue up a revert of 058504edd026 as a for-3.16 bandaid. >>> >>> Sasha, it would be great if you could delve a bit further into this, >>> see if we can identify a way for others to reproduce the bug. >>> >>> Did you test that little patchlet which Heiko sent? >> >> I've been trying, but as there isn't an easy way to reproduce it, breakage >> elsewhere in -next (there is a long list) is slowing me down. I'll revert >> back to the -next where I've initially observed this issue and will retry. >> > > Thanks. The bug should be present in current mainline. > I've been running mainline for a while, and while I can reproduce a different bug for Hugh, I can't reproduce this one at all. Not sure how to call it now, maybe memory corruption from somewhere else (I did find an unrelated memory corruption couple days ago)? Thanks, Sasha ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-15 13:52 ` Sasha Levin @ 2014-07-15 23:01 ` David Rientjes 2014-07-23 20:57 ` Andrew Morton 0 siblings, 1 reply; 19+ messages in thread From: David Rientjes @ 2014-07-15 23:01 UTC (permalink / raw) To: Sasha Levin Cc: Andrew Morton, Heiko Carstens, linux-fsdevel, Al Viro, Jan Kara, Dave Jones, LKML, Linus Torvalds On Tue, 15 Jul 2014, Sasha Levin wrote: > I've been running mainline for a while, and while I can reproduce a different > bug for Hugh, I can't reproduce this one at all. Not sure how to call it now, > maybe memory corruption from somewhere else (I did find an unrelated memory > corruption couple days ago)? > I have a very large /proc/pid/mountinfo on one of my busy systems with VM and slub debugging enabled and have not been able to reproduce the issue while reading it in a loop for 24 hours. I would call the patch good on that basis, but it's very strange you'd report this problem a day after the patch was merged and could not reproduce it once it was reverted. (Unless the patch was misapplied locally and reapplying it now worked, but I doubt you originally merged -mm patches into a local tree yourself.) ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fs: use after free in /proc/pid/mountinfo 2014-07-15 23:01 ` David Rientjes @ 2014-07-23 20:57 ` Andrew Morton 0 siblings, 0 replies; 19+ messages in thread From: Andrew Morton @ 2014-07-23 20:57 UTC (permalink / raw) To: David Rientjes Cc: Sasha Levin, Heiko Carstens, linux-fsdevel, Al Viro, Jan Kara, Dave Jones, LKML, Linus Torvalds On Tue, 15 Jul 2014 16:01:34 -0700 (PDT) David Rientjes <rientjes@google.com> wrote: > On Tue, 15 Jul 2014, Sasha Levin wrote: > > > I've been running mainline for a while, and while I can reproduce a different > > bug for Hugh, I can't reproduce this one at all. Not sure how to call it now, > > maybe memory corruption from somewhere else (I did find an unrelated memory > > corruption couple days ago)? > > > > I have a very large /proc/pid/mountinfo on one of my busy systems with VM > and slub debugging enabled and have not been able to reproduce the issue > while reading it in a loop for 24 hours. I would call the patch good on > that basis, but it's very strange you'd report this problem a day after > the patch was merged and could not reproduce it once it was reverted. > (Unless the patch was misapplied locally and reapplying it now worked, but > I doubt you originally merged -mm patches into a local tree yourself.) I was going to revert "fs/seq_file: fallback to vmalloc allocation" but the idea of doing that is so lame and pathetic that I think I'll not do that. If we have a bug then let's just find and fix it when someone is able to trigger it. ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2014-07-23 20:57 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2014-07-02 11:55 fs: use after free in /proc/pid/mountinfo Sasha Levin 2014-07-03 3:15 ` Sasha Levin 2014-07-03 21:37 ` David Rientjes 2014-07-04 14:55 ` Sasha Levin 2014-07-04 22:35 ` [stable] please do not merge 058504edd026 (was Re: fs: use after free in /proc/pid/mountinfo) David Rientjes 2014-07-04 22:35 ` David Rientjes 2014-07-07 23:05 ` [stable] " Greg KH 2014-07-07 23:05 ` Greg KH 2014-07-07 23:06 ` [stable] " Andrew Morton 2014-07-07 23:19 ` Greg KH 2014-07-07 23:19 ` Greg KH 2014-07-06 10:04 ` fs: use after free in /proc/pid/mountinfo Heiko Carstens 2014-07-09 14:24 ` Heiko Carstens 2014-07-09 20:31 ` Andrew Morton 2014-07-09 22:10 ` Sasha Levin 2014-07-09 22:59 ` Andrew Morton 2014-07-15 13:52 ` Sasha Levin 2014-07-15 23:01 ` David Rientjes 2014-07-23 20:57 ` Andrew Morton
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.