* net: netlink executing RO memory
@ 2014-06-05 20:21 Sasha Levin
2014-06-06 5:45 ` Sasha Levin
2014-06-06 9:02 ` David Laight
0 siblings, 2 replies; 6+ messages in thread
From: Sasha Levin @ 2014-06-05 20:21 UTC (permalink / raw)
To: David S. Miller; +Cc: Eric W. Biederman, Eric Dumazet, netdev, LKML
Hi all,
While fuzzing with trinity inside a KVM tools guest running the latest -next
kernel I've stumbled on the following spew:
[ 306.065161] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
[ 306.067295] BUG: unable to handle kernel paging request at ffff880053b8fd08
[ 306.069237] IP: 0xffff880053b8fd08 (??:?)
[ 306.070071] PGD 24b9c067 PUD 705dd2067 PMD 705d34067 PTE 8000000053b8f163
[ 306.070071] Oops: 0011 [#1] PREEMPT SMP DEBUG_PAGEALLOC
[ 306.070071] Dumping ftrace buffer:
[ 306.070071] (ftrace buffer empty)
[ 306.070071] Modules linked in:
[ 306.070071] CPU: 16 PID: 9577 Comm: trinity-c194 Tainted: G W 3.15.0-rc8-next-20140605-sasha-00020-g833a807 #592
[ 306.070071] task: ffff880053b90000 ti: ffff880053b8c000 task.ti: ffff880053b8c000
[ 306.070071] RIP: 0xffff880053b8fd08 (??:?)
[ 306.070071] RSP: 0018:ffff880053b8fcc8 EFLAGS: 00010287
[ 306.070071] RAX: ffff8806286e8000 RBX: 0000000000000000 RCX: 0000004742e748d4
[ 306.070071] RDX: 0000000000000007 RSI: ffffffff9fffd31c RDI: ffffffffa0558ed5
[ 306.070071] RBP: ffff880053b8fd08 R08: 0000000000005d3c R09: 0000000000000000
[ 306.070071] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880053b8fdf8
[ 306.070071] R13: ffff8803d55b8000 R14: ffff88065ea35cd0 R15: 0000000000000000
[ 306.070071] FS: 00007f5c3bb31700(0000) GS:ffff8803d7000000(0000) knlGS:0000000000000000
[ 306.070071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 306.070071] CR2: ffff880053b8fd08 CR3: 0000000053b6b000 CR4: 00000000000006a0
[ 306.070071] DR0: 00000000006d6000 DR1: 00000000006d6000 DR2: 0000000000000000
[ 306.070071] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 000000000095060a
[ 306.070071] Stack:
[ 306.070071] ffff880053b8fdc8 00000000026d7de0 0000000000000010 7fffffffffffffff
[ 306.070071] 0000000000000000 ffff88065ea35cd0 0000000000000001 0000000000000000
[ 306.095047] ffff880053b8fda8 ffffffffa0000ad1 ffff8803d55b8000 ffff8803d71d8340
[ 306.095047] Call Trace:
[ 306.095047] netlink_sendmsg (net/netlink/af_netlink.c:2398)
[ 306.095047] sock_aio_write (net/socket.c:959 net/socket.c:974)
[ 306.095047] ? sched_clock_local (kernel/sched/clock.c:214)
[ 306.095047] ? vtime_account_user (kernel/sched/cputime.c:687)
[ 306.095047] do_sync_write (fs/read_write.c:458)
[ 306.095047] vfs_write (fs/read_write.c:534)
[ 306.095047] SyS_write (fs/read_write.c:584 fs/read_write.c:576)
[ 306.095047] tracesys (arch/x86/kernel/entry_64.S:542)
[ 306.095047] Code: 00 00 00 ff ff ff ff ff ff ff 7f 00 00 00 00 00 00 00 00 d0 5c a3 5e 06 88 ff ff 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <a8> fd b8 53 00 88 ff ff d1 0a 00 a0 ff ff ff ff 00 80 5b d5 03
All code
========
0: 00 00 add %al,(%rax)
2: 00 ff add %bh,%bh
4: ff (bad)
5: ff (bad)
6: ff (bad)
7: ff (bad)
8: ff (bad)
9: ff (bad)
a: 7f 00 jg 0xc
c: 00 00 add %al,(%rax)
e: 00 00 add %al,(%rax)
10: 00 00 add %al,(%rax)
12: 00 d0 add %dl,%al
14: 5c pop %rsp
15: a3 5e 06 88 ff ff 01 movabs %eax,0x1ffff88065e
1c: 00 00
...
2a:* 00 a8 fd b8 53 00 add %ch,0x53b8fd(%rax) <-- trapping instruction
30: 88 ff mov %bh,%bh
32: ff d1 callq *%rcx
34: 0a 00 or (%rax),%al
36: a0 ff ff ff ff 00 80 movabs 0xd55b8000ffffffff,%al
3d: 5b d5
3f: 03 00 add (%rax),%eax
Code starting with the faulting instruction
===========================================
0: a8 fd test $0xfd,%al
2: b8 53 00 88 ff mov $0xff880053,%eax
7: ff d1 callq *%rcx
9: 0a 00 or (%rax),%al
b: a0 ff ff ff ff 00 80 movabs 0xd55b8000ffffffff,%al
12: 5b d5
14: 03 00 add (%rax),%eax
[ 306.095047] RIP 0xffff880053b8fd08 (??:?)
[ 306.095047] RSP <ffff880053b8fcc8>
[ 306.095047] CR2: ffff880053b8fd08
Thanks,
Sasha
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: net: netlink executing RO memory
2014-06-05 20:21 net: netlink executing RO memory Sasha Levin
@ 2014-06-06 5:45 ` Sasha Levin
2014-06-07 15:07 ` Sasha Levin
2014-06-06 9:02 ` David Laight
1 sibling, 1 reply; 6+ messages in thread
From: Sasha Levin @ 2014-06-06 5:45 UTC (permalink / raw)
To: David S. Miller; +Cc: Eric W. Biederman, Eric Dumazet, netdev, LKML
On 06/05/2014 04:21 PM, Sasha Levin wrote:
> Hi all,
>
> While fuzzing with trinity inside a KVM tools guest running the latest -next
> kernel I've stumbled on the following spew:
>
> [ 306.065161] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
> [ 306.067295] BUG: unable to handle kernel paging request at ffff880053b8fd08
Same issue reproduced multiple times with exactly the same trace, so I think that it
rules out random memory corruption.
Thanks,
Sasha
^ permalink raw reply [flat|nested] 6+ messages in thread
* RE: netlink executing RO memory
2014-06-05 20:21 net: netlink executing RO memory Sasha Levin
2014-06-06 5:45 ` Sasha Levin
@ 2014-06-06 9:02 ` David Laight
1 sibling, 0 replies; 6+ messages in thread
From: David Laight @ 2014-06-06 9:02 UTC (permalink / raw)
To: 'Sasha Levin', David S. Miller
Cc: Eric W. Biederman, Eric Dumazet, netdev, LKML
From: Sasha Levin
> While fuzzing with trinity inside a KVM tools guest running the latest -next
> kernel I've stumbled on the following spew:
>
> [ 306.065161] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
> [ 306.067295] BUG: unable to handle kernel paging request at ffff880053b8fd08
> [ 306.069237] IP: 0xffff880053b8fd08 (??:?)
> [ 306.070071] PGD 24b9c067 PUD 705dd2067 PMD 705d34067 PTE 8000000053b8f163
> [ 306.070071] Oops: 0011 [#1] PREEMPT SMP DEBUG_PAGEALLOC
> [ 306.070071] Dumping ftrace buffer:
> [ 306.070071] (ftrace buffer empty)
> [ 306.070071] Modules linked in:
> [ 306.070071] CPU: 16 PID: 9577 Comm: trinity-c194 Tainted: G W 3.15.0-rc8-next-20140605-
> sasha-00020-g833a807 #592
> [ 306.070071] task: ffff880053b90000 ti: ffff880053b8c000 task.ti: ffff880053b8c000
> [ 306.070071] RIP: 0xffff880053b8fd08 (??:?)
> [ 306.070071] RSP: 0018:ffff880053b8fcc8 EFLAGS: 00010287
> [ 306.070071] RAX: ffff8806286e8000 RBX: 0000000000000000 RCX: 0000004742e748d4
> [ 306.070071] RDX: 0000000000000007 RSI: ffffffff9fffd31c RDI: ffffffffa0558ed5
> [ 306.070071] RBP: ffff880053b8fd08 R08: 0000000000005d3c R09: 0000000000000000
> [ 306.070071] R10: 0000000000000000 R11: 0000000000000001 R12: ffff880053b8fdf8
> [ 306.070071] R13: ffff8803d55b8000 R14: ffff88065ea35cd0 R15: 0000000000000000
> [ 306.070071] FS: 00007f5c3bb31700(0000) GS:ffff8803d7000000(0000) knlGS:0000000000000000
> [ 306.070071] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [ 306.070071] CR2: ffff880053b8fd08 CR3: 0000000053b6b000 CR4: 00000000000006a0
> [ 306.070071] DR0: 00000000006d6000 DR1: 00000000006d6000 DR2: 0000000000000000
> [ 306.070071] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 000000000095060a
> [ 306.070071] Stack:
> [ 306.070071] ffff880053b8fdc8 00000000026d7de0 0000000000000010 7fffffffffffffff
> [ 306.070071] 0000000000000000 ffff88065ea35cd0 0000000000000001 0000000000000000
> [ 306.095047] ffff880053b8fda8 ffffffffa0000ad1 ffff8803d55b8000 ffff8803d71d8340
> [ 306.095047] Call Trace:
> [ 306.095047] netlink_sendmsg (net/netlink/af_netlink.c:2398)
> [ 306.095047] sock_aio_write (net/socket.c:959 net/socket.c:974)
> [ 306.095047] ? sched_clock_local (kernel/sched/clock.c:214)
> [ 306.095047] ? vtime_account_user (kernel/sched/cputime.c:687)
> [ 306.095047] do_sync_write (fs/read_write.c:458)
> [ 306.095047] vfs_write (fs/read_write.c:534)
> [ 306.095047] SyS_write (fs/read_write.c:584 fs/read_write.c:576)
> [ 306.095047] tracesys (arch/x86/kernel/entry_64.S:542)
> [ 306.095047] Code: 00 00 00 ff ff ff ff ff ff ff 7f 00 00 00 00 00 00 00 00 d0 5c a3 5e 06 88 ff ff
> 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 <a8> fd b8 53 00 88 ff ff d1 0a 00 a0 ff ff ff ff 00
> 80 5b d5 03
> All code
> ========
> 0: 00 00 add %al,(%rax)
> 2: 00 ff add %bh,%bh
> 4: ff (bad)
> 5: ff (bad)
> 6: ff (bad)
> 7: ff (bad)
> 8: ff (bad)
> 9: ff (bad)
> a: 7f 00 jg 0xc
> c: 00 00 add %al,(%rax)
> e: 00 00 add %al,(%rax)
> 10: 00 00 add %al,(%rax)
> 12: 00 d0 add %dl,%al
> 14: 5c pop %rsp
> 15: a3 5e 06 88 ff ff 01 movabs %eax,0x1ffff88065e
> 1c: 00 00
> ...
> 2a:* 00 a8 fd b8 53 00 add %ch,0x53b8fd(%rax) <-- trapping instruction
> 30: 88 ff mov %bh,%bh
> 32: ff d1 callq *%rcx
> 34: 0a 00 or (%rax),%al
> 36: a0 ff ff ff ff 00 80 movabs 0xd55b8000ffffffff,%al
> 3d: 5b d5
> 3f: 03 00 add (%rax),%eax
The disassembly hasn't aligned itself with the instruction boundaries.
> Code starting with the faulting instruction
> ===========================================
> 0: a8 fd test $0xfd,%al
> 2: b8 53 00 88 ff mov $0xff880053,%eax
> 7: ff d1 callq *%rcx
> 9: 0a 00 or (%rax),%al
> b: a0 ff ff ff ff 00 80 movabs 0xd55b8000ffffffff,%al
> 12: 5b d5
> 14: 03 00 add (%rax),%eax
And that doesn't look like valid code at all.
The code must have jumped to the instruction - otherwise it would
have faulted on the previous one.
So most likely there was an indirect call from the previous frame.
Note that $rbp also contains the faulting address.
David
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: net: netlink executing RO memory
2014-06-06 5:45 ` Sasha Levin
@ 2014-06-07 15:07 ` Sasha Levin
2014-06-07 20:36 ` Thomas Gleixner
0 siblings, 1 reply; 6+ messages in thread
From: Sasha Levin @ 2014-06-07 15:07 UTC (permalink / raw)
To: David S. Miller
Cc: Eric W. Biederman, Eric Dumazet, netdev, LKML, Thomas Gleixner
On 06/06/2014 01:45 AM, Sasha Levin wrote:
> On 06/05/2014 04:21 PM, Sasha Levin wrote:
>> Hi all,
>>
>> While fuzzing with trinity inside a KVM tools guest running the latest -next
>> kernel I've stumbled on the following spew:
>>
>> [ 306.065161] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
>> [ 306.067295] BUG: unable to handle kernel paging request at ffff880053b8fd08
>
> Same issue reproduced multiple times with exactly the same trace, so I think that it
> rules out random memory corruption.
I might have another lead of this: I caught debug objects complaining about freeing
active objects:
[ 592.020501] ODEBUG: free active (active state 1) object type: rcu_head hint: (null)
[ 592.020501] Modules linked in:
[ 592.020501] CPU: 15 PID: 16543 Comm: trinity-c47 Not tainted 3.15.0-rc8-next-20140606-sasha-00021-ga9d3a0b-dirty #596
[ 592.020501] 0000000000000009 ffff880224793988 ffffffff9350fe6b 0000000000000002
[ 592.020501] ffff8802247939d8 ffff8802247939c8 ffffffff9015f96c ffff88000fd40cf8
[ 592.020501] ffff88006d9a9870 ffffffff9508a000 ffffffff948fbd48 ffffffff97891e90
[ 592.020501] Call Trace:
[ 592.020501] dump_stack (lib/dump_stack.c:52)
[ 592.020501] warn_slowpath_common (kernel/panic.c:430)
[ 592.020501] warn_slowpath_fmt (kernel/panic.c:445)
[ 592.020501] debug_print_object (lib/debugobjects.c:265)
[ 592.020501] __debug_check_no_obj_freed (lib/debugobjects.c:698)
[ 592.020501] debug_check_no_obj_freed (lib/debugobjects.c:727)
[ 592.020501] __vunmap (mm/vmalloc.c:1457)
[ 592.020501] vfree (mm/vmalloc.c:1505)
[ 592.020501] netlink_skb_destructor (net/netlink/af_netlink.c:882)
[ 592.020501] skb_release_head_state (net/core/skbuff.c:566)
[ 592.020501] skb_release_all (net/core/skbuff.c:584)
[ 592.020501] __kfree_skb (net/core/skbuff.c:529 net/core/skbuff.c:600)
[ 592.020501] consume_skb (net/core/skbuff.c:672)
[ 592.020501] skb_free_datagram (include/net/sock.h:1419 include/net/sock.h:1450 net/core/datagram.c:244)
[ 592.020501] netlink_recvmsg (net/netlink/af_netlink.c:2482)
[ 592.020501] ? preempt_count_sub (kernel/sched/core.c:2602)
[ 592.020501] sock_aio_read (net/socket.c:917 net/socket.c:935)
[ 592.020501] ? rw_copy_check_uvector (fs/read_write.c:753)
[ 592.020501] do_sync_readv_writev (fs/read_write.c:683)
[ 592.020501] do_readv_writev (fs/read_write.c:837)
[ 592.020501] ? vtime_account_user (kernel/sched/cputime.c:687)
[ 592.020501] ? get_parent_ip (kernel/sched/core.c:2546)
[ 592.020501] ? __fget_light (include/linux/rcupdate.h:428 include/linux/fdtable.h:80 fs/file.c:684)
[ 592.020501] vfs_readv (fs/read_write.c:866)
[ 592.020501] SyS_readv (fs/read_write.c:892 fs/read_write.c:884)
[ 592.020501] tracesys (arch/x86/kernel/entry_64.S:542)
I can't link them together (stacks don't match with what I'd expect to see in that
case), but that might be related somehow.
Thanks,
Sasha
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: net: netlink executing RO memory
2014-06-07 15:07 ` Sasha Levin
@ 2014-06-07 20:36 ` Thomas Gleixner
2014-06-07 20:40 ` Thomas Gleixner
0 siblings, 1 reply; 6+ messages in thread
From: Thomas Gleixner @ 2014-06-07 20:36 UTC (permalink / raw)
To: Sasha Levin
Cc: David S. Miller, Eric W. Biederman, Eric Dumazet, netdev, LKML
On Sat, 7 Jun 2014, Sasha Levin wrote:
> On 06/06/2014 01:45 AM, Sasha Levin wrote:
> > On 06/05/2014 04:21 PM, Sasha Levin wrote:
> >> Hi all,
> >>
> >> While fuzzing with trinity inside a KVM tools guest running the latest -next
> >> kernel I've stumbled on the following spew:
> >>
> >> [ 306.065161] kernel tried to execute NX-protected page - exploit attempt? (uid: 0)
> >> [ 306.067295] BUG: unable to handle kernel paging request at ffff880053b8fd08
> >
> > Same issue reproduced multiple times with exactly the same trace, so I think that it
> > rules out random memory corruption.
>
> I might have another lead of this: I caught debug objects complaining about freeing
> active objects:
>
> [ 592.020501] ODEBUG: free active (active state 1) object type: rcu_head hint: (null)
So something in the memory which is freed is queued in rcu. state 1:
STATE_RCU_HEAD_QUEUED. But why is that rcu_head in the vmalloced skb
memory?
That's going to be a nice puzzle to find the culprit.
So one thing which might give us at least some data is the debug patch
below. With CONFIG_STACKTRACE enabled and
# echo 1 >/sys/kernel/debug/tracing/options/stacktrace
we should get a recording of rcu_free() calls along with the
stacktrace for each. So we should be able to see which code path
actually queued the thing. Maybe that's enough of an hint, but at
least it gives us an idea which code path to instrument further.
Thanks,
tglx
---------------------
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 962d1d5..7241235 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -698,6 +698,7 @@ EXPORT_SYMBOL_GPL(call_rcu);
void kfree_call_rcu(struct rcu_head *head,
void (*func)(struct rcu_head *rcu))
{
+ trace_printk("head: %p func: %pS\n", head, func);
__call_rcu(head, func, &rcu_preempt_state, -1, 1);
}
EXPORT_SYMBOL_GPL(kfree_call_rcu);
@@ -1091,6 +1092,7 @@ static void rcu_preempt_check_callbacks(int cpu)
void kfree_call_rcu(struct rcu_head *head,
void (*func)(struct rcu_head *rcu))
{
+ trace_printk("head: %p func: %pS\n", head, func);
__call_rcu(head, func, &rcu_sched_state, -1, 1);
}
EXPORT_SYMBOL_GPL(kfree_call_rcu);
diff --git a/lib/debugobjects.c b/lib/debugobjects.c
index e0731c3..7610834 100644
--- a/lib/debugobjects.c
+++ b/lib/debugobjects.c
@@ -252,8 +252,9 @@ static void debug_print_object(struct debug_obj *obj, char *msg)
if (limit < 5 && descr != descr_test) {
void *hint = descr->debug_hint ?
- descr->debug_hint(obj->object) : NULL;
+ descr->debug_hint(obj->object) : obj->object;
limit++;
+ tracing_off();
WARN(1, KERN_ERR "ODEBUG: %s %s (active state %u) "
"object type: %s hint: %pS\n",
msg, obj_states[obj->state], obj->astate,
^ permalink raw reply related [flat|nested] 6+ messages in thread
* Re: net: netlink executing RO memory
2014-06-07 20:36 ` Thomas Gleixner
@ 2014-06-07 20:40 ` Thomas Gleixner
0 siblings, 0 replies; 6+ messages in thread
From: Thomas Gleixner @ 2014-06-07 20:40 UTC (permalink / raw)
To: Sasha Levin
Cc: David S. Miller, Eric W. Biederman, Eric Dumazet, netdev, LKML
On Sat, 7 Jun 2014, Thomas Gleixner wrote:
> On Sat, 7 Jun 2014, Sasha Levin wrote:
> So one thing which might give us at least some data is the debug patch
> below. With CONFIG_STACKTRACE enabled and
>
> # echo 1 >/sys/kernel/debug/tracing/options/stacktrace
>
> we should get a recording of rcu_free() calls along with the
> stacktrace for each. So we should be able to see which code path
> actually queued the thing. Maybe that's enough of an hint, but at
> least it gives us an idea which code path to instrument further.
This one is better..
Thanks,
tglx
---------------------
diff --git a/kernel/rcu/tree_plugin.h b/kernel/rcu/tree_plugin.h
index 962d1d5..7241235 100644
--- a/kernel/rcu/tree_plugin.h
+++ b/kernel/rcu/tree_plugin.h
@@ -698,6 +698,7 @@ EXPORT_SYMBOL_GPL(call_rcu);
void kfree_call_rcu(struct rcu_head *head,
void (*func)(struct rcu_head *rcu))
{
+ trace_printk("head: %p func: %pS\n", head, func);
__call_rcu(head, func, &rcu_preempt_state, -1, 1);
}
EXPORT_SYMBOL_GPL(kfree_call_rcu);
@@ -1091,6 +1092,7 @@ static void rcu_preempt_check_callbacks(int cpu)
void kfree_call_rcu(struct rcu_head *head,
void (*func)(struct rcu_head *rcu))
{
+ trace_printk("head: %p func: %pS\n", head, func);
__call_rcu(head, func, &rcu_sched_state, -1, 1);
}
EXPORT_SYMBOL_GPL(kfree_call_rcu);
diff --git a/lib/debugobjects.c b/lib/debugobjects.c
index e0731c3..fd5cafd 100644
--- a/lib/debugobjects.c
+++ b/lib/debugobjects.c
@@ -693,6 +693,9 @@ repeat:
switch (obj->state) {
case ODEBUG_STATE_ACTIVE:
+ trace_printk("free %p obj %p\n", address,
+ obj->object);
+ tracing_off();
debug_print_object(obj, "free");
descr = obj->descr;
state = obj->state;
^ permalink raw reply related [flat|nested] 6+ messages in thread
end of thread, other threads:[~2014-06-07 20:40 UTC | newest]
Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-06-05 20:21 net: netlink executing RO memory Sasha Levin
2014-06-06 5:45 ` Sasha Levin
2014-06-07 15:07 ` Sasha Levin
2014-06-07 20:36 ` Thomas Gleixner
2014-06-07 20:40 ` Thomas Gleixner
2014-06-06 9:02 ` David Laight
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).