* fanotify use after free.
@ 2014-01-22 6:27 Dave Jones
2014-01-22 16:43 ` Dave Jones
2014-01-22 18:20 ` Linus Torvalds
0 siblings, 2 replies; 19+ messages in thread
From: Dave Jones @ 2014-01-22 6:27 UTC (permalink / raw)
To: jack; +Cc: Linux Kernel
Jan,
since yesterdays changes, on boot I see a flood of messages from slub debug during boot..
=============================================================================
BUG fanotify_event_info (Not tainted): Poison overwritten
-----------------------------------------------------------------------------
Disabling lock debugging due to kernel taint
INFO: 0xffff880247e45bc8-0xffff880247e45bcb. First byte 0x0 instead of 0x6b
INFO: Allocated in fanotify_handle_event+0x136/0x390 age=0 cpu=0 pid=293
__slab_alloc+0x456/0x565
kmem_cache_alloc+0x1fe/0x260
fanotify_handle_event+0x136/0x390
send_to_group+0xd3/0x1c0
fsnotify+0x1c8/0x340
open_exec+0xe2/0x120
load_elf_binary+0x7b7/0x18e0
search_binary_handler+0x94/0x1b0
do_execve_common.isra.26+0x5d7/0x7d0
SyS_execve+0x36/0x50
stub_execve+0x69/0xa0
INFO: Freed in fanotify_free_event+0x2e/0x40 age=0 cpu=3 pid=290
__slab_free+0x4a/0x382
kmem_cache_free+0x1c9/0x210
fanotify_free_event+0x2e/0x40
fsnotify_destroy_event+0x21/0x30
fanotify_read+0x39e/0x5e0
vfs_read+0x9b/0x160
SyS_read+0x58/0xb0
tracesys+0xdd/0xe2
INFO: Slab 0xffffea00091f9100 objects=20 used=20 fp=0x (null) flags=0x20000000004080
INFO: Object 0xffff880247e45b90 @offset=7056 fp=0xffff880247e44000
Bytes b4 ffff880247e45b80: 00 00 00 00 00 00 00 00 5a 5a 5a 5a 5a 5a 5a 5a ........ZZZZZZZZ
Object ffff880247e45b90: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff880247e45ba0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff880247e45bb0: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Object ffff880247e45bc0: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk.
Redzone ffff880247e45bd0: bb bb bb bb bb bb bb bb ........
Padding ffff880247e45d10: 5a 5a 5a 5a 5a 5a 5a 5a ZZZZZZZZ
CPU: 0 PID: 293 Comm: mount Tainted: G B 3.13.0+ #28
ffff880247e45b90 000000008c7fe87c ffff8800874cbb28 ffffffff9c710632
ffff88024a776ac0 ffff8800874cbb68 ffffffff9c194dad 0000000000000008
ffff880200000001 ffff880247e45bcc ffff88024a776ac0 000000000000006b
Call Trace:
[<ffffffff9c710632>] dump_stack+0x4e/0x7a
[<ffffffff9c194dad>] print_trailer+0x14d/0x200
[<ffffffff9c19505f>] check_bytes_and_report+0xcf/0x110
[<ffffffff9c196037>] check_object+0x1d7/0x250
[<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390
[<ffffffff9c70ead7>] alloc_debug_processing+0x76/0x118
[<ffffffff9c70f77d>] __slab_alloc+0x456/0x565
[<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390
[<ffffffff9c1ccea4>] ? mntput+0x24/0x40
[<ffffffff9c1b5dc9>] ? terminate_walk+0x69/0x70
[<ffffffff9c1ba6fe>] ? do_last+0x25e/0x1390
[<ffffffff9c1b6cf8>] ? inode_permission+0x18/0x50
[<ffffffff9c1f4ae6>] ? fanotify_handle_event+0x136/0x390
[<ffffffff9c1980fe>] kmem_cache_alloc+0x1fe/0x260
[<ffffffff9c1f4ae6>] fanotify_handle_event+0x136/0x390
[<ffffffff9c1bb8fd>] ? path_openat+0xcd/0x6a0
[<ffffffff9c1f0e63>] send_to_group+0xd3/0x1c0
[<ffffffff9c1f0fdf>] ? fsnotify+0x8f/0x340
[<ffffffff9c1f1118>] fsnotify+0x1c8/0x340
[<ffffffff9c1a9b4f>] do_sys_open+0x19f/0x230
[<ffffffff9c1a9bfe>] SyS_open+0x1e/0x20
[<ffffffff9c723764>] tracesys+0xdd/0xe2
FIX fanotify_event_info: Restoring 0xffff880247e45bc8-0xffff880247e45bcb=0x6b
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-22 6:27 fanotify use after free Dave Jones
@ 2014-01-22 16:43 ` Dave Jones
2014-01-22 18:20 ` Linus Torvalds
1 sibling, 0 replies; 19+ messages in thread
From: Dave Jones @ 2014-01-22 16:43 UTC (permalink / raw)
To: jack, Linux Kernel; +Cc: Linus Torvalds
On Wed, Jan 22, 2014 at 01:27:30AM -0500, Dave Jones wrote:
> Jan,
>
> since yesterdays changes, on boot I see a flood of messages from slub debug during boot..
>
> =============================================================================
> BUG fanotify_event_info (Not tainted): Poison overwritten
> -----------------------------------------------------------------------------
>
> Disabling lock debugging due to kernel taint
> INFO: 0xffff880247e45bc8-0xffff880247e45bcb. First byte 0x0 instead of 0x6b
> INFO: Allocated in fanotify_handle_event+0x136/0x390 age=0 cpu=0 pid=293
> __slab_alloc+0x456/0x565
> kmem_cache_alloc+0x1fe/0x260
> fanotify_handle_event+0x136/0x390
> send_to_group+0xd3/0x1c0
> fsnotify+0x1c8/0x340
> open_exec+0xe2/0x120
> load_elf_binary+0x7b7/0x18e0
> search_binary_handler+0x94/0x1b0
> do_execve_common.isra.26+0x5d7/0x7d0
> SyS_execve+0x36/0x50
> stub_execve+0x69/0xa0
> INFO: Freed in fanotify_free_event+0x2e/0x40 age=0 cpu=3 pid=290
> __slab_free+0x4a/0x382
> kmem_cache_free+0x1c9/0x210
> fanotify_free_event+0x2e/0x40
> fsnotify_destroy_event+0x21/0x30
> fanotify_read+0x39e/0x5e0
> vfs_read+0x9b/0x160
> SyS_read+0x58/0xb0
> tracesys+0xdd/0xe2
> INFO: Slab 0xffffea00091f9100 objects=20 used=20 fp=0x (null) flags=0x20000000004080
Reverting 7053aee26a3548ebaba046ae2e52396ccf56ac6c makes this go away.
Dave
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-22 6:27 fanotify use after free Dave Jones
2014-01-22 16:43 ` Dave Jones
@ 2014-01-22 18:20 ` Linus Torvalds
2014-01-22 23:36 ` Jan Kara
1 sibling, 1 reply; 19+ messages in thread
From: Linus Torvalds @ 2014-01-22 18:20 UTC (permalink / raw)
To: Dave Jones, Jan Kara, Linux Kernel
On Tue, Jan 21, 2014 at 10:27 PM, Dave Jones <davej@redhat.com> wrote:
>
> BUG fanotify_event_info (Not tainted): Poison overwritten
Looking at the poison data, it seems that is is the
u32 response;
field that has been overwritten (with all zero).
That doesn't really help me guess where the bug is, though. That code
is crazy. For example, looking at one place where it is set, we have:
- process_access_response():
re->event->response = response;
wake_up(&group->fanotify_data.access_waitq);
kmem_cache_free(fanotify_response_event_cache, re);
which looks buggy in *so* many ways. In particular, we're doing a
kmem_cache_free() on "re", but what happened to "re->event" that we
just used? There was no release of that anywhere. Wut?
So it seems that the lifetime of these "fanotify_event_info"
structures is completely buggered. I don't even see any *attempt* to
maintain reference counts or other lifetime info. People free the
containers that point to them without doing anything at all about the
fsnotify_event that contains the fanotify_event_info that they point
to.
Jan - how is the lifetime of the fanotify_event_info tied to the
lifetime of the fanotify_response_event structure that contains
pointers into it? Because I don't see it.
And considering that it's the response field that gets overwritten, it
really sounds like *that* is the exact issue at play here - there is
some fanotify_response_event structure holding a pointer to the
fanotify_event that is embedded into a fanotify_event_info that has
been freed.
Jan?
Linus
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-22 18:20 ` Linus Torvalds
@ 2014-01-22 23:36 ` Jan Kara
2014-01-23 0:08 ` Linus Torvalds
0 siblings, 1 reply; 19+ messages in thread
From: Jan Kara @ 2014-01-22 23:36 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Dave Jones, Jan Kara, Linux Kernel, jkosina
[-- Attachment #1: Type: text/plain, Size: 2479 bytes --]
On Wed 22-01-14 10:20:01, Linus Torvalds wrote:
> On Tue, Jan 21, 2014 at 10:27 PM, Dave Jones <davej@redhat.com> wrote:
> >
> > BUG fanotify_event_info (Not tainted): Poison overwritten
>
> Looking at the poison data, it seems that is is the
>
> u32 response;
>
> field that has been overwritten (with all zero).
>
> That doesn't really help me guess where the bug is, though. That code
> is crazy. For example, looking at one place where it is set, we have:
>
> - process_access_response():
>
> re->event->response = response;
>
> wake_up(&group->fanotify_data.access_waitq);
>
> kmem_cache_free(fanotify_response_event_cache, re);
>
> which looks buggy in *so* many ways. In particular, we're doing a
> kmem_cache_free() on "re", but what happened to "re->event" that we
> just used? There was no release of that anywhere. Wut?
>
> So it seems that the lifetime of these "fanotify_event_info"
> structures is completely buggered. I don't even see any *attempt* to
> maintain reference counts or other lifetime info. People free the
> containers that point to them without doing anything at all about the
> fsnotify_event that contains the fanotify_event_info that they point
> to.
>
> Jan - how is the lifetime of the fanotify_event_info tied to the
> lifetime of the fanotify_response_event structure that contains
> pointers into it? Because I don't see it.
Yeah, I messed that up. They aren't tied in any way - well, in fact they
end up being tied but in a wrong way. fanotify_event_info lives from the
moment event happens to the moment user reads the event. At that moment the
fanotify_response_event gets created (for those special permission events),
pointing to fanotify_event_info which will get freed just several lines
further :-|
But refcounting seems like an overkill for this - there is exactly one
fanotify_response_event structure iff it is a permission event. So
something like the (completely untested) attached patch should fix the
problem. But I agree it's a bit ugly so we might want something different.
I'll try to think about something better tomorrow.
> And considering that it's the response field that gets overwritten, it
> really sounds like *that* is the exact issue at play here - there is
> some fanotify_response_event structure holding a pointer to the
> fanotify_event that is embedded into a fanotify_event_info that has
> been freed.
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
[-- Attachment #2: fanotify_corruption.diff --]
[-- Type: text/x-patch, Size: 1098 bytes --]
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index 58772623f02a..756e9b047e27 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group,
}
#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
- if (fsn_event->mask & FAN_ALL_PERM_EVENTS)
+ if (fsn_event->mask & FAN_ALL_PERM_EVENTS) {
ret = fanotify_get_response_from_access(group, event);
+ fsnotify_destroy_event(group, event);
+ }
#endif
return ret;
}
diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c
index 57d7c083cb4b..d493c72c71fd 100644
--- a/fs/notify/fanotify/fanotify_user.c
+++ b/fs/notify/fanotify/fanotify_user.c
@@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,
if (IS_ERR(kevent))
break;
ret = copy_event_to_user(group, kevent, buf);
- fsnotify_destroy_event(group, kevent);
+ if (!(kevent->mask & FAN_ALL_PERM_EVENTS))
+ fsnotify_destroy_event(group, kevent);
if (ret < 0)
break;
buf += ret;
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-22 23:36 ` Jan Kara
@ 2014-01-23 0:08 ` Linus Torvalds
2014-01-23 0:32 ` Dave Jones
2014-01-23 10:23 ` Jiri Kosina
0 siblings, 2 replies; 19+ messages in thread
From: Linus Torvalds @ 2014-01-23 0:08 UTC (permalink / raw)
To: Jan Kara; +Cc: Dave Jones, Linux Kernel, Jiri Kosina
On Wed, Jan 22, 2014 at 3:36 PM, Jan Kara <jack@suse.cz> wrote:
>
> But refcounting seems like an overkill for this - there is exactly one
> fanotify_response_event structure iff it is a permission event. So
> something like the (completely untested) attached patch should fix the
> problem. But I agree it's a bit ugly so we might want something different.
> I'll try to think about something better tomorrow.
Ok, In the meantime, Dave, can you verify whether this hacky patch
fixes your problem?
Linus
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-23 0:08 ` Linus Torvalds
@ 2014-01-23 0:32 ` Dave Jones
2014-01-23 15:05 ` Jan Kara
2014-01-23 10:23 ` Jiri Kosina
1 sibling, 1 reply; 19+ messages in thread
From: Dave Jones @ 2014-01-23 0:32 UTC (permalink / raw)
To: Linus Torvalds; +Cc: Jan Kara, Linux Kernel, Jiri Kosina
On Wed, Jan 22, 2014 at 04:08:52PM -0800, Linus Torvalds wrote:
> On Wed, Jan 22, 2014 at 3:36 PM, Jan Kara <jack@suse.cz> wrote:
> >
> > But refcounting seems like an overkill for this - there is exactly one
> > fanotify_response_event structure iff it is a permission event. So
> > something like the (completely untested) attached patch should fix the
> > problem. But I agree it's a bit ugly so we might want something different.
> > I'll try to think about something better tomorrow.
>
> Ok, In the meantime, Dave, can you verify whether this hacky patch
> fixes your problem?
It actually seems worse. I see the tail end of what looks like a slab corruption
trace, and then a total lockup. And of course none of this makes it over ttyUSB0
because it happens so early. Grr.
Dave
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-23 0:08 ` Linus Torvalds
2014-01-23 0:32 ` Dave Jones
@ 2014-01-23 10:23 ` Jiri Kosina
2014-01-23 15:05 ` Jan Kara
1 sibling, 1 reply; 19+ messages in thread
From: Jiri Kosina @ 2014-01-23 10:23 UTC (permalink / raw)
To: Linus Torvalds, Jan Kara; +Cc: Dave Jones, Linux Kernel
On Wed, 22 Jan 2014, Linus Torvalds wrote:
> > But refcounting seems like an overkill for this - there is exactly one
> > fanotify_response_event structure iff it is a permission event. So
> > something like the (completely untested) attached patch should fix the
> > problem. But I agree it's a bit ugly so we might want something different.
> > I'll try to think about something better tomorrow.
>
> Ok, In the meantime, Dave, can you verify whether this hacky patch
> fixes your problem?
I reported the same slab corruption yesterday as well here:
https://lkml.org/lkml/2014/1/22/173
With the patch applied, I am still seeing the slab corruption, preceeded
by GPF (which is not there without the patch) in
lockref_put_or_lock(&dentry->d_lockref) in dput():
general protection fault: 0000 [#1] SMP
Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys
CPU: 1 PID: 275 Comm: systemd-readahe Not tainted 3.13.0-03478-g670d0ac #1
Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
task: ffff880037c09150 ti: ffff88007359e000 task.ti: ffff88007359e000
RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
RSP: 0018:ffff88007359fc78 EFLAGS: 00010286
RAX: ffff880037c09150 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000
tpm_tis 00:09: 1.2 TPM (device-id 0x1020, rev-id 6)
tpm_tis 00:09: Intel iTPM workaround enabled
RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb
RBP: ffff88007359fc98 R08: 0000000000000002 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb
R13: ffff880037310690 R14: 0000000000000020 R15: ffff880036dbfc10
FS: 00007fa953e4a700(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007fff76a10258 CR3: 000000003659c000 CR4: 00000000000007e0
Stack:
6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880037310690
ffff88007359fcb8 ffffffff8159c59c ffffffff812fe101 6b6b6b6b6b6b6beb
ffff88007359fcd8 ffffffff812fe101 ffff88007359fd08 6b6b6b6b6b6b6b6b
Call Trace:
[<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
[<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
[<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
[<ffffffff811b1442>] dput+0x22/0x130
[<ffffffff811a3d45>] path_put+0x15/0x30
[<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
[<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
[<ffffffff811dd18b>] send_to_group+0xfb/0x180
[<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff81198859>] SyS_open+0x19/0x20
[<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff
RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
RSP <ffff88007359fc78>
---[ end trace 7a918209ee213d28 ]---
BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20
in_atomic(): 1, irqs_disabled(): 0, pid: 275, name: systemd-readahe
INFO: lockdep is turned off.
CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1
Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
ffff880037c09150 ffff88007359fa78 ffffffff8159702b ffff88007359fa98
ffffffff8107f621 ffffffff81a3d000 ffff880037b98d90 ffff88007359fab8
ffffffff8159b4ff ffff880037c09150 ffff880037c09150 ffff88007359fae8
Call Trace:
[<ffffffff8159702b>] dump_stack+0x72/0x87
[<ffffffff8107f621>] __might_sleep+0xe1/0x100
[<ffffffff8159b4ff>] down_read+0x1f/0x60
[<ffffffff810627ff>] exit_signals+0x1f/0x140
[<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20
[<ffffffff81052844>] do_exit+0xb4/0x4b0
[<ffffffff8159e23c>] oops_end+0xdc/0xe0
[<ffffffff81005f86>] die+0x56/0x90
[<ffffffff8159dea2>] do_general_protection+0x162/0x170
[<ffffffff8159d40c>] ? restore_args+0x30/0x30
[<ffffffff8159d592>] general_protection+0x22/0x30
[<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
[<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
[<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
[<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
[<ffffffff811b1442>] dput+0x22/0x130
[<ffffffff811a3d45>] path_put+0x15/0x30
[<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
[<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
[<ffffffff811dd18b>] send_to_group+0xfb/0x180
[<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff81198859>] SyS_open+0x19/0x20
[<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
note: systemd-readahe[275] exited with preempt_count 1
BUG: scheduling while atomic: systemd-readahe/275/0x00000002
INFO: lockdep is turned off.
Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys
CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1
Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
ffff88007c293a00 ffff88007359f6f8 ffffffff8159702b ffff88007359f718
ffffffff810810f1 ffff88007c293a00 0000000000000001 ffff88007359f848
ffffffff8159789c ffff88007359f758 ffff88007359f768 ffff88007359e010
Call Trace:
[<ffffffff8159702b>] dump_stack+0x72/0x87
[<ffffffff810810f1>] __schedule_bug+0x61/0x80
[<ffffffff8159789c>] __schedule+0xbc/0x7c0
[<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0
[<ffffffff815980e4>] schedule+0x24/0x70
[<ffffffff81597205>] schedule_timeout+0x1c5/0x210
[<ffffffff8159914f>] ? wait_for_completion+0xcf/0x120
[<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10
[<ffffffff81599157>] wait_for_completion+0xd7/0x120
[<ffffffff81083330>] ? try_to_wake_up+0x250/0x250
[<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0
[<ffffffff810b965c>] __synchronize_srcu+0xec/0x130
[<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10
[<ffffffff810b96c8>] synchronize_srcu+0x18/0x20
[<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40
[<ffffffff811dfdf1>] inotify_release+0x21/0x50
[<ffffffff8119b2dd>] __fput+0xbd/0x2b0
[<ffffffff8119b569>] ____fput+0x9/0x10
[<ffffffff81070f41>] task_work_run+0xb1/0xe0
[<ffffffff81052979>] do_exit+0x1e9/0x4b0
[<ffffffff8159e23c>] oops_end+0xdc/0xe0
[<ffffffff81005f86>] die+0x56/0x90
[<ffffffff8159dea2>] do_general_protection+0x162/0x170
[<ffffffff8159d40c>] ? restore_args+0x30/0x30
[<ffffffff8159d592>] general_protection+0x22/0x30
[<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
[<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
[<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
[<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
[<ffffffff811b1442>] dput+0x22/0x130
[<ffffffff811a3d45>] path_put+0x15/0x30
[<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
[<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
[<ffffffff811dd18b>] send_to_group+0xfb/0x180
[<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[<ffffffff81198859>] SyS_open+0x19/0x20
[<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
ACPI: AC Adapter [AC] (on-line)
Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880037310690, len=64
Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk.
Prev obj: start=ffff880037310638, len=64
Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
Next obj: start=ffff8800373106e8, len=64
Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
--
Jiri Kosina
SUSE Labs
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-23 0:32 ` Dave Jones
@ 2014-01-23 15:05 ` Jan Kara
0 siblings, 0 replies; 19+ messages in thread
From: Jan Kara @ 2014-01-23 15:05 UTC (permalink / raw)
To: Dave Jones; +Cc: Linus Torvalds, Jan Kara, Linux Kernel, Jiri Kosina
On Wed 22-01-14 19:32:40, Dave Jones wrote:
> On Wed, Jan 22, 2014 at 04:08:52PM -0800, Linus Torvalds wrote:
> > On Wed, Jan 22, 2014 at 3:36 PM, Jan Kara <jack@suse.cz> wrote:
> > >
> > > But refcounting seems like an overkill for this - there is exactly one
> > > fanotify_response_event structure iff it is a permission event. So
> > > something like the (completely untested) attached patch should fix the
> > > problem. But I agree it's a bit ugly so we might want something different.
> > > I'll try to think about something better tomorrow.
> >
> > Ok, In the meantime, Dave, can you verify whether this hacky patch
> > fixes your problem?
>
> It actually seems worse. I see the tail end of what looks like a slab corruption
> trace, and then a total lockup. And of course none of this makes it over ttyUSB0
> because it happens so early. Grr.
Drat. Since this seems reasonably reproducible, I'll try to reproduce
and debug this (it seems systemd is using fanotify in a way that triggers
this). Thanks for testing.
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-23 10:23 ` Jiri Kosina
@ 2014-01-23 15:05 ` Jan Kara
2014-01-23 15:07 ` Jiri Kosina
0 siblings, 1 reply; 19+ messages in thread
From: Jan Kara @ 2014-01-23 15:05 UTC (permalink / raw)
To: Jiri Kosina; +Cc: Linus Torvalds, Jan Kara, Dave Jones, Linux Kernel
On Thu 23-01-14 11:23:53, Jiri Kosina wrote:
> On Wed, 22 Jan 2014, Linus Torvalds wrote:
>
> > > But refcounting seems like an overkill for this - there is exactly one
> > > fanotify_response_event structure iff it is a permission event. So
> > > something like the (completely untested) attached patch should fix the
> > > problem. But I agree it's a bit ugly so we might want something different.
> > > I'll try to think about something better tomorrow.
> >
> > Ok, In the meantime, Dave, can you verify whether this hacky patch
> > fixes your problem?
>
> I reported the same slab corruption yesterday as well here:
>
> https://lkml.org/lkml/2014/1/22/173
>
> With the patch applied, I am still seeing the slab corruption, preceeded
> by GPF (which is not there without the patch) in
> lockref_put_or_lock(&dentry->d_lockref) in dput():
Hmm, OK. Can you please send me your .config? I'll try to reproduce this
myself.
Honza
>
> general protection fault: 0000 [#1] SMP
> Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys
> CPU: 1 PID: 275 Comm: systemd-readahe Not tainted 3.13.0-03478-g670d0ac #1
> Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
> task: ffff880037c09150 ti: ffff88007359e000 task.ti: ffff88007359e000
> RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
> RSP: 0018:ffff88007359fc78 EFLAGS: 00010286
> RAX: ffff880037c09150 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000
> tpm_tis 00:09: 1.2 TPM (device-id 0x1020, rev-id 6)
> tpm_tis 00:09: Intel iTPM workaround enabled
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb
> RBP: ffff88007359fc98 R08: 0000000000000002 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb
> R13: ffff880037310690 R14: 0000000000000020 R15: ffff880036dbfc10
> FS: 00007fa953e4a700(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000
> CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007fff76a10258 CR3: 000000003659c000 CR4: 00000000000007e0
> Stack:
> 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880037310690
> ffff88007359fcb8 ffffffff8159c59c ffffffff812fe101 6b6b6b6b6b6b6beb
> ffff88007359fcd8 ffffffff812fe101 ffff88007359fd08 6b6b6b6b6b6b6b6b
> Call Trace:
> [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
> [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
> [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
> [<ffffffff811b1442>] dput+0x22/0x130
> [<ffffffff811a3d45>] path_put+0x15/0x30
> [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
> [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
> [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
> [<ffffffff811dd18b>] send_to_group+0xfb/0x180
> [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
> [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
> [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
> [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
> [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> [<ffffffff81198859>] SyS_open+0x19/0x20
> [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
> Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff
> RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
> RSP <ffff88007359fc78>
> ---[ end trace 7a918209ee213d28 ]---
> BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20
> in_atomic(): 1, irqs_disabled(): 0, pid: 275, name: systemd-readahe
> INFO: lockdep is turned off.
> CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1
> Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
> ffff880037c09150 ffff88007359fa78 ffffffff8159702b ffff88007359fa98
> ffffffff8107f621 ffffffff81a3d000 ffff880037b98d90 ffff88007359fab8
> ffffffff8159b4ff ffff880037c09150 ffff880037c09150 ffff88007359fae8
> Call Trace:
> [<ffffffff8159702b>] dump_stack+0x72/0x87
> [<ffffffff8107f621>] __might_sleep+0xe1/0x100
> [<ffffffff8159b4ff>] down_read+0x1f/0x60
> [<ffffffff810627ff>] exit_signals+0x1f/0x140
> [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20
> [<ffffffff81052844>] do_exit+0xb4/0x4b0
> [<ffffffff8159e23c>] oops_end+0xdc/0xe0
> [<ffffffff81005f86>] die+0x56/0x90
> [<ffffffff8159dea2>] do_general_protection+0x162/0x170
> [<ffffffff8159d40c>] ? restore_args+0x30/0x30
> [<ffffffff8159d592>] general_protection+0x22/0x30
> [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
> [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
> [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
> [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
> [<ffffffff811b1442>] dput+0x22/0x130
> [<ffffffff811a3d45>] path_put+0x15/0x30
> [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
> [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
> [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
> [<ffffffff811dd18b>] send_to_group+0xfb/0x180
> [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
> [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
> [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
> [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
> [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> [<ffffffff81198859>] SyS_open+0x19/0x20
> [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
> note: systemd-readahe[275] exited with preempt_count 1
> BUG: scheduling while atomic: systemd-readahe/275/0x00000002
> INFO: lockdep is turned off.
> Modules linked in: tpm_tis(+) tpm wmi acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm i2c_algo_bit button usbcore video usb_common edd fan processor ata_generic thermal thermal_sys
> CPU: 1 PID: 275 Comm: systemd-readahe Tainted: G D 3.13.0-03478-g670d0ac #1
> Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
> ffff88007c293a00 ffff88007359f6f8 ffffffff8159702b ffff88007359f718
> ffffffff810810f1 ffff88007c293a00 0000000000000001 ffff88007359f848
> ffffffff8159789c ffff88007359f758 ffff88007359f768 ffff88007359e010
> Call Trace:
> [<ffffffff8159702b>] dump_stack+0x72/0x87
> [<ffffffff810810f1>] __schedule_bug+0x61/0x80
> [<ffffffff8159789c>] __schedule+0xbc/0x7c0
> [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0
> [<ffffffff815980e4>] schedule+0x24/0x70
> [<ffffffff81597205>] schedule_timeout+0x1c5/0x210
> [<ffffffff8159914f>] ? wait_for_completion+0xcf/0x120
> [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10
> [<ffffffff81599157>] wait_for_completion+0xd7/0x120
> [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250
> [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0
> [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130
> [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10
> [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20
> [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40
> [<ffffffff811dfdf1>] inotify_release+0x21/0x50
> [<ffffffff8119b2dd>] __fput+0xbd/0x2b0
> [<ffffffff8119b569>] ____fput+0x9/0x10
> [<ffffffff81070f41>] task_work_run+0xb1/0xe0
> [<ffffffff81052979>] do_exit+0x1e9/0x4b0
> [<ffffffff8159e23c>] oops_end+0xdc/0xe0
> [<ffffffff81005f86>] die+0x56/0x90
> [<ffffffff8159dea2>] do_general_protection+0x162/0x170
> [<ffffffff8159d40c>] ? restore_args+0x30/0x30
> [<ffffffff8159d592>] general_protection+0x22/0x30
> [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
> [<ffffffff8159c59c>] _raw_spin_lock+0x3c/0x50
> [<ffffffff812fe101>] ? lockref_put_or_lock+0x11/0x40
> [<ffffffff812fe101>] lockref_put_or_lock+0x11/0x40
> [<ffffffff811b1442>] dput+0x22/0x130
> [<ffffffff811a3d45>] path_put+0x15/0x30
> [<ffffffff811e0bc5>] fanotify_free_event+0x15/0x40
> [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
> [<ffffffff811e1041>] fanotify_handle_event+0x341/0x390
> [<ffffffff811dd18b>] send_to_group+0xfb/0x180
> [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
> [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
> [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
> [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
> [<ffffffff812fdd6e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
> [<ffffffff81198859>] SyS_open+0x19/0x20
> [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
> ACPI: AC Adapter [AC] (on-line)
> Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880037310690, len=64
> Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
> Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
> 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk.
> Prev obj: start=ffff880037310638, len=64
> Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
> Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
> 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> Next obj: start=ffff8800373106e8, len=64
> Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
> Last user: [<ffffffff811e0bdd>](fanotify_free_event+0x2d/0x40)
> 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
> 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
>
> --
> Jiri Kosina
> SUSE Labs
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-23 15:05 ` Jan Kara
@ 2014-01-23 15:07 ` Jiri Kosina
2014-01-23 23:55 ` Jan Kara
0 siblings, 1 reply; 19+ messages in thread
From: Jiri Kosina @ 2014-01-23 15:07 UTC (permalink / raw)
To: Jan Kara; +Cc: Linus Torvalds, Dave Jones, Linux Kernel
[-- Attachment #1: Type: TEXT/PLAIN, Size: 983 bytes --]
On Thu, 23 Jan 2014, Jan Kara wrote:
> > > > But refcounting seems like an overkill for this - there is exactly one
> > > > fanotify_response_event structure iff it is a permission event. So
> > > > something like the (completely untested) attached patch should fix the
> > > > problem. But I agree it's a bit ugly so we might want something different.
> > > > I'll try to think about something better tomorrow.
> > >
> > > Ok, In the meantime, Dave, can you verify whether this hacky patch
> > > fixes your problem?
> >
> > I reported the same slab corruption yesterday as well here:
> >
> > https://lkml.org/lkml/2014/1/22/173
> >
> > With the patch applied, I am still seeing the slab corruption, preceeded
> > by GPF (which is not there without the patch) in
> > lockref_put_or_lock(&dentry->d_lockref) in dput():
> Hmm, OK. Can you please send me your .config? I'll try to reproduce this
> myself.
Attached.
The userspace is systemd-based.
--
Jiri Kosina
SUSE Labs
[-- Attachment #2: Type: APPLICATION/x-gzip, Size: 22901 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-23 15:07 ` Jiri Kosina
@ 2014-01-23 23:55 ` Jan Kara
2014-01-24 7:26 ` Jiri Kosina
0 siblings, 1 reply; 19+ messages in thread
From: Jan Kara @ 2014-01-23 23:55 UTC (permalink / raw)
To: Jiri Kosina; +Cc: Jan Kara, Linus Torvalds, Dave Jones, Linux Kernel
On Thu 23-01-14 16:07:45, Jiri Kosina wrote:
> On Thu, 23 Jan 2014, Jan Kara wrote:
>
> > > > > But refcounting seems like an overkill for this - there is exactly one
> > > > > fanotify_response_event structure iff it is a permission event. So
> > > > > something like the (completely untested) attached patch should fix the
> > > > > problem. But I agree it's a bit ugly so we might want something different.
> > > > > I'll try to think about something better tomorrow.
> > > >
> > > > Ok, In the meantime, Dave, can you verify whether this hacky patch
> > > > fixes your problem?
> > >
> > > I reported the same slab corruption yesterday as well here:
> > >
> > > https://lkml.org/lkml/2014/1/22/173
> > >
> > > With the patch applied, I am still seeing the slab corruption, preceeded
> > > by GPF (which is not there without the patch) in
> > > lockref_put_or_lock(&dentry->d_lockref) in dput():
> > Hmm, OK. Can you please send me your .config? I'll try to reproduce this
> > myself.
>
> Attached.
>
> The userspace is systemd-based.
Strange. I've installed systemd system (openSUSE 13.1) and it boots with
the latest Linus' kernel just fine (and I have at least FANOTIFY and
SLAB debugging set the same way as you). But it was only a KVM guest. I'll
try tomorrow with a physical machine I guess.
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-23 23:55 ` Jan Kara
@ 2014-01-24 7:26 ` Jiri Kosina
2014-01-27 23:40 ` Jan Kara
0 siblings, 1 reply; 19+ messages in thread
From: Jiri Kosina @ 2014-01-24 7:26 UTC (permalink / raw)
To: Jan Kara; +Cc: Linus Torvalds, Dave Jones, Linux Kernel
On Fri, 24 Jan 2014, Jan Kara wrote:
> Strange. I've installed systemd system (openSUSE 13.1) and it boots
> with the latest Linus' kernel just fine (and I have at least FANOTIFY
> and SLAB debugging set the same way as you). But it was only a KVM
> guest. I'll try tomorrow with a physical machine I guess.
FWIW the system I am reliably able to reproduce this on is opensuse 12.3
with this systemd version:
Version : 195
Release : 13.18.1
--
Jiri Kosina
SUSE Labs
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-24 7:26 ` Jiri Kosina
@ 2014-01-27 23:40 ` Jan Kara
2014-01-28 6:10 ` Dave Jones
2014-01-28 10:53 ` Jiri Kosina
0 siblings, 2 replies; 19+ messages in thread
From: Jan Kara @ 2014-01-27 23:40 UTC (permalink / raw)
To: Jiri Kosina; +Cc: Jan Kara, Linus Torvalds, Dave Jones, Linux Kernel
[-- Attachment #1: Type: text/plain, Size: 1000 bytes --]
On Fri 24-01-14 08:26:45, Jiri Kosina wrote:
> On Fri, 24 Jan 2014, Jan Kara wrote:
>
> > Strange. I've installed systemd system (openSUSE 13.1) and it boots
> > with the latest Linus' kernel just fine (and I have at least FANOTIFY
> > and SLAB debugging set the same way as you). But it was only a KVM
> > guest. I'll try tomorrow with a physical machine I guess.
>
> FWIW the system I am reliably able to reproduce this on is opensuse 12.3
> with this systemd version:
>
> Version : 195
> Release : 13.18.1
Hum, still no luck with reproduction (either on physical machine or with
KVM). Anyway, I've looked at the code again and the previous patch had a
stupid bug (passing different pointer to fsnotify_destroy_event() than we
should have), plus also the merging function in fanotify was too
aggressive. Can you try the attached patch? It boots for me but that means
nothing since I cannot reproduce the issue... Thanks!
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
[-- Attachment #2: fanotify.diff --]
[-- Type: text/x-patch, Size: 1737 bytes --]
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index 58772623f02a..1b3dd9de8518 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -18,7 +18,7 @@ static bool should_merge(struct fsnotify_event *old_fsn,
#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
/* dont merge two permission events */
- if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) &&
+ if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) ||
(new_fsn->mask & FAN_ALL_PERM_EVENTS))
return false;
#endif
@@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group,
}
#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
- if (fsn_event->mask & FAN_ALL_PERM_EVENTS)
+ if (fsn_event->mask & FAN_ALL_PERM_EVENTS) {
ret = fanotify_get_response_from_access(group, event);
+ fsnotify_destroy_event(group, fsn_event);
+ }
#endif
return ret;
}
@@ -221,7 +223,8 @@ static void fanotify_free_event(struct fsnotify_event *fsn_event)
struct fanotify_event_info *event;
event = FANOTIFY_E(fsn_event);
- path_put(&event->path);
+ if (event->path.mnt)
+ path_put(&event->path);
put_pid(event->tgid);
kmem_cache_free(fanotify_event_cachep, event);
}
diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c
index 57d7c083cb4b..d493c72c71fd 100644
--- a/fs/notify/fanotify/fanotify_user.c
+++ b/fs/notify/fanotify/fanotify_user.c
@@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,
if (IS_ERR(kevent))
break;
ret = copy_event_to_user(group, kevent, buf);
- fsnotify_destroy_event(group, kevent);
+ if (!(kevent->mask & FAN_ALL_PERM_EVENTS))
+ fsnotify_destroy_event(group, kevent);
if (ret < 0)
break;
buf += ret;
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-27 23:40 ` Jan Kara
@ 2014-01-28 6:10 ` Dave Jones
2014-01-28 8:02 ` Jan Kara
2014-01-28 10:53 ` Jiri Kosina
1 sibling, 1 reply; 19+ messages in thread
From: Dave Jones @ 2014-01-28 6:10 UTC (permalink / raw)
To: Jan Kara; +Cc: Jiri Kosina, Linus Torvalds, Linux Kernel
On Tue, Jan 28, 2014 at 12:40:17AM +0100, Jan Kara wrote:
> On Fri 24-01-14 08:26:45, Jiri Kosina wrote:
> > On Fri, 24 Jan 2014, Jan Kara wrote:
> >
> > > Strange. I've installed systemd system (openSUSE 13.1) and it boots
> > > with the latest Linus' kernel just fine (and I have at least FANOTIFY
> > > and SLAB debugging set the same way as you). But it was only a KVM
> > > guest. I'll try tomorrow with a physical machine I guess.
> >
> > FWIW the system I am reliably able to reproduce this on is opensuse 12.3
> > with this systemd version:
> >
> > Version : 195
> > Release : 13.18.1
> Hum, still no luck with reproduction (either on physical machine or with
> KVM). Anyway, I've looked at the code again and the previous patch had a
> stupid bug (passing different pointer to fsnotify_destroy_event() than we
> should have), plus also the merging function in fanotify was too
> aggressive. Can you try the attached patch? It boots for me but that means
> nothing since I cannot reproduce the issue... Thanks!
still not good I'm afraid. I still see corruption very early on in boot
and now it panics and locks up too.
Again, this happens so early that I can't grab it over usb-serial.
I stuck an mdelay(10000) in the slub corruption detector, and managed
to grab a photo of the first trace.
Trace:
? preempt_schedule
lock_acquire
? lockref_put_or_lock
_raw_spin_lock
? lockref_put_or_lock
dput
path_put
fanotify_free_event
fsnotify_destroy_event
fanotify_handle_event
? mntput
? path_openat
? handle_mm_fault
send_to_group
? fsnotify
fsnotify
do_sys_open
sys_open
RIP: lock_acquire
2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction
R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.
I also notice you mention SLAB above, but I've been using SLUB. I don't
know if the choice of allocator makes a difference in reproducability.
It's also worth noting that I have lockdep enabled, which may be perturbing things
to some degree.
Dave
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-28 6:10 ` Dave Jones
@ 2014-01-28 8:02 ` Jan Kara
2014-01-28 11:07 ` Jiri Kosina
0 siblings, 1 reply; 19+ messages in thread
From: Jan Kara @ 2014-01-28 8:02 UTC (permalink / raw)
To: Dave Jones; +Cc: Jan Kara, Jiri Kosina, Linus Torvalds, Linux Kernel
[-- Attachment #1: Type: text/plain, Size: 3153 bytes --]
On Tue 28-01-14 01:10:37, Dave Jones wrote:
> On Tue, Jan 28, 2014 at 12:40:17AM +0100, Jan Kara wrote:
> > On Fri 24-01-14 08:26:45, Jiri Kosina wrote:
> > > On Fri, 24 Jan 2014, Jan Kara wrote:
> > >
> > > > Strange. I've installed systemd system (openSUSE 13.1) and it boots
> > > > with the latest Linus' kernel just fine (and I have at least FANOTIFY
> > > > and SLAB debugging set the same way as you). But it was only a KVM
> > > > guest. I'll try tomorrow with a physical machine I guess.
> > >
> > > FWIW the system I am reliably able to reproduce this on is opensuse 12.3
> > > with this systemd version:
> > >
> > > Version : 195
> > > Release : 13.18.1
> > Hum, still no luck with reproduction (either on physical machine or with
> > KVM). Anyway, I've looked at the code again and the previous patch had a
> > stupid bug (passing different pointer to fsnotify_destroy_event() than we
> > should have), plus also the merging function in fanotify was too
> > aggressive. Can you try the attached patch? It boots for me but that means
> > nothing since I cannot reproduce the issue... Thanks!
>
> still not good I'm afraid. I still see corruption very early on in boot
> and now it panics and locks up too.
Ew, thanks for testing.
> Again, this happens so early that I can't grab it over usb-serial.
> I stuck an mdelay(10000) in the slub corruption detector, and managed
> to grab a photo of the first trace.
>
> Trace:
> ? preempt_schedule
> lock_acquire
> ? lockref_put_or_lock
> _raw_spin_lock
> ? lockref_put_or_lock
> dput
> path_put
> fanotify_free_event
> fsnotify_destroy_event
> fanotify_handle_event
> ? mntput
> ? path_openat
> ? handle_mm_fault
> send_to_group
> ? fsnotify
> fsnotify
> do_sys_open
> sys_open
> RIP: lock_acquire
>
> 2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction
>
> R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.
Yup. But I'm somewhat puzzled by the trace. We crash when calling
fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code
has been called from do_sys_open() so the event was a 'FS_OPEN' which fails
the fsn_event->mask & FAN_ALL_PERM_EVENTS test.
Slapping my forehead, that's a really stupid bug. The event
fsnotify_add_notify_event() returns may be freed by the time we return
because we already dropped the notification mutex. And then fsn_event->mask
& FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches
with the poison pattern 0x6b6b6b6b. So yet another hacked up version of
fanotify fix is attached. And I have to seriously think about use counts
for fanotify version of that struct.
> I also notice you mention SLAB above, but I've been using SLUB. I don't
> know if the choice of allocator makes a difference in reproducability.
Jiri Kosina has SLAB so SLAB/SLUB apparently doesn't matter.
> It's also worth noting that I have lockdep enabled, which may be perturbing things
> to some degree.
And I've compiled my kernel with lockdep as well since Jiri has it in his
config. But no luck.
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
[-- Attachment #2: fanotify.diff --]
[-- Type: text/x-patch, Size: 1726 bytes --]
diff --git a/fs/notify/fanotify/fanotify.c b/fs/notify/fanotify/fanotify.c
index 58772623f02a..f80895fb5ca1 100644
--- a/fs/notify/fanotify/fanotify.c
+++ b/fs/notify/fanotify/fanotify.c
@@ -18,7 +18,7 @@ static bool should_merge(struct fsnotify_event *old_fsn,
#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
/* dont merge two permission events */
- if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) &&
+ if ((old_fsn->mask & FAN_ALL_PERM_EVENTS) ||
(new_fsn->mask & FAN_ALL_PERM_EVENTS))
return false;
#endif
@@ -201,8 +201,10 @@ static int fanotify_handle_event(struct fsnotify_group *group,
}
#ifdef CONFIG_FANOTIFY_ACCESS_PERMISSIONS
- if (fsn_event->mask & FAN_ALL_PERM_EVENTS)
+ if (mask & FAN_ALL_PERM_EVENTS) {
ret = fanotify_get_response_from_access(group, event);
+ fsnotify_destroy_event(group, fsn_event);
+ }
#endif
return ret;
}
@@ -221,7 +223,8 @@ static void fanotify_free_event(struct fsnotify_event *fsn_event)
struct fanotify_event_info *event;
event = FANOTIFY_E(fsn_event);
- path_put(&event->path);
+ if (event->path.mnt)
+ path_put(&event->path);
put_pid(event->tgid);
kmem_cache_free(fanotify_event_cachep, event);
}
diff --git a/fs/notify/fanotify/fanotify_user.c b/fs/notify/fanotify/fanotify_user.c
index 57d7c083cb4b..d493c72c71fd 100644
--- a/fs/notify/fanotify/fanotify_user.c
+++ b/fs/notify/fanotify/fanotify_user.c
@@ -319,7 +319,8 @@ static ssize_t fanotify_read(struct file *file, char __user *buf,
if (IS_ERR(kevent))
break;
ret = copy_event_to_user(group, kevent, buf);
- fsnotify_destroy_event(group, kevent);
+ if (!(kevent->mask & FAN_ALL_PERM_EVENTS))
+ fsnotify_destroy_event(group, kevent);
if (ret < 0)
break;
buf += ret;
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-27 23:40 ` Jan Kara
2014-01-28 6:10 ` Dave Jones
@ 2014-01-28 10:53 ` Jiri Kosina
1 sibling, 0 replies; 19+ messages in thread
From: Jiri Kosina @ 2014-01-28 10:53 UTC (permalink / raw)
To: Jan Kara; +Cc: Linus Torvalds, Dave Jones, Linux Kernel
On Tue, 28 Jan 2014, Jan Kara wrote:
> Hum, still no luck with reproduction (either on physical machine or with
> KVM). Anyway, I've looked at the code again and the previous patch had a
> stupid bug (passing different pointer to fsnotify_destroy_event() than we
> should have), plus also the merging function in fanotify was too
> aggressive. Can you try the attached patch? It boots for me but that means
> nothing since I cannot reproduce the issue... Thanks!
I am attaching dmesg with the patch applied; I've removed irrelevant
parts.
There is a GPF, followed by scheduling in atomic context, followed by slab
corruption, followed by another scheduling while atomic and leak of
preempt_count.
[ 0.000000] Initializing cgroup subsys cpuset
[ 5.081301] systemd-udevd[332]: starting version 195
[ 5.083694] random: nonblocking pool is initialized
[ 5.299400] systemd-journald[307]: Received SIGUSR1
[ 5.625120] general protection fault: 0000 [#1] SMP
[ 5.626464] Modules linked in: acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys
[ 5.628008] CPU: 0 PID: 302 Comm: systemd-readahe Not tainted 3.13.0-03478-gae75a37 #1
[ 5.628008] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
[ 5.628008] task: ffff8800364b04d0 ti: ffff8800734b8000 task.ti: ffff8800734b8000
[ 5.628008] RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
[ 5.628008] RSP: 0018:ffff8800734b9c68 EFLAGS: 00010282
[ 5.628008] RAX: ffff8800364b04d0 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000
[ 5.628008] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb
[ 5.628008] RBP: ffff8800734b9c88 R08: 0000000000000002 R09: 0000000000000000
[ 5.628008] R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb
[ 5.628008] R13: ffff880035d0db28 R14: 0000000000000020 R15: ffff880037fffd50
[ 5.628008] FS: 00007fd2f4728700(0000) GS:ffff88007c200000(0000) knlGS:0000000000000000
[ 5.628008] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 5.628008] CR2: 00007fa7dc98c000 CR3: 0000000037f3b000 CR4: 00000000000007f0
[ 5.628008] Stack:
[ 5.628008] 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880035d0db28
[ 5.628008] ffff8800734b9ca8 ffffffff8159c5ac ffffffff812fe111 6b6b6b6b6b6b6beb
[ 5.628008] ffff8800734b9cc8 ffffffff812fe111 ffff8800734b9cf8 6b6b6b6b6b6b6b6b
[ 5.628008] Call Trace:
[ 5.628008] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50
[ 5.628008] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40
[ 5.628008] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40
[ 5.628008] [<ffffffff811b1442>] dput+0x22/0x130
[ 5.628008] [<ffffffff811a3d45>] path_put+0x15/0x30
[ 5.628008] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40
[ 5.628008] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[ 5.628008] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390
[ 5.628008] [<ffffffff811a3d4d>] ? path_put+0x1d/0x30
[ 5.628008] [<ffffffff811dd18b>] send_to_group+0xfb/0x180
[ 5.628008] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[ 5.628008] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[ 5.628008] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[ 5.628008] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[ 5.628008] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 5.628008] [<ffffffff81198859>] SyS_open+0x19/0x20
[ 5.628008] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
[ 5.628008] Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff
[ 5.628008] RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
[ 5.628008] RSP <ffff8800734b9c68>
[ 5.683491] ---[ end trace 5b4e9ae52ab9b0f6 ]---
[ 5.685076] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20
[ 5.686578] in_atomic(): 1, irqs_disabled(): 0, pid: 302, name: systemd-readahe
[ 5.688058] INFO: lockdep is turned off.
[ 5.689503] CPU: 0 PID: 302 Comm: systemd-readahe Tainted: G D 3.13.0-03478-gae75a37 #1
[ 5.690966] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
[ 5.692464] ffff8800364b04d0 ffff8800734b9a68 ffffffff8159703b ffff8800734b9a88
[ 5.694027] ffffffff8107f621 ffffffff81a3d000 ffff88003641c750 ffff8800734b9aa8
[ 5.695555] ffffffff8159b50f ffff8800364b04d0 ffff8800364b04d0 ffff8800734b9ad8
[ 5.697111] Call Trace:
[ 5.698595] [<ffffffff8159703b>] dump_stack+0x72/0x87
[ 5.700108] [<ffffffff8107f621>] __might_sleep+0xe1/0x100
[ 5.701623] [<ffffffff8159b50f>] down_read+0x1f/0x60
[ 5.703133] [<ffffffff810627ff>] exit_signals+0x1f/0x140
[ 5.704661] [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20
[ 5.706105] [<ffffffff81052844>] do_exit+0xb4/0x4b0
[ 5.707470] [<ffffffff8159e23c>] oops_end+0xdc/0xe0
[ 5.708845] [<ffffffff81005f86>] die+0x56/0x90
[ 5.710229] [<ffffffff8159dea2>] do_general_protection+0x162/0x170
[ 5.711545] [<ffffffff8159d40c>] ? restore_args+0x30/0x30
[ 5.712883] [<ffffffff8159d592>] general_protection+0x22/0x30
[ 5.714213] [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
[ 5.715491] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50
[ 5.716781] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40
[ 5.718113] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40
[ 5.719390] [<ffffffff811b1442>] dput+0x22/0x130
[ 5.720681] [<ffffffff811a3d45>] path_put+0x15/0x30
[ 5.721984] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40
[ 5.723243] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[ 5.724507] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390
[ 5.725779] [<ffffffff811a3d4d>] ? path_put+0x1d/0x30
[ 5.727014] [<ffffffff811dd18b>] send_to_group+0xfb/0x180
[ 5.728258] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[ 5.729515] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[ 5.730734] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[ 5.731945] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[ 5.733169] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 5.734456] [<ffffffff81198859>] SyS_open+0x19/0x20
[ 5.735681] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
[ 5.736933] note: systemd-readahe[302] exited with preempt_count 1
[ 5.738378] BUG: scheduling while atomic: systemd-readahe/302/0x00000002
[ 5.739652] INFO: lockdep is turned off.
[ 5.740925] Modules linked in: acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys
[ 5.743781] CPU: 0 PID: 302 Comm: systemd-readahe Tainted: G D 3.13.0-03478-gae75a37 #1
[ 5.745231] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
[ 5.746708] ffff88007c213a00 ffff8800734b96e8 ffffffff8159703b ffff8800734b9708
[ 5.748190] ffffffff810810f1 ffff88007c213a00 0000000000000000 ffff8800734b9838
[ 5.749690] ffffffff815978ac ffff8800734b9748 ffff8800734b9758 ffff8800734b8010
[ 5.751161] Call Trace:
[ 5.752635] [<ffffffff8159703b>] dump_stack+0x72/0x87
[ 5.754126] [<ffffffff810810f1>] __schedule_bug+0x61/0x80
[ 5.755551] [<ffffffff815978ac>] __schedule+0xbc/0x7c0
[ 5.756925] [<ffffffff8105defc>] ? mod_timer+0x14c/0x1f0
[ 5.758302] [<ffffffff815980f4>] schedule+0x24/0x70
[ 5.759632] [<ffffffff81597215>] schedule_timeout+0x1c5/0x210
[ 5.760982] [<ffffffff8159915f>] ? wait_for_completion+0xcf/0x120
[ 5.762327] [<ffffffff810a0d8d>] ? trace_hardirqs_on+0xd/0x10
[ 5.763630] [<ffffffff81599167>] wait_for_completion+0xd7/0x120
[ 5.764935] [<ffffffff81083330>] ? try_to_wake_up+0x250/0x250
[ 5.766261] [<ffffffff810b93bf>] ? srcu_reschedule+0x4f/0xf0
[ 5.767521] [<ffffffff810b965c>] __synchronize_srcu+0xec/0x130
[ 5.768775] [<ffffffff810b96e0>] ? srcu_barrier+0x10/0x10
[ 5.770059] [<ffffffff810b96c8>] synchronize_srcu+0x18/0x20
[ 5.771302] [<ffffffff811ddbdd>] fsnotify_destroy_group+0x1d/0x40
[ 5.772550] [<ffffffff811dfdf1>] inotify_release+0x21/0x50
[ 5.773814] [<ffffffff8119b2dd>] __fput+0xbd/0x2b0
[ 5.775055] [<ffffffff8119b569>] ____fput+0x9/0x10
[ 5.776311] [<ffffffff81070f41>] task_work_run+0xb1/0xe0
[ 5.777577] [<ffffffff81052979>] do_exit+0x1e9/0x4b0
[ 5.778803] [<ffffffff8159e23c>] oops_end+0xdc/0xe0
[ 5.780027] [<ffffffff81005f86>] die+0x56/0x90
[ 5.781264] [<ffffffff8159dea2>] do_general_protection+0x162/0x170
[ 5.782472] [<ffffffff8159d40c>] ? restore_args+0x30/0x30
[ 5.783687] [<ffffffff8159d592>] general_protection+0x22/0x30
[ 5.784917] [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
[ 5.786160] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50
[ 5.787367] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40
[ 5.788606] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40
[ 5.789848] [<ffffffff811b1442>] dput+0x22/0x130
[ 5.791073] [<ffffffff811a3d45>] path_put+0x15/0x30
[ 5.792324] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40
[ 5.793589] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[ 5.794809] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390
[ 5.796040] [<ffffffff811a3d4d>] ? path_put+0x1d/0x30
[ 5.797283] [<ffffffff811dd18b>] send_to_group+0xfb/0x180
[ 5.798505] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[ 5.799732] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[ 5.800970] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[ 5.802219] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[ 5.803461] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 5.804729] [<ffffffff81198859>] SyS_open+0x19/0x20
[ 5.805970] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
[ ... snip ... ]
[ 5.968718] Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880035d2a798, len=64
[ 5.968923] hub 7-0:1.0: USB hub found
[ 5.971406] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[ 5.972756] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[ 5.974098] 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk.
[ 5.974837] hub 7-0:1.0: 6 ports detected
[ 5.976799] Prev obj: start=ffff880035d2a740, len=64
[ 5.978189] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[ 5.979526] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[ 5.980910] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 5.982307] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 5.983691] Next obj: start=ffff880035d2a7f0, len=64
[ 5.985070] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[ 5.986455] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[ 5.987835] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 5.989248] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ ... snip ... ]
[ 7.044083] usb 2-1: new full-speed USB device number 4 using uhci_hcd
[ 7.131735] general protection fault: 0000 [#2] SMP
[ 7.131842] Slab corruption (Tainted: G D W ): fanotify_event_info start=ffff880035da5320, len=64
[ 7.131844] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[ 7.131850] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[ 7.131853] 030: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 6b 6b 6b a5 kkkkkkkk....kkk.
[ 7.131854] Prev obj: start=ffff880035da52c8, len=64
[ 7.131855] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[ 7.131857] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[ 7.131859] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 7.131861] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 7.131862] Next obj: start=ffff880035da5378, len=64
[ 7.131863] Redzone: 0x9f911029d74e35b/0x9f911029d74e35b.
[ 7.131864] Last user: [<ffffffff811e0be4>](fanotify_free_event+0x34/0x40)
[ 7.131866] 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 7.131868] 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b kkkkkkkkkkkkkkkk
[ 7.135045] Modules linked in: cpufreq_conservative cpufreq_userspace snd_hda_codec_conexant cpufreq_powersave snd_hda_codec_generic snd_hda_intel snd_hda_codec snd_hwdep snd_pcm thinkpad_acpi kvm_intel snd_seq iTCO_wdt iTCO_vendor_support kvm iwldvm mac80211 snd_timer snd_seq_device btusb bluetooth iwlwifi sg cfg80211 e1000e snd ptp pcspkr lpc_ich i2c_i801 mfd_core rfkill pps_core ehci_pci wmi soundcore battery ac tpm_tis tpm acpi_cpufreq autofs4 uhci_hcd ehci_hcd i915 drm_kms_helper drm usbcore i2c_algo_bit usb_common button video edd fan processor ata_generic thermal thermal_sys
[ 7.135045] CPU: 1 PID: 757 Comm: grep Tainted: G D W 3.13.0-03478-gae75a37 #1
[ 7.135045] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
[ 7.135045] task: ffff8800362fddd0 ti: ffff880036d42000 task.ti: ffff880036d42000
[ 7.135045] RIP: 0010:[<ffffffff810a51e7>] [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
[ 7.135045] RSP: 0018:ffff880036d43c68 EFLAGS: 00010282
[ 7.135045] RAX: ffff8800362fddd0 RBX: 6b6b6b6b6b6b6beb RCX: 0000000000000000
[ 7.135045] RDX: 0000000000000000 RSI: 0000000000000000 RDI: 6b6b6b6b6b6b6beb
[ 7.135045] RBP: ffff880036d43c88 R08: 0000000000000002 R09: 0000000000000000
[ 7.135045] R10: 0000000000000000 R11: 0000000000000000 R12: 6b6b6b6b6b6b6beb
[ 7.135045] R13: ffff880035d0db28 R14: 0000000000000020 R15: ffff880037900310
[ 7.135045] FS: 0000000000000000(0000) GS:ffff88007c280000(0000) knlGS:0000000000000000
[ 7.135045] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 7.135045] CR2: 00007fff296c9f54 CR3: 0000000079067000 CR4: 00000000000007e0
[ 7.135045] Stack:
[ 7.135045] 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb 6b6b6b6b6b6b6beb ffff880035d0db28
[ 7.135045] ffff880036d43ca8 ffffffff8159c5ac ffffffff812fe111 6b6b6b6b6b6b6beb
[ 7.135045] ffff880036d43cc8 ffffffff812fe111 ffff880036d43cf8 6b6b6b6b6b6b6b6b
[ 7.135045] Call Trace:
[ 7.135045] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50
[ 7.135045] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40
[ 7.135045] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40
[ 7.135045] [<ffffffff811b1442>] dput+0x22/0x130
[ 7.135045] [<ffffffff811a3d45>] path_put+0x15/0x30
[ 7.135045] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40
[ 7.135045] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[ 7.135045] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390
[ 7.135045] [<ffffffff815a0c84>] ? __do_page_fault+0x2c4/0x480
[ 7.135045] [<ffffffff811dd18b>] send_to_group+0xfb/0x180
[ 7.135045] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[ 7.135045] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[ 7.135045] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[ 7.135045] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[ 7.135045] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 7.135045] [<ffffffff81198859>] SyS_open+0x19/0x20
[ 7.135045] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
[ 7.135045] Code: 0d 7e 81 48 89 df e8 29 ff ff ff eb 94 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 20 48 89 5d e8 4c 89 65 f0 48 89 fb 4c 89 6d f8 <81> 7f 04 ad 4e ad de 74 0c 48 c7 c6 b5 0d 7e 81 e8 f4 fe ff ff
[ 7.135045] RIP [<ffffffff810a51e7>] do_raw_spin_lock+0x17/0x160
[ 7.135045] RSP <ffff880036d43c68>
[ 7.212496] ---[ end trace 5b4e9ae52ab9b0f7 ]---
[ 7.214048] BUG: sleeping function called from invalid context at kernel/locking/rwsem.c:20
[ 7.214049] in_atomic(): 1, irqs_disabled(): 0, pid: 757, name: grep
[ 7.214050] INFO: lockdep is turned off.
[ 7.214051] CPU: 1 PID: 757 Comm: grep Tainted: G D W 3.13.0-03478-gae75a37 #1
[ 7.214052] Hardware name: LENOVO 7470BN2/7470BN2, BIOS 6DET38WW (2.02 ) 12/19/2008
[ 7.214055] ffff8800362fddd0 ffff880036d43a68 ffffffff8159703b ffff880036d43a88
[ 7.214057] ffffffff8107f621 ffffffff81a3d000 ffff880037946cd0 ffff880036d43aa8
[ 7.214059] ffffffff8159b50f ffff8800362fddd0 ffff8800362fddd0 ffff880036d43ad8
[ 7.214060] Call Trace:
[ 7.214071] [<ffffffff8159703b>] dump_stack+0x72/0x87
[ 7.214074] [<ffffffff8107f621>] __might_sleep+0xe1/0x100
[ 7.214076] [<ffffffff8159b50f>] down_read+0x1f/0x60
[ 7.214079] [<ffffffff810627ff>] exit_signals+0x1f/0x140
[ 7.214083] [<ffffffff81079491>] ? blocking_notifier_call_chain+0x11/0x20
[ 7.214086] [<ffffffff81052844>] do_exit+0xb4/0x4b0
[ 7.214089] [<ffffffff8159e23c>] oops_end+0xdc/0xe0
[ 7.214092] [<ffffffff81005f86>] die+0x56/0x90
[ 7.214095] [<ffffffff8159dea2>] do_general_protection+0x162/0x170
[ 7.214097] [<ffffffff8159d40c>] ? restore_args+0x30/0x30
[ 7.214099] [<ffffffff8159d592>] general_protection+0x22/0x30
[ 7.214102] [<ffffffff810a51e7>] ? do_raw_spin_lock+0x17/0x160
[ 7.214104] [<ffffffff8159c5ac>] _raw_spin_lock+0x3c/0x50
[ 7.214107] [<ffffffff812fe111>] ? lockref_put_or_lock+0x11/0x40
[ 7.214109] [<ffffffff812fe111>] lockref_put_or_lock+0x11/0x40
[ 7.214113] [<ffffffff811b1442>] dput+0x22/0x130
[ 7.214115] [<ffffffff811a3d45>] path_put+0x15/0x30
[ 7.214117] [<ffffffff811e0bcc>] fanotify_free_event+0x1c/0x40
[ 7.214119] [<ffffffff811dd7ac>] fsnotify_destroy_event+0x1c/0x30
[ 7.214121] [<ffffffff811e1052>] fanotify_handle_event+0x342/0x390
[ 7.214124] [<ffffffff815a0c84>] ? __do_page_fault+0x2c4/0x480
[ 7.214127] [<ffffffff811dd18b>] send_to_group+0xfb/0x180
[ 7.214129] [<ffffffff811dd290>] ? fsnotify+0x80/0x2d0
[ 7.214131] [<ffffffff811ab325>] ? do_filp_open+0x45/0xa0
[ 7.214134] [<ffffffff811dd3d4>] fsnotify+0x1c4/0x2d0
[ 7.214136] [<ffffffff811987ad>] do_sys_open+0x1ad/0x220
[ 7.214139] [<ffffffff812fdd7e>] ? trace_hardirqs_on_thunk+0x3a/0x3f
[ 7.214141] [<ffffffff81198859>] SyS_open+0x19/0x20
[ 7.214143] [<ffffffff815a5222>] system_call_fastpath+0x16/0x1b
[ 7.214145] note: grep[757] exited with preempt_count 1
--
Jiri Kosina
SUSE Labs
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-28 8:02 ` Jan Kara
@ 2014-01-28 11:07 ` Jiri Kosina
2014-01-28 14:53 ` Jan Kara
0 siblings, 1 reply; 19+ messages in thread
From: Jiri Kosina @ 2014-01-28 11:07 UTC (permalink / raw)
To: Jan Kara; +Cc: Dave Jones, Linus Torvalds, Linux Kernel
On Tue, 28 Jan 2014, Jan Kara wrote:
> > 2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction
> >
> > R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.
> Yup. But I'm somewhat puzzled by the trace. We crash when calling
> fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code
> has been called from do_sys_open() so the event was a 'FS_OPEN' which fails
> the fsn_event->mask & FAN_ALL_PERM_EVENTS test.
>
> Slapping my forehead, that's a really stupid bug. The event
> fsnotify_add_notify_event() returns may be freed by the time we return
> because we already dropped the notification mutex. And then fsn_event->mask
> & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches
> with the poison pattern 0x6b6b6b6b. So yet another hacked up version of
> fanotify fix is attached. And I have to seriously think about use counts
> for fanotify version of that struct.
With the fixed version of the patch, all the fanotify-related oopses are
gone on my system.
--
Jiri Kosina
SUSE Labs
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-28 11:07 ` Jiri Kosina
@ 2014-01-28 14:53 ` Jan Kara
2014-01-28 15:24 ` Dave Jones
0 siblings, 1 reply; 19+ messages in thread
From: Jan Kara @ 2014-01-28 14:53 UTC (permalink / raw)
To: Jiri Kosina; +Cc: Jan Kara, Dave Jones, Linus Torvalds, Linux Kernel
On Tue 28-01-14 12:07:51, Jiri Kosina wrote:
> On Tue, 28 Jan 2014, Jan Kara wrote:
>
> > > 2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction
> > >
> > > R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.
> > Yup. But I'm somewhat puzzled by the trace. We crash when calling
> > fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code
> > has been called from do_sys_open() so the event was a 'FS_OPEN' which fails
> > the fsn_event->mask & FAN_ALL_PERM_EVENTS test.
> >
> > Slapping my forehead, that's a really stupid bug. The event
> > fsnotify_add_notify_event() returns may be freed by the time we return
> > because we already dropped the notification mutex. And then fsn_event->mask
> > & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches
> > with the poison pattern 0x6b6b6b6b. So yet another hacked up version of
> > fanotify fix is attached. And I have to seriously think about use counts
> > for fanotify version of that struct.
>
> With the fixed version of the patch, all the fanotify-related oopses are
> gone on my system.
Thanks for testing. So now I have to come up with something mergeable :)
Honza
--
Jan Kara <jack@suse.cz>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: fanotify use after free.
2014-01-28 14:53 ` Jan Kara
@ 2014-01-28 15:24 ` Dave Jones
0 siblings, 0 replies; 19+ messages in thread
From: Dave Jones @ 2014-01-28 15:24 UTC (permalink / raw)
To: Jan Kara; +Cc: Jiri Kosina, Linus Torvalds, Linux Kernel
On Tue, Jan 28, 2014 at 03:53:27PM +0100, Jan Kara wrote:
> On Tue 28-01-14 12:07:51, Jiri Kosina wrote:
> > On Tue, 28 Jan 2014, Jan Kara wrote:
> >
> > > > 2b:* 4d 8b 64 c6 08 mov 0x8(%r14,%rax,8),%r12 <-- trapping instruction
> > > >
> > > > R14 is 0x6b6b6b6b6b6b6c03, which looks like a use-after-free.
> > > Yup. But I'm somewhat puzzled by the trace. We crash when calling
> > > fsnotify_destroy_event() from fanotify_handle_event(). The fsnotify code
> > > has been called from do_sys_open() so the event was a 'FS_OPEN' which fails
> > > the fsn_event->mask & FAN_ALL_PERM_EVENTS test.
> > >
> > > Slapping my forehead, that's a really stupid bug. The event
> > > fsnotify_add_notify_event() returns may be freed by the time we return
> > > because we already dropped the notification mutex. And then fsn_event->mask
> > > & FAN_ALL_PERM_EVENTS test will pass because FAN_ALL_PERM_EVENTS matches
> > > with the poison pattern 0x6b6b6b6b. So yet another hacked up version of
> > > fanotify fix is attached. And I have to seriously think about use counts
> > > for fanotify version of that struct.
> >
> > With the fixed version of the patch, all the fanotify-related oopses are
> > gone on my system.
> Thanks for testing. So now I have to come up with something mergeable :)
Yep, looks good to me too. Thanks.
Dave
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2014-01-28 15:24 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2014-01-22 6:27 fanotify use after free Dave Jones
2014-01-22 16:43 ` Dave Jones
2014-01-22 18:20 ` Linus Torvalds
2014-01-22 23:36 ` Jan Kara
2014-01-23 0:08 ` Linus Torvalds
2014-01-23 0:32 ` Dave Jones
2014-01-23 15:05 ` Jan Kara
2014-01-23 10:23 ` Jiri Kosina
2014-01-23 15:05 ` Jan Kara
2014-01-23 15:07 ` Jiri Kosina
2014-01-23 23:55 ` Jan Kara
2014-01-24 7:26 ` Jiri Kosina
2014-01-27 23:40 ` Jan Kara
2014-01-28 6:10 ` Dave Jones
2014-01-28 8:02 ` Jan Kara
2014-01-28 11:07 ` Jiri Kosina
2014-01-28 14:53 ` Jan Kara
2014-01-28 15:24 ` Dave Jones
2014-01-28 10:53 ` Jiri Kosina
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).