* Crashes in next-20160915 (BUG at fs/notify/notification.c:66!)
@ 2016-09-16 2:20 Guenter Roeck
2016-09-16 6:45 ` Jan Kara
0 siblings, 1 reply; 8+ messages in thread
From: Guenter Roeck @ 2016-09-16 2:20 UTC (permalink / raw)
To: linux-kernel, Jan Kara, Andrew Morton
Hi,
I see various architectures crashing in -next with the following error.
------------[ cut here ]------------
kernel BUG at fs/notify/notification.c:66!
invalid opcode: 0000 [#1] PREEMPT
Modules linked in:
CPU: 0 PID: 110 Comm: udevd Not tainted 4.8.0-rc6-next-20160915-yocto-standard #1
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.9.1-0-gb3ef39f-prebuilt.qemu-project.org 04/01/2014
task: ffff88000dd58000 task.stack: ffffc90000250000
RIP: 0010:[<ffffffff811bb399>] [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10
RSP: 0018:ffffc90000253e68 EFLAGS: 00000296
RAX: ffff88000dd58000 RBX: ffff88000dc9a9c0 RCX: 0000000000011ff0
RDX: ffffffff811bd52a RSI: ffff88000e3c1258 RDI: ffff88000dc9a9c0
RBP: ffffc90000253e68 R08: ffff88000ffd37e0 R09: ffff88000dcdd600
R10: ffff88000dcdd600 R11: 0000000000000000 R12: 0000000000000001
R13: ffff88000dd28c00 R14: ffff88000dc0a380 R15: ffff88000dc0a398
FS: 00007ff5bea85740(0000) GS:ffffffff81c31000(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007ff5bea79000 CR3: 000000000dd63000 CR4: 00000000000406f0
Stack:
ffffc90000253e88 ffffffff811bd532 0000000000000000 ffff88000dd28c00
ffffc90000253f48 ffffffff811bfeba ffffffff00000006 0000000000000000
ffff88000dcdd600 ffff88000dcdd600 0000001900253f58 0000040000080002
Call Trace:
[<ffffffff811bd532>] inotify_poll+0x42/0x70
[<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60
[<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180
[<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f
Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48
RIP [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10
RSP <ffffc90000253e68>
---[ end trace 7dc4a27003f0b575 ]---
I didn't bisect, but I would guess the culprit is one of the new patches in the
affected file.
22e9cf146d3b fanotify: fix possible false warning when freeing events
ced89591817c fsnotify: convert notification_mutex to a spinlock
f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event
782fbc7e8685 fanotify: fix list corruption in fanotify_get_response()
56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown
Guenter
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!)
2016-09-16 2:20 Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Guenter Roeck
@ 2016-09-16 6:45 ` Jan Kara
2016-09-16 11:10 ` Guenter Roeck
0 siblings, 1 reply; 8+ messages in thread
From: Jan Kara @ 2016-09-16 6:45 UTC (permalink / raw)
To: Guenter Roeck; +Cc: linux-kernel, Jan Kara, Andrew Morton
Hi,
On Thu 15-09-16 19:20:10, Guenter Roeck wrote:
> I see various architectures crashing in -next with the following error.
>
> ------------[ cut here ]------------
> kernel BUG at fs/notify/notification.c:66!
Thanks for report!
...
> Call Trace:
> [<ffffffff811bd532>] inotify_poll+0x42/0x70
> [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60
> [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180
> [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f
> Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48
> RIP [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10
> RSP <ffffc90000253e68>
> ---[ end trace 7dc4a27003f0b575 ]---
>
> I didn't bisect, but I would guess the culprit is one of the new patches in the
> affected file.
>
> 22e9cf146d3b fanotify: fix possible false warning when freeing events
> ced89591817c fsnotify: convert notification_mutex to a spinlock
> f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event
> 782fbc7e8685 fanotify: fix list corruption in fanotify_get_response()
> 56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown
Yes, very likely my patches cause this. But it must be some config issue
because my test machine does not hit this and the code is "obviously
correct" - famous last words ;). I've seen zero-day robot complain as well
so I'll try the config it had sent me and see what's going on (likely
spin_is_locked() does not do what I thought under some configs...).
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!)
2016-09-16 6:45 ` Jan Kara
@ 2016-09-16 11:10 ` Guenter Roeck
2016-09-16 12:49 ` Jan Kara
0 siblings, 1 reply; 8+ messages in thread
From: Guenter Roeck @ 2016-09-16 11:10 UTC (permalink / raw)
To: Jan Kara; +Cc: linux-kernel, Andrew Morton
On 09/15/2016 11:45 PM, Jan Kara wrote:
> Hi,
>
> On Thu 15-09-16 19:20:10, Guenter Roeck wrote:
>> I see various architectures crashing in -next with the following error.
>>
>> ------------[ cut here ]------------
>> kernel BUG at fs/notify/notification.c:66!
>
> Thanks for report!
>
> ...
>
>> Call Trace:
>> [<ffffffff811bd532>] inotify_poll+0x42/0x70
>> [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60
>> [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180
>> [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f
>> Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48
>> RIP [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10
>> RSP <ffffc90000253e68>
>> ---[ end trace 7dc4a27003f0b575 ]---
>>
>> I didn't bisect, but I would guess the culprit is one of the new patches in the
>> affected file.
>>
>> 22e9cf146d3b fanotify: fix possible false warning when freeing events
>> ced89591817c fsnotify: convert notification_mutex to a spinlock
>> f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event
>> 782fbc7e8685 fanotify: fix list corruption in fanotify_get_response()
>> 56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown
>
> Yes, very likely my patches cause this. But it must be some config issue
> because my test machine does not hit this and the code is "obviously
> correct" - famous last words ;). I've seen zero-day robot complain as well
> so I'll try the config it had sent me and see what's going on (likely
> spin_is_locked() does not do what I thought under some configs...).
>
Your builds are probably all SMP builds. Looks like this happens if SMP is disabled.
Guenter
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!)
2016-09-16 11:10 ` Guenter Roeck
@ 2016-09-16 12:49 ` Jan Kara
2016-09-20 1:10 ` Michael Ellerman
0 siblings, 1 reply; 8+ messages in thread
From: Jan Kara @ 2016-09-16 12:49 UTC (permalink / raw)
To: Guenter Roeck; +Cc: Jan Kara, linux-kernel, Andrew Morton
On Fri 16-09-16 04:10:38, Guenter Roeck wrote:
> On 09/15/2016 11:45 PM, Jan Kara wrote:
> >Hi,
> >
> >On Thu 15-09-16 19:20:10, Guenter Roeck wrote:
> >>I see various architectures crashing in -next with the following error.
> >>
> >>------------[ cut here ]------------
> >>kernel BUG at fs/notify/notification.c:66!
> >
> >Thanks for report!
> >
> >...
> >
> >>Call Trace:
> >> [<ffffffff811bd532>] inotify_poll+0x42/0x70
> >> [<ffffffff811bfeba>] SyS_epoll_ctl+0x84a/0xf60
> >> [<ffffffff811be4a0>] ? ep_send_events_proc+0x180/0x180
> >> [<ffffffff8176be98>] entry_SYSCALL_64_fastpath+0x13/0x8f
> >>Code: 90 90 0f 1f 44 00 00 55 b8 01 00 00 00 48 89 e5 0f c1 05 bb e4 d4 00 83 c0 01 5d c3 66 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 <0f> 0b 0f 1f 44 00 00 0f 1f 44 00 00 55 48 89 e5 48 83 ec 10 48
> >>RIP [<ffffffff811bb399>] fsnotify_notify_queue_is_empty+0x9/0x10
> >> RSP <ffffc90000253e68>
> >>---[ end trace 7dc4a27003f0b575 ]---
> >>
> >>I didn't bisect, but I would guess the culprit is one of the new patches in the
> >>affected file.
> >>
> >>22e9cf146d3b fanotify: fix possible false warning when freeing events
> >>ced89591817c fsnotify: convert notification_mutex to a spinlock
> >>f82fa3d0e7f5 fsnotify: drop notification_mutex before destroying event
> >>782fbc7e8685 fanotify: fix list corruption in fanotify_get_response()
> >>56cf1c8a1b35 fsnotify: add a way to stop queueing events on group shutdown
> >
> >Yes, very likely my patches cause this. But it must be some config issue
> >because my test machine does not hit this and the code is "obviously
> >correct" - famous last words ;). I've seen zero-day robot complain as well
> >so I'll try the config it had sent me and see what's going on (likely
> >spin_is_locked() does not do what I thought under some configs...).
> >
>
> Your builds are probably all SMP builds. Looks like this happens if SMP
> is disabled.
Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG
returns always 0.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!)
2016-09-16 12:49 ` Jan Kara
@ 2016-09-20 1:10 ` Michael Ellerman
2016-09-20 1:13 ` Andrew Morton
0 siblings, 1 reply; 8+ messages in thread
From: Michael Ellerman @ 2016-09-20 1:10 UTC (permalink / raw)
To: Jan Kara, Guenter Roeck; +Cc: linux-kernel, Andrew Morton
Jan Kara <jack@suse.cz> writes:
> On Fri 16-09-16 04:10:38, Guenter Roeck wrote:
>> On 09/15/2016 11:45 PM, Jan Kara wrote:
>> >On Thu 15-09-16 19:20:10, Guenter Roeck wrote:
>> >>I see various architectures crashing in -next with the following error.
>> >>------------[ cut here ]------------
>> >>kernel BUG at fs/notify/notification.c:66!
>>
>> >correct" - famous last words ;). I've seen zero-day robot complain as well
>> >so I'll try the config it had sent me and see what's going on (likely
>> >spin_is_locked() does not do what I thought under some configs...).
>>
>> Your builds are probably all SMP builds. Looks like this happens if SMP
>> is disabled.
>
> Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG
> returns always 0.
Can we get this fixed soon please? It's breaking all my CI runs.
cheers
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!)
2016-09-20 1:10 ` Michael Ellerman
@ 2016-09-20 1:13 ` Andrew Morton
2016-09-20 12:05 ` Michael Ellerman
0 siblings, 1 reply; 8+ messages in thread
From: Andrew Morton @ 2016-09-20 1:13 UTC (permalink / raw)
To: Michael Ellerman; +Cc: Jan Kara, Guenter Roeck, linux-kernel
On Tue, 20 Sep 2016 11:10:35 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote:
> Jan Kara <jack@suse.cz> writes:
> > On Fri 16-09-16 04:10:38, Guenter Roeck wrote:
> >> On 09/15/2016 11:45 PM, Jan Kara wrote:
> >> >On Thu 15-09-16 19:20:10, Guenter Roeck wrote:
> >> >>I see various architectures crashing in -next with the following error.
> >> >>------------[ cut here ]------------
> >> >>kernel BUG at fs/notify/notification.c:66!
> >>
> >> >correct" - famous last words ;). I've seen zero-day robot complain as well
> >> >so I'll try the config it had sent me and see what's going on (likely
> >> >spin_is_locked() does not do what I thought under some configs...).
> >>
> >> Your builds are probably all SMP builds. Looks like this happens if SMP
> >> is disabled.
> >
> > Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG
> > returns always 0.
>
> Can we get this fixed soon please? It's breaking all my CI runs.
>
It should be fixed in next -next.
From: Jan Kara <jack@suse.cz>
Subject: fsnotify-convert-notification_mutex-to-a-spinlock-fix
This is a fixed version of the patch that fixes the BUG_ON hitting on UP
kernels.
Link: http://lkml.kernel.org/r/1474031567-1831-1-git-send-email-jack@suse.cz
Signed-off-by: Jan Kara <jack@suse.cz>
Reviewed-by: Lino Sanfilippo <LinoSanfilippo@gmx.de>
Tested-by: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
---
fs/notify/fanotify/fanotify_user.c | 3 ++-
fs/notify/notification.c | 9 ++++++---
2 files changed, 8 insertions(+), 4 deletions(-)
diff -puN fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/fanotify/fanotify_user.c
--- a/fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix
+++ a/fs/notify/fanotify/fanotify_user.c
@@ -54,7 +54,8 @@ struct kmem_cache *fanotify_perm_event_c
static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
size_t count)
{
- BUG_ON(!spin_is_locked(&group->notification_lock));
+ BUG_ON(IS_ENABLED(CONFIG_SMP) &&
+ !spin_is_locked(&group->notification_lock));
pr_debug("%s: group=%p count=%zd\n", __func__, group, count);
diff -puN fs/notify/notification.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/notification.c
--- a/fs/notify/notification.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix
+++ a/fs/notify/notification.c
@@ -63,7 +63,8 @@ EXPORT_SYMBOL_GPL(fsnotify_get_cookie);
/* return true if the notify queue is empty, false otherwise */
bool fsnotify_notify_queue_is_empty(struct fsnotify_group *group)
{
- BUG_ON(!spin_is_locked(&group->notification_lock));
+ BUG_ON(IS_ENABLED(CONFIG_SMP) &&
+ !spin_is_locked(&group->notification_lock));
return list_empty(&group->notification_list) ? true : false;
}
@@ -139,7 +140,8 @@ struct fsnotify_event *fsnotify_remove_f
{
struct fsnotify_event *event;
- BUG_ON(!spin_is_locked(&group->notification_lock));
+ BUG_ON(IS_ENABLED(CONFIG_SMP) &&
+ !spin_is_locked(&group->notification_lock));
pr_debug("%s: group=%p\n", __func__, group);
@@ -161,7 +163,8 @@ struct fsnotify_event *fsnotify_remove_f
*/
struct fsnotify_event *fsnotify_peek_first_event(struct fsnotify_group *group)
{
- BUG_ON(!spin_is_locked(&group->notification_lock));
+ BUG_ON(IS_ENABLED(CONFIG_SMP) &&
+ !spin_is_locked(&group->notification_lock));
return list_first_entry(&group->notification_list,
struct fsnotify_event, list);
_
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!)
2016-09-20 1:13 ` Andrew Morton
@ 2016-09-20 12:05 ` Michael Ellerman
2016-09-21 9:15 ` Jan Kara
0 siblings, 1 reply; 8+ messages in thread
From: Michael Ellerman @ 2016-09-20 12:05 UTC (permalink / raw)
To: Andrew Morton; +Cc: Jan Kara, Guenter Roeck, linux-kernel
Andrew Morton <akpm@linux-foundation.org> writes:
> On Tue, 20 Sep 2016 11:10:35 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote:
>> Jan Kara <jack@suse.cz> writes:
>> >
>> > Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG
>> > returns always 0.
>>
>> Can we get this fixed soon please? It's breaking all my CI runs.
>
> It should be fixed in next -next.
Great thanks.
I did search LKML to see if Jan had sent a fix but I guess I missed it.
> diff -puN fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/fanotify/fanotify_user.c
> --- a/fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix
> +++ a/fs/notify/fanotify/fanotify_user.c
> @@ -54,7 +54,8 @@ struct kmem_cache *fanotify_perm_event_c
> static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
> size_t count)
> {
> - BUG_ON(!spin_is_locked(&group->notification_lock));
> + BUG_ON(IS_ENABLED(CONFIG_SMP) &&
> + !spin_is_locked(&group->notification_lock));
I thought lockdep_assert_held() was preferred for checks like this that
are purely sanity checking, ie. not part of the algorithm.
cheers
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Crashes in next-20160915 (BUG at fs/notify/notification.c:66!)
2016-09-20 12:05 ` Michael Ellerman
@ 2016-09-21 9:15 ` Jan Kara
0 siblings, 0 replies; 8+ messages in thread
From: Jan Kara @ 2016-09-21 9:15 UTC (permalink / raw)
To: Michael Ellerman; +Cc: Andrew Morton, Jan Kara, Guenter Roeck, linux-kernel
On Tue 20-09-16 22:05:00, Michael Ellerman wrote:
> Andrew Morton <akpm@linux-foundation.org> writes:
> > On Tue, 20 Sep 2016 11:10:35 +1000 Michael Ellerman <mpe@ellerman.id.au> wrote:
> >> Jan Kara <jack@suse.cz> writes:
> >> >
> >> > Right, that was the problem. spin_is_locked() without CONFIG_SPINLOCK_DEBUG
> >> > returns always 0.
> >>
> >> Can we get this fixed soon please? It's breaking all my CI runs.
> >
> > It should be fixed in next -next.
>
> Great thanks.
>
> I did search LKML to see if Jan had sent a fix but I guess I missed it.
I didn't send the fix to LKML but only to linux-fsdevel...
> > diff -puN fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix fs/notify/fanotify/fanotify_user.c
> > --- a/fs/notify/fanotify/fanotify_user.c~fsnotify-convert-notification_mutex-to-a-spinlock-fix
> > +++ a/fs/notify/fanotify/fanotify_user.c
> > @@ -54,7 +54,8 @@ struct kmem_cache *fanotify_perm_event_c
> > static struct fsnotify_event *get_one_event(struct fsnotify_group *group,
> > size_t count)
> > {
> > - BUG_ON(!spin_is_locked(&group->notification_lock));
> > + BUG_ON(IS_ENABLED(CONFIG_SMP) &&
> > + !spin_is_locked(&group->notification_lock));
>
> I thought lockdep_assert_held() was preferred for checks like this that
> are purely sanity checking, ie. not part of the algorithm.
As pointed out by other guy, assert_spin_locked() is probably the right way
to do it (it's a wrapper that does essentially what I did here). Somehow I
missed that when looking for the right fix.
Honza
--
Jan Kara <jack@suse.com>
SUSE Labs, CR
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2016-09-21 9:15 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-09-16 2:20 Crashes in next-20160915 (BUG at fs/notify/notification.c:66!) Guenter Roeck
2016-09-16 6:45 ` Jan Kara
2016-09-16 11:10 ` Guenter Roeck
2016-09-16 12:49 ` Jan Kara
2016-09-20 1:10 ` Michael Ellerman
2016-09-20 1:13 ` Andrew Morton
2016-09-20 12:05 ` Michael Ellerman
2016-09-21 9:15 ` Jan Kara
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.