All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH] fuse: clear FR_PENDING flag when moving requests out of pending queue
@ 2017-01-12 20:04 Tahsin Erdogan
  2017-01-13 11:13 ` Miklos Szeredi
  0 siblings, 1 reply; 6+ messages in thread
From: Tahsin Erdogan @ 2017-01-12 20:04 UTC (permalink / raw)
  To: Miklos Szeredi, linux-fsdevel; +Cc: linux-kernel, Tahsin Erdogan

fuse_abort_conn() moves requests from pending list to a temporary list
before canceling them. This operation races with request_wait_answer()
which also tries to remove the request after it gets a fatal signal. It
checks FR_PENDING flag to determine whether the request is still in the
pending list.

Make fuse_abort_conn() clear FR_PENDING flag so that request_wait_answer()
does not remove the request from temporary list.

This bug manifests itself as a panic that looks like this:

 general protection fault: 0000 [#1] SMP
 CPU: 2 PID: 1888 Comm: fusexmp Not tainted 4.9.0-rc8+ #47
 Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS Bochs
01/01/2011
 task: ffff88023616a100 task.stack: ffffc90001c20000
 RIP: 0010:[<ffffffff804d5a8f>]  [<ffffffff804d5a8f>]
end_requests+0x5f/0x90
 RSP: 0018:ffffc90001c23b78  EFLAGS: 00010246
 RAX: dead000000000200 RBX: 0000000000000000 RCX: 00000010e447a7df
 RDX: ffff8802331dc000 RSI: ffff8802331dc190 RDI: ffff8802336d7800
 RBP: ffffc90001c23b98 R08: 0000000000000000 R09: 0000000000000000
 R10: ffff880235c2b778 R11: ffff880235a22910 R12: ffff8802331dc190
 R13: ffffc90001c23bc8 R14: ffff8802336d7800 R15: ffff8802331166a0
 FS:  0000000000000000(0000) GS:ffff88023fd00000(0000)
knlGS:0000000000000000
 CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
 CR2: 0000000000b3c1c8 CR3: 0000000001008000 CR4: 00000000000006e0
 Stack:
  ffff8802336d7800 ffff8802336d7878 ffff8802331166a0 ffff8802336d7800
  ffffc90001c23c00 ffffffff804d77d8 ffff8802336d7d60 ffff8802336d7800
  ffffc90001c23bb8 ffffc90001c23bb8 ffffc90001c23bc8 ffffc90001c23bc8
 Call Trace:
  [<ffffffff804d77d8>] fuse_abort_conn+0x2a8/0x310
  [<ffffffff804d7899>] fuse_dev_release+0x59/0x90
  [<ffffffff80379c7d>] __fput+0x9d/0x1d0
  [<ffffffff80379de9>] ____fput+0x9/0x10
  [<ffffffff8028b51e>] task_work_run+0x7e/0xa0
  [<ffffffff8027510b>] do_exit+0x27b/0xa60
  [<ffffffff8027596a>] do_group_exit+0x3a/0xa0
  [<ffffffff8027f34a>] get_signal+0x1aa/0x5b0
  [<ffffffff802a8400>] ? __wake_up_common+0x80/0x80
  [<ffffffff8021a053>] do_signal+0x23/0x660
  [<ffffffff804d6f8f>] ? fuse_dev_read+0x4f/0x60
  [<ffffffff8026c738>] exit_to_usermode_loop+0x34/0x6b
  [<ffffffff80201565>] syscall_return_slowpath+0x55/0x60
  [<ffffffff80a2d79f>] entry_SYSCALL_64_fastpath+0x92/0x94
 Code: 84 24 9c 00 00 00 99 ff ff ff f0 41 80 64 24 30 7f f0 41 80 64 24
31 fe 49 8b 44 24 08 49 8b 14 24 4c 89 e6 4c 89 f7 48 89 42 08 <48> 89
10 4d 89 24 24 4d 89 64 24 08 e8 d0 fd ff ff 49 8b 45 00
 RIP  [<ffffffff804d5a8f>] end_requests+0x5f/0x90
  RSP <ffffc90001c23b78>
 ---[ end trace 7da3774b682d0b94 ]---

Signed-off-by: Tahsin Erdogan <tahsin@google.com>
---
 fs/fuse/dev.c | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/fs/fuse/dev.c b/fs/fuse/dev.c
index 70ea57c7b6bb..4e06a27ed7f8 100644
--- a/fs/fuse/dev.c
+++ b/fs/fuse/dev.c
@@ -2025,7 +2025,6 @@ static void end_requests(struct fuse_conn *fc, struct list_head *head)
 		struct fuse_req *req;
 		req = list_entry(head->next, struct fuse_req, list);
 		req->out.h.error = -ECONNABORTED;
-		clear_bit(FR_PENDING, &req->flags);
 		clear_bit(FR_SENT, &req->flags);
 		list_del_init(&req->list);
 		request_end(fc, req);
@@ -2103,6 +2102,8 @@ void fuse_abort_conn(struct fuse_conn *fc)
 		spin_lock(&fiq->waitq.lock);
 		fiq->connected = 0;
 		list_splice_init(&fiq->pending, &to_end2);
+		list_for_each_entry(req, &to_end2, list)
+			clear_bit(FR_PENDING, &req->flags);
 		while (forget_pending(fiq))
 			kfree(dequeue_forget(fiq, 1, NULL));
 		wake_up_all_locked(&fiq->waitq);
-- 
2.11.0.390.gc69c2f50cf-goog

^ permalink raw reply related	[flat|nested] 6+ messages in thread

* Re: [PATCH] fuse: clear FR_PENDING flag when moving requests out of pending queue
  2017-01-12 20:04 [PATCH] fuse: clear FR_PENDING flag when moving requests out of pending queue Tahsin Erdogan
@ 2017-01-13 11:13 ` Miklos Szeredi
  2017-01-16 17:57   ` Borislav Petkov
  0 siblings, 1 reply; 6+ messages in thread
From: Miklos Szeredi @ 2017-01-13 11:13 UTC (permalink / raw)
  To: Tahsin Erdogan; +Cc: linux-fsdevel, linux-kernel

On Thu, Jan 12, 2017 at 9:04 PM, Tahsin Erdogan <tahsin@google.com> wrote:
> fuse_abort_conn() moves requests from pending list to a temporary list
> before canceling them. This operation races with request_wait_answer()
> which also tries to remove the request after it gets a fatal signal. It
> checks FR_PENDING flag to determine whether the request is still in the
> pending list.
>
> Make fuse_abort_conn() clear FR_PENDING flag so that request_wait_answer()
> does not remove the request from temporary list.
>
> This bug manifests itself as a panic that looks like this:

Perfect patch; applied and pushed.

Thanks,
Miklos

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] fuse: clear FR_PENDING flag when moving requests out of pending queue
  2017-01-13 11:13 ` Miklos Szeredi
@ 2017-01-16 17:57   ` Borislav Petkov
  2017-01-27  9:35     ` Miklos Szeredi
  0 siblings, 1 reply; 6+ messages in thread
From: Borislav Petkov @ 2017-01-16 17:57 UTC (permalink / raw)
  To: Miklos Szeredi; +Cc: Tahsin Erdogan, linux-fsdevel, linux-kernel

Hi,

On Fri, Jan 13, 2017 at 12:13:04PM +0100, Miklos Szeredi wrote:
> On Thu, Jan 12, 2017 at 9:04 PM, Tahsin Erdogan <tahsin@google.com> wrote:
> > fuse_abort_conn() moves requests from pending list to a temporary list
> > before canceling them. This operation races with request_wait_answer()
> > which also tries to remove the request after it gets a fatal signal. It
> > checks FR_PENDING flag to determine whether the request is still in the
> > pending list.
> >
> > Make fuse_abort_conn() clear FR_PENDING flag so that request_wait_answer()
> > does not remove the request from temporary list.
> >
> > This bug manifests itself as a panic that looks like this:
> 
> Perfect patch; applied and pushed.

I'm seeing something similar here while the grub OS prober checks the
other partitions. It is not always reproducible, I saw it only twice so
far.

Related?

[  568.562174] ntfs: driver 2.1.32 [Flags: R/W MODULE].
[  568.575436] fuse init (API version 7.26)
[  568.707737] general protection fault: 0000 [#1] PREEMPT SMP
[  568.708509] Modules linked in: fuse ntfs msdos ext2 msr cpufreq_powersave cpufreq_userspace cpufreq_conservative binfmt_misc uinput vfat fat loop dm_crypt dm_mod hid_generic usbhid hid snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal coretemp kvm_intel arc4 kvm irqbypass iwldvm crc32_pclmul mac80211 crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate snd_hda_intel intel_rapl_perf snd_hda_codec serio_raw iwlwifi pcspkr snd_hwdep snd_hda_core sdhci_pci sg cfg80211 sdhci snd_pcm mmc_core i2c_i801 xhci_pci lpc_ich snd_timer ehci_pci thinkpad_acpi e1000e mfd_core xhci_hcd ehci_hcd nvram snd soundcore wmi thermal led_class battery ac
[  568.712473] CPU: 1 PID: 8817 Comm: grub-mount Not tainted 4.10.0-rc3+ #3
[  568.713283] Hardware name: LENOVO 2320CTO/2320CTO, BIOS G2ET86WW (2.06 ) 11/13/2012
[  568.714105] task: ffffa3b98f418000 task.stack: ffffbb9801834000
[  568.714944] RIP: 0010:memcpy_erms+0x6/0x10
[  568.715795] RSP: 0018:ffffbb9801837ce8 EFLAGS: 00010202
[  568.716942] RAX: ffff6731b2486010 RBX: ffffbb9801837e08 RCX: 0000000000000028
[  568.718098] RDX: 0000000000000028 RSI: ffffa3b95ee12578 RDI: ffff6731b2486010
[  568.719216] RBP: ffffbb9801837d18 R08: 0000000000000000 R09: 0000000000000000
[  568.720087] R10: 0000000000000001 R11: 0000000000000000 R12: ffffbb9801837d2c
[  568.720948] R13: 0000000000000028 R14: ffffbb9801837d30 R15: 0000000000000028
[  568.721814] FS:  00007f8fc5f85800(0000) GS:ffffa3b99d280000(0000) knlGS:0000000000000000
[  568.723023] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[  568.723920] CR2: 00007f8fc5592a10 CR3: 00000001deedd000 CR4: 00000000001406e0
[  568.725136] Call Trace:
[  568.726358]  ? fuse_copy_do+0xec/0x110 [fuse]
[  568.727279]  fuse_copy_one+0x53/0x70 [fuse]
[  568.728510]  fuse_dev_do_read.isra.29.constprop.34+0x478/0x630 [fuse]
[  568.729427]  ? filemap_map_pages+0x258/0x450
[  568.730344]  ? filemap_map_pages+0x5/0x450
[  568.731457]  fuse_dev_read+0x54/0x60 [fuse]
[  568.732697]  __vfs_read+0xbd/0x110
[  568.733932]  vfs_read+0x93/0x130
[  568.735138]  SyS_read+0x49/0xa0
[  568.736361]  entry_SYSCALL_64_fastpath+0x1c/0xb1
[  568.737270] RIP: 0033:0x7f8fc564c160
[  568.738498] RSP: 002b:00007ffc89d1c408 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  568.739583] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007f8fc564c160
[  568.740521] RDX: 0000000000021000 RSI: 00007f8fc5fac010 RDI: 0000000000000004
[  568.741802] RBP: 00007ffc89d1c560 R08: ffffffffffffffff R09: 0000000000000000
[  568.743083] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000c07910
[  568.744048] R13: 0000000000c07250 R14: 0000000000c07250 R15: 0000000000021000
[  568.745337] Code: e9 6d ff ff ff eb 1e 0f 1f 00 48 89 f8 48 89 d1 48 c1 e9 03 83 e2 07 f3 48 a5 89 d1 f3 a4 c3 66 0f 1f 44 00 00 48 89 f8 48 89 d1 <f3> a4 c3 0f 1f 80 00 00 00 00 48 89 f8 48 83 fa 20 72 7e 40 38 
[  568.747875] RIP: memcpy_erms+0x6/0x10 RSP: ffffbb9801837ce8
[  568.748957] ---[ end trace 6abe98048307269b ]---
[  568.749717] note: grub-mount[8817] exited with preempt_count 1
[  568.749934] ------------[ cut here ]------------
[  568.749938] WARNING: CPU: 1 PID: 8817 at fs/fuse/dev.c:2136 fuse_dev_release+0x71/0x90 [fuse]
[  568.749939] Modules linked in: fuse ntfs msdos ext2 msr cpufreq_powersave cpufreq_userspace cpufreq_conservative binfmt_misc uinput vfat fat loop dm_crypt dm_mod hid_generic usbhid hid snd_hda_codec_hdmi snd_hda_codec_realtek snd_hda_codec_generic iTCO_wdt iTCO_vendor_support x86_pkg_temp_thermal coretemp kvm_intel arc4 kvm irqbypass iwldvm crc32_pclmul mac80211 crc32c_intel ghash_clmulni_intel aesni_intel aes_x86_64 crypto_simd cryptd glue_helper intel_cstate snd_hda_intel intel_rapl_perf snd_hda_codec serio_raw iwlwifi pcspkr snd_hwdep snd_hda_core sdhci_pci sg cfg80211 sdhci snd_pcm mmc_core i2c_i801 xhci_pci lpc_ich snd_timer ehci_pci thinkpad_acpi e1000e mfd_core xhci_hcd ehci_hcd nvram snd soundcore wmi thermal led_class battery ac
[  568.749967] CPU: 1 PID: 8817 Comm: grub-mount Tainted: G      D         4.10.0-rc3+ #3
[  568.749968] Hardware name: LENOVO 2320CTO/2320CTO, BIOS G2ET86WW (2.06 ) 11/13/2012
[  568.749969] Call Trace:
[  568.749972]  dump_stack+0x67/0x92
[  568.749975]  __warn+0xcb/0xf0
[  568.749978]  warn_slowpath_null+0x1d/0x20
[  568.749981]  fuse_dev_release+0x71/0x90 [fuse]
[  568.749983]  __fput+0xd9/0x1e0
[  568.749985]  ____fput+0xe/0x10
[  568.749988]  task_work_run+0x7e/0xa0
[  568.749989]  do_exit+0x2d8/0xbd0
[  568.749991]  ? SyS_read+0x49/0xa0
[  568.749993]  rewind_stack_do_exit+0x17/0x20
[  568.749995] RIP: 0033:0x7f8fc564c160
[  568.749995] RSP: 002b:00007ffc89d1c408 EFLAGS: 00000246 ORIG_RAX: 0000000000000000
[  568.749997] RAX: ffffffffffffffda RBX: 0000000000000046 RCX: 00007f8fc564c160
[  568.749998] RDX: 0000000000021000 RSI: 00007f8fc5fac010 RDI: 0000000000000004
[  568.749999] RBP: 00007ffc89d1c560 R08: ffffffffffffffff R09: 0000000000000000
[  568.750000] R10: 0000000000000022 R11: 0000000000000246 R12: 0000000000c07910
[  568.750000] R13: 0000000000c07250 R14: 0000000000c07250 R15: 0000000000021000
[  568.750002] ---[ end trace 6abe98048307269c ]---

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] fuse: clear FR_PENDING flag when moving requests out of pending queue
  2017-01-16 17:57   ` Borislav Petkov
@ 2017-01-27  9:35     ` Miklos Szeredi
  2017-01-27  9:50       ` Borislav Petkov
  0 siblings, 1 reply; 6+ messages in thread
From: Miklos Szeredi @ 2017-01-27  9:35 UTC (permalink / raw)
  To: Borislav Petkov; +Cc: Tahsin Erdogan, linux-fsdevel, linux-kernel

On Mon, Jan 16, 2017 at 6:57 PM, Borislav Petkov <bp@alien8.de> wrote:
> Hi,
>
> On Fri, Jan 13, 2017 at 12:13:04PM +0100, Miklos Szeredi wrote:
>> On Thu, Jan 12, 2017 at 9:04 PM, Tahsin Erdogan <tahsin@google.com> wrote:
>> > fuse_abort_conn() moves requests from pending list to a temporary list
>> > before canceling them. This operation races with request_wait_answer()
>> > which also tries to remove the request after it gets a fatal signal. It
>> > checks FR_PENDING flag to determine whether the request is still in the
>> > pending list.
>> >
>> > Make fuse_abort_conn() clear FR_PENDING flag so that request_wait_answer()
>> > does not remove the request from temporary list.
>> >
>> > This bug manifests itself as a panic that looks like this:
>>
>> Perfect patch; applied and pushed.
>
> I'm seeing something similar here while the grub OS prober checks the
> other partitions. It is not always reproducible, I saw it only twice so
> far.
>
> Related?
>
> [  568.562174] ntfs: driver 2.1.32 [Flags: R/W MODULE].
> [  568.575436] fuse init (API version 7.26)
> [  568.707737] general protection fault: 0000 [#1] PREEMPT SMP


Doesn't look related.

"general protection fault" doesn't tell much about what happened.
Can't even tell which address was guilty.

Do you still have the vmlinux for this?  Or better yet, fs/fuse/dev.o?

Thanks,
Miklos

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] fuse: clear FR_PENDING flag when moving requests out of pending queue
  2017-01-27  9:35     ` Miklos Szeredi
@ 2017-01-27  9:50       ` Borislav Petkov
  2017-01-27  9:54         ` Miklos Szeredi
  0 siblings, 1 reply; 6+ messages in thread
From: Borislav Petkov @ 2017-01-27  9:50 UTC (permalink / raw)
  To: Miklos Szeredi; +Cc: Tahsin Erdogan, linux-fsdevel, linux-kernel

On Fri, Jan 27, 2017 at 10:35:26AM +0100, Miklos Szeredi wrote:
> Do you still have the vmlinux for this?  Or better yet, fs/fuse/dev.o?

Not really.

But I haven't been able to reproduce since and I was bisecting then. My
current guess is that perhaps there were some stale objects involved in
the build.

I'll ping you if I see it again.

Thanks.

-- 
Regards/Gruss,
    Boris.

Good mailing practices for 400: avoid top-posting and trim the reply.

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] fuse: clear FR_PENDING flag when moving requests out of pending queue
  2017-01-27  9:50       ` Borislav Petkov
@ 2017-01-27  9:54         ` Miklos Szeredi
  0 siblings, 0 replies; 6+ messages in thread
From: Miklos Szeredi @ 2017-01-27  9:54 UTC (permalink / raw)
  To: Borislav Petkov; +Cc: Tahsin Erdogan, linux-fsdevel, linux-kernel

On Fri, Jan 27, 2017 at 10:50 AM, Borislav Petkov <bp@alien8.de> wrote:
> On Fri, Jan 27, 2017 at 10:35:26AM +0100, Miklos Szeredi wrote:
>> Do you still have the vmlinux for this?  Or better yet, fs/fuse/dev.o?
>
> Not really.
>
> But I haven't been able to reproduce since and I was bisecting then. My
> current guess is that perhaps there were some stale objects involved in
> the build.
>
> I'll ping you if I see it again.

Ok.  Thanks.

Miklos

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2017-01-27  9:55 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-01-12 20:04 [PATCH] fuse: clear FR_PENDING flag when moving requests out of pending queue Tahsin Erdogan
2017-01-13 11:13 ` Miklos Szeredi
2017-01-16 17:57   ` Borislav Petkov
2017-01-27  9:35     ` Miklos Szeredi
2017-01-27  9:50       ` Borislav Petkov
2017-01-27  9:54         ` Miklos Szeredi

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.