Stable Archive on lore.kernel.org
 help / color / Atom feed
From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Lei Xue <carmark.dlut@gmail.com>,
	Dave Wysochanski <dwysocha@redhat.com>,
	David Howells <dhowells@redhat.com>,
	Sasha Levin <sashal@kernel.org>,
	linux-cachefs@redhat.com
Subject: [PATCH AUTOSEL 5.6 21/41] cachefiles: Fix race between read_waiter and read_copier involving op->to_do
Date: Fri, 22 May 2020 10:49:38 -0400
Message-ID: <20200522144959.434379-21-sashal@kernel.org> (raw)
In-Reply-To: <20200522144959.434379-1-sashal@kernel.org>

From: Lei Xue <carmark.dlut@gmail.com>

[ Upstream commit 7bb0c5338436dae953622470d52689265867f032 ]

There is a potential race in fscache operation enqueuing for reading and
copying multiple pages from cachefiles to netfs.  The problem can be seen
easily on a heavy loaded system (for example many processes reading files
continually on an NFS share covered by fscache triggered this problem within
a few minutes).

The race is due to cachefiles_read_waiter() adding the op to the monitor
to_do list and then then drop the object->work_lock spinlock before
completing fscache_enqueue_operation().  Once the lock is dropped,
cachefiles_read_copier() grabs the op, completes processing it, and
makes it through fscache_retrieval_complete() which sets the op->state to
the final state of FSCACHE_OP_ST_COMPLETE(4).  When cachefiles_read_waiter()
finally gets through the remainder of fscache_enqueue_operation()
it sees the invalid state, and hits the ASSERTCMP and the following
oops is seen:
[ 2259.612361] FS-Cache:
[ 2259.614785] FS-Cache: Assertion failed
[ 2259.618639] FS-Cache: 4 == 5 is false
[ 2259.622456] ------------[ cut here ]------------
[ 2259.627190] kernel BUG at fs/fscache/operation.c:70!
...
[ 2259.791675] RIP: 0010:[<ffffffffc061b4cf>]  [<ffffffffc061b4cf>] fscache_enqueue_operation+0xff/0x170 [fscache]
[ 2259.802059] RSP: 0000:ffffa0263d543be0  EFLAGS: 00010046
[ 2259.807521] RAX: 0000000000000019 RBX: ffffa01a4d390480 RCX: 0000000000000006
[ 2259.814847] RDX: 0000000000000000 RSI: 0000000000000046 RDI: ffffa0263d553890
[ 2259.822176] RBP: ffffa0263d543be8 R08: 0000000000000000 R09: ffffa0263c2d8708
[ 2259.829502] R10: 0000000000001e7f R11: 0000000000000000 R12: ffffa01a4d390480
[ 2259.844483] R13: ffff9fa9546c5920 R14: ffffa0263d543c80 R15: ffffa0293ff9bf10
[ 2259.859554] FS:  00007f4b6efbd700(0000) GS:ffffa0263d540000(0000) knlGS:0000000000000000
[ 2259.875571] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[ 2259.889117] CR2: 00007f49e1624ff0 CR3: 0000012b38b38000 CR4: 00000000007607e0
[ 2259.904015] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[ 2259.918764] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
[ 2259.933449] PKRU: 55555554
[ 2259.943654] Call Trace:
[ 2259.953592]  <IRQ>
[ 2259.955577]  [<ffffffffc03a7c12>] cachefiles_read_waiter+0x92/0xf0 [cachefiles]
[ 2259.978039]  [<ffffffffa34d3942>] __wake_up_common+0x82/0x120
[ 2259.991392]  [<ffffffffa34d3a63>] __wake_up_common_lock+0x83/0xc0
[ 2260.004930]  [<ffffffffa34d3510>] ? task_rq_unlock+0x20/0x20
[ 2260.017863]  [<ffffffffa34d3ab3>] __wake_up+0x13/0x20
[ 2260.030230]  [<ffffffffa34c72a0>] __wake_up_bit+0x50/0x70
[ 2260.042535]  [<ffffffffa35bdcdb>] unlock_page+0x2b/0x30
[ 2260.054495]  [<ffffffffa35bdd09>] page_endio+0x29/0x90
[ 2260.066184]  [<ffffffffa368fc81>] mpage_end_io+0x51/0x80

CPU1
cachefiles_read_waiter()
 20 static int cachefiles_read_waiter(wait_queue_entry_t *wait, unsigned mode,
 21                                   int sync, void *_key)
 22 {
...
 61         spin_lock(&object->work_lock);
 62         list_add_tail(&monitor->op_link, &op->to_do);
 63         spin_unlock(&object->work_lock);
<begin race window>
 64
 65         fscache_enqueue_retrieval(op);
182 static inline void fscache_enqueue_retrieval(struct fscache_retrieval *op)
183 {
184         fscache_enqueue_operation(&op->op);
185 }
 58 void fscache_enqueue_operation(struct fscache_operation *op)
 59 {
 60         struct fscache_cookie *cookie = op->object->cookie;
 61
 62         _enter("{OBJ%x OP%x,%u}",
 63                op->object->debug_id, op->debug_id, atomic_read(&op->usage));
 64
 65         ASSERT(list_empty(&op->pend_link));
 66         ASSERT(op->processor != NULL);
 67         ASSERT(fscache_object_is_available(op->object));
 68         ASSERTCMP(atomic_read(&op->usage), >, 0);
<end race window>

CPU2
cachefiles_read_copier()
168         while (!list_empty(&op->to_do)) {
...
202                 fscache_end_io(op, monitor->netfs_page, error);
203                 put_page(monitor->netfs_page);
204                 fscache_retrieval_complete(op, 1);

CPU1
 58 void fscache_enqueue_operation(struct fscache_operation *op)
 59 {
...
 69         ASSERTIFCMP(op->state != FSCACHE_OP_ST_IN_PROGRESS,
 70                     op->state, ==,  FSCACHE_OP_ST_CANCELLED);

Signed-off-by: Lei Xue <carmark.dlut@gmail.com>
Signed-off-by: Dave Wysochanski <dwysocha@redhat.com>
Signed-off-by: David Howells <dhowells@redhat.com>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 fs/cachefiles/rdwr.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/fs/cachefiles/rdwr.c b/fs/cachefiles/rdwr.c
index d3d78176b23c..e7726f5f1241 100644
--- a/fs/cachefiles/rdwr.c
+++ b/fs/cachefiles/rdwr.c
@@ -60,9 +60,9 @@ static int cachefiles_read_waiter(wait_queue_entry_t *wait, unsigned mode,
 	object = container_of(op->op.object, struct cachefiles_object, fscache);
 	spin_lock(&object->work_lock);
 	list_add_tail(&monitor->op_link, &op->to_do);
+	fscache_enqueue_retrieval(op);
 	spin_unlock(&object->work_lock);
 
-	fscache_enqueue_retrieval(op);
 	fscache_put_retrieval(op);
 	return 0;
 }
-- 
2.25.1


  parent reply index

Thread overview: 41+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-05-22 14:49 [PATCH AUTOSEL 5.6 01/41] arm64: dts: qcom: db820c: fix audio configuration Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 02/41] ARM: dts: rockchip: fix phy nodename for rk3228-evb Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 03/41] ARM: dts: rockchip: fix phy nodename for rk3229-xms6 Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 04/41] arm64: dts: rockchip: fix status for &gmac2phy in rk3328-evb.dts Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 05/41] arm64: dts: rockchip: swap interrupts interrupt-names rk3399 gpu node Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 06/41] ARM: dts: rockchip: swap clock-names of gpu nodes Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 07/41] ARM: dts: rockchip: fix pinctrl sub nodename for spi in rk322x.dtsi Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 08/41] gpio: tegra: mask GPIO IRQs during IRQ shutdown Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 09/41] ALSA: usb-audio: add mapping for ASRock TRX40 Creator Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 10/41] ARM: dts: omap4-droid4: Fix flakey wlan by disabling internal pull for gpio Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 11/41] ARM: dts: omap4-droid4: Fix occasional lost wakeirq for uart1 Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 12/41] net: microchip: encx24j600: add missed kthread_stop Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 13/41] kselftests: dmabuf-heaps: Fix confused return value on expected error testing Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 14/41] gfs2: move privileged user check to gfs2_quota_lock_check Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 15/41] gfs2: don't call quota_unhold if quotas are not locked Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 16/41] gfs2: Grab glock reference sooner in gfs2_add_revoke Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 17/41] drm/amdgpu: drop unnecessary cancel_delayed_work_sync on PG ungate Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 18/41] drm/amd/powerplay: perform PG ungate prior to CG ungate Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 19/41] drm/amdgpu: Use GEM obj reference for KFD BOs Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 20/41] Revert "gfs2: Don't demote a glock until its revokes are written" Sasha Levin
2020-05-22 14:49 ` Sasha Levin [this message]
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 22/41] usb: dwc3: pci: Enable extcon driver for Intel Merrifield Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 23/41] usb: phy: twl6030-usb: Fix a resource leak in an error handling path in 'twl6030_usb_probe()' Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 24/41] usb: gadget: legacy: fix redundant initialization warnings Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 25/41] net: freescale: select CONFIG_FIXED_PHY where needed Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 26/41] drm/amd/amdgpu: Update update_config() logic Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 27/41] IB/i40iw: Remove bogus call to netdev_master_upper_dev_get() Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 28/41] riscv: Fix unmet direct dependencies built based on SOC_VIRT Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 29/41] riscv: stacktrace: Fix undefined reference to `walk_stackframe' Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 30/41] riscv: Add pgprot_writecombine/device and PAGE_SHARED defination if NOMMU Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 31/41] clk: ti: am33xx: fix RTC clock parent Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 32/41] csky: Fixup msa highest 3 bits mask Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 33/41] csky: Fixup perf callchain unwind Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 34/41] csky: Fixup remove duplicate irq_disable Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 35/41] drm/meson: pm resume add return errno branch Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 36/41] hwmon: (nct7904) Fix incorrect range of temperature limit registers Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 37/41] riscv: pgtable: Fix __kernel_map_pages build error if NOMMU Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 38/41] cifs: Fix null pointer check in cifs_read Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 39/41] csky: Fixup raw_copy_from_user() Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 40/41] samples: bpf: Fix build error Sasha Levin
2020-05-22 14:49 ` [PATCH AUTOSEL 5.6 41/41] drivers: net: hamradio: Fix suspicious RCU usage warning in bpqether.c Sasha Levin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200522144959.434379-21-sashal@kernel.org \
    --to=sashal@kernel.org \
    --cc=carmark.dlut@gmail.com \
    --cc=dhowells@redhat.com \
    --cc=dwysocha@redhat.com \
    --cc=linux-cachefs@redhat.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Stable Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/stable/0 stable/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 stable stable/ https://lore.kernel.org/stable \
		stable@vger.kernel.org
	public-inbox-index stable

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.stable


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git