From: Sasha Levin <sashal@kernel.org>
To: linux-kernel@vger.kernel.org, stable@vger.kernel.org
Cc: Josef Bacik <josef@toxicpanda.com>,
Mike Christie <mchristi@redhat.com>, Jens Axboe <axboe@kernel.dk>,
Sasha Levin <sashal@kernel.org>,
linux-block@vger.kernel.org, nbd@other.debian.org
Subject: [PATCH AUTOSEL 4.14 23/24] nbd: handle racing with error'ed out commands
Date: Wed, 30 Oct 2019 11:55:54 -0400 [thread overview]
Message-ID: <20191030155555.10494-23-sashal@kernel.org> (raw)
In-Reply-To: <20191030155555.10494-1-sashal@kernel.org>
From: Josef Bacik <josef@toxicpanda.com>
[ Upstream commit 7ce23e8e0a9cd38338fc8316ac5772666b565ca9 ]
We hit the following warning in production
print_req_error: I/O error, dev nbd0, sector 7213934408 flags 80700
------------[ cut here ]------------
refcount_t: underflow; use-after-free.
WARNING: CPU: 25 PID: 32407 at lib/refcount.c:190 refcount_sub_and_test_checked+0x53/0x60
Workqueue: knbd-recv recv_work [nbd]
RIP: 0010:refcount_sub_and_test_checked+0x53/0x60
Call Trace:
blk_mq_free_request+0xb7/0xf0
blk_mq_complete_request+0x62/0xf0
recv_work+0x29/0xa1 [nbd]
process_one_work+0x1f5/0x3f0
worker_thread+0x2d/0x3d0
? rescuer_thread+0x340/0x340
kthread+0x111/0x130
? kthread_create_on_node+0x60/0x60
ret_from_fork+0x1f/0x30
---[ end trace b079c3c67f98bb7c ]---
This was preceded by us timing out everything and shutting down the
sockets for the device. The problem is we had a request in the queue at
the same time, so we completed the request twice. This can actually
happen in a lot of cases, we fail to get a ref on our config, we only
have one connection and just error out the command, etc.
Fix this by checking cmd->status in nbd_read_stat. We only change this
under the cmd->lock, so we are safe to check this here and see if we've
already error'ed this command out, which would indicate that we've
completed it as well.
Reviewed-by: Mike Christie <mchristi@redhat.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>
Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
drivers/block/nbd.c | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index a234600849558..f322bb3286910 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -648,6 +648,12 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index)
ret = -ENOENT;
goto out;
}
+ if (cmd->status != BLK_STS_OK) {
+ dev_err(disk_to_dev(nbd->disk), "Command already handled %p\n",
+ req);
+ ret = -ENOENT;
+ goto out;
+ }
if (test_bit(NBD_CMD_REQUEUED, &cmd->flags)) {
dev_err(disk_to_dev(nbd->disk), "Raced with timeout on req %p\n",
req);
--
2.20.1
next prev parent reply other threads:[~2019-10-30 15:57 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-30 15:55 [PATCH AUTOSEL 4.14 01/24] arm64: dts: Fix gpio to pinmux mapping Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 02/24] regulator: ti-abb: Fix timeout in ti_abb_wait_txdone/ti_abb_clear_all_txdone Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 03/24] regulator: pfuze100-regulator: Variable "val" in pfuze100_regulator_probe() could be uninitialized Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 04/24] ASoC: wm_adsp: Don't generate kcontrols without READ flags Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 05/24] ASoc: rockchip: i2s: Fix RPM imbalance Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 06/24] ARM: dts: logicpd-torpedo-som: Remove twl_keypad Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 07/24] pinctrl: ns2: Fix off by one bugs in ns2_pinmux_enable() Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 08/24] ARM: mm: fix alignment handler faults under memory pressure Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 09/24] scsi: scsi_dh_alua: handle RTPG sense code correctly during state transitions Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 10/24] scsi: sni_53c710: fix compilation error Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 11/24] scsi: fix kconfig dependency warning related to 53C700_LE_ON_BE Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 12/24] ARM: dts: imx7s: Correct GPT's ipg clock source Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 13/24] perf c2c: Fix memory leak in build_cl_output() Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 14/24] USB: legousbtower: fix a signedness bug in tower_probe() Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 15/24] perf kmem: Fix memory leak in compact_gfp_flags() Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 16/24] ARM: davinci: dm365: Fix McBSP dma_slave_map entry Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 17/24] scsi: target: core: Do not overwrite CDB byte 1 Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 18/24] ARM: 8926/1: v7m: remove register save to stack before svc Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 19/24] of: unittest: fix memory leak in unittest_data_add Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 20/24] MIPS: bmips: mark exception vectors as char arrays Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 21/24] i2c: stm32f7: remove warning when compiling with W=1 Sasha Levin
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 22/24] cifs: Fix cifsInodeInfo lock_sem deadlock when reconnect occurs Sasha Levin
2019-10-30 15:55 ` Sasha Levin [this message]
2019-10-30 15:55 ` [PATCH AUTOSEL 4.14 24/24] nbd: verify socket is supported during setup Sasha Levin
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191030155555.10494-23-sashal@kernel.org \
--to=sashal@kernel.org \
--cc=axboe@kernel.dk \
--cc=josef@toxicpanda.com \
--cc=linux-block@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=mchristi@redhat.com \
--cc=nbd@other.debian.org \
--cc=stable@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).