Stable Archive on lore.kernel.org
 help / color / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Mike Christie <mchristi@redhat.com>,
	Josef Bacik <josef@toxicpanda.com>, Jens Axboe <axboe@kernel.dk>,
	Sasha Levin <sashal@kernel.org>
Subject: [PATCH 4.14 22/62] nbd: handle racing with errored out commands
Date: Fri,  8 Nov 2019 19:50:10 +0100
Message-ID: <20191108174737.710372101@linuxfoundation.org> (raw)
In-Reply-To: <20191108174719.228826381@linuxfoundation.org>

From: Josef Bacik <josef@toxicpanda.com>

[ Upstream commit 7ce23e8e0a9cd38338fc8316ac5772666b565ca9 ]

We hit the following warning in production

print_req_error: I/O error, dev nbd0, sector 7213934408 flags 80700
------------[ cut here ]------------
refcount_t: underflow; use-after-free.
WARNING: CPU: 25 PID: 32407 at lib/refcount.c:190 refcount_sub_and_test_checked+0x53/0x60
Workqueue: knbd-recv recv_work [nbd]
RIP: 0010:refcount_sub_and_test_checked+0x53/0x60
Call Trace:
 blk_mq_free_request+0xb7/0xf0
 blk_mq_complete_request+0x62/0xf0
 recv_work+0x29/0xa1 [nbd]
 process_one_work+0x1f5/0x3f0
 worker_thread+0x2d/0x3d0
 ? rescuer_thread+0x340/0x340
 kthread+0x111/0x130
 ? kthread_create_on_node+0x60/0x60
 ret_from_fork+0x1f/0x30
---[ end trace b079c3c67f98bb7c ]---

This was preceded by us timing out everything and shutting down the
sockets for the device.  The problem is we had a request in the queue at
the same time, so we completed the request twice.  This can actually
happen in a lot of cases, we fail to get a ref on our config, we only
have one connection and just error out the command, etc.

Fix this by checking cmd->status in nbd_read_stat.  We only change this
under the cmd->lock, so we are safe to check this here and see if we've
already error'ed this command out, which would indicate that we've
completed it as well.

Reviewed-by: Mike Christie <mchristi@redhat.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 drivers/block/nbd.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index f3d0bc9a99058..34dfadd4dcd41 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -648,6 +648,12 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index)
 		ret = -ENOENT;
 		goto out;
 	}
+	if (cmd->status != BLK_STS_OK) {
+		dev_err(disk_to_dev(nbd->disk), "Command already handled %p\n",
+			req);
+		ret = -ENOENT;
+		goto out;
+	}
 	if (test_bit(NBD_CMD_REQUEUED, &cmd->flags)) {
 		dev_err(disk_to_dev(nbd->disk), "Raced with timeout on req %p\n",
 			req);
-- 
2.20.1




  parent reply index

Thread overview: 66+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-08 18:49 [PATCH 4.14 00/62] 4.14.153-stable review Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 01/62] arm64: dts: Fix gpio to pinmux mapping Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 02/62] regulator: ti-abb: Fix timeout in ti_abb_wait_txdone/ti_abb_clear_all_txdone Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 03/62] regulator: pfuze100-regulator: Variable "val" in pfuze100_regulator_probe() could be uninitialized Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 04/62] ASoC: wm_adsp: Dont generate kcontrols without READ flags Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 05/62] ASoc: rockchip: i2s: Fix RPM imbalance Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 06/62] ARM: dts: logicpd-torpedo-som: Remove twl_keypad Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 07/62] pinctrl: ns2: Fix off by one bugs in ns2_pinmux_enable() Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 08/62] ARM: mm: fix alignment handler faults under memory pressure Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 09/62] scsi: scsi_dh_alua: handle RTPG sense code correctly during state transitions Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 10/62] scsi: sni_53c710: fix compilation error Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.14 11/62] scsi: fix kconfig dependency warning related to 53C700_LE_ON_BE Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 12/62] ARM: dts: imx7s: Correct GPTs ipg clock source Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 13/62] perf c2c: Fix memory leak in build_cl_output() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 14/62] perf kmem: Fix memory leak in compact_gfp_flags() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 15/62] ARM: davinci: dm365: Fix McBSP dma_slave_map entry Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 16/62] scsi: target: core: Do not overwrite CDB byte 1 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 17/62] ARM: 8926/1: v7m: remove register save to stack before svc Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 18/62] of: unittest: fix memory leak in unittest_data_add Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 19/62] MIPS: bmips: mark exception vectors as char arrays Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 20/62] i2c: stm32f7: remove warning when compiling with W=1 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 21/62] cifs: Fix cifsInodeInfo lock_sem deadlock when reconnect occurs Greg Kroah-Hartman
2019-11-08 18:50 ` Greg Kroah-Hartman [this message]
2019-11-08 18:50 ` [PATCH 4.14 23/62] cxgb4: fix panic when attaching to ULD fail Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 24/62] dccp: do not leak jiffies on the wire Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 25/62] net: annotate accesses to sk->sk_incoming_cpu Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 26/62] net: annotate lockless accesses to sk->sk_napi_id Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 27/62] net: dsa: bcm_sf2: Fix IMP setup for port different than 8 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 28/62] net: ethernet: ftgmac100: Fix DMA coherency issue with SW checksum Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 29/62] net: fix sk_page_frag() recursion from memory reclaim Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 30/62] net: hisilicon: Fix ping latency when deal with high throughput Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 31/62] net/mlx4_core: Dynamically set guaranteed amount of counters per VF Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 32/62] net: Zeroing the structure ethtool_wolinfo in ethtool_get_wol() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 33/62] selftests: net: reuseport_dualstack: fix uninitalized parameter Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 34/62] udp: fix data-race in udp_set_dev_scratch() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 35/62] net: add READ_ONCE() annotation in __skb_wait_for_more_packets() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 36/62] net/mlx5e: Fix handling of compressed CQEs in case of low NAPI budget Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 37/62] net: dsa: b53: Do not clear existing mirrored port mask Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 38/62] net: usb: lan78xx: Connect PHY before registering MAC Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 39/62] r8152: add device id for Lenovo ThinkPad USB-C Dock Gen 2 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 40/62] net: dsa: fix switch tree list Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 41/62] net: bcmgenet: reset 40nm EPHY on energy detect Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 42/62] net: add skb_queue_empty_lockless() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 43/62] udp: use skb_queue_empty_lockless() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 44/62] net: use skb_queue_empty_lockless() in poll() handlers Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 45/62] net: use skb_queue_empty_lockless() in busy poll contexts Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 46/62] vxlan: check tun_info options_len properly Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 47/62] erspan: fix the tun_info options_len check for erspan Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 48/62] inet: stop leaking jiffies on the wire Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 49/62] net/flow_dissector: switch to siphash Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 50/62] dmaengine: qcom: bam_dma: Fix resource leak Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 51/62] sched/wake_q: Fix wakeup ordering for wake_q Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 52/62] kbuild: use -fmacro-prefix-map to make __FILE__ a relative path Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 53/62] kbuild: add -fcf-protection=none when using retpoline flags Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 54/62] platform/x86: pmc_atom: Add Siemens SIMATIC IPC227E to critclk_systems DMI table Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 55/62] iio: adc: stm32-adc: move registers definitions Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 56/62] iio: adc: stm32-adc: fix a race when using several adcs with dma and irq Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 57/62] powerpc/mm: Fixup tlbie vs store ordering issue on POWER9 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 58/62] powerpc/book3s64/mm: Dont do tlbie fixup for some hardware revisions Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 59/62] powerpc/book3s64/radix: Rename CPU_FTR_P9_TLBIE_BUG feature flag Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 60/62] powerpc/mm: Fixup tlbie vs mtpidr/mtlpidr ordering issue on POWER9 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 61/62] selftests/powerpc: Add test case for tlbie vs mtpidr ordering issue Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.14 62/62] selftests/powerpc: Fix compile error on tlbie_test due to newer gcc Greg Kroah-Hartman
2019-11-09  1:57 ` [PATCH 4.14 00/62] 4.14.153-stable review kernelci.org bot
2019-11-09 10:19 ` Naresh Kamboju
2019-11-09 15:40 ` Guenter Roeck

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191108174737.710372101@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=axboe@kernel.dk \
    --cc=josef@toxicpanda.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mchristi@redhat.com \
    --cc=sashal@kernel.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Stable Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/stable/0 stable/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 stable stable/ https://lore.kernel.org/stable \
		stable@vger.kernel.org
	public-inbox-index stable

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.stable


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git