Stable Archive on lore.kernel.org
 help / color / Atom feed
From: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
To: linux-kernel@vger.kernel.org
Cc: Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	stable@vger.kernel.org, Mike Christie <mchristi@redhat.com>,
	Josef Bacik <josef@toxicpanda.com>, Jens Axboe <axboe@kernel.dk>,
	Sasha Levin <sashal@kernel.org>
Subject: [PATCH 4.19 36/79] nbd: handle racing with errored out commands
Date: Fri,  8 Nov 2019 19:50:16 +0100
Message-ID: <20191108174806.174452403@linuxfoundation.org> (raw)
In-Reply-To: <20191108174745.495640141@linuxfoundation.org>

From: Josef Bacik <josef@toxicpanda.com>

[ Upstream commit 7ce23e8e0a9cd38338fc8316ac5772666b565ca9 ]

We hit the following warning in production

print_req_error: I/O error, dev nbd0, sector 7213934408 flags 80700
------------[ cut here ]------------
refcount_t: underflow; use-after-free.
WARNING: CPU: 25 PID: 32407 at lib/refcount.c:190 refcount_sub_and_test_checked+0x53/0x60
Workqueue: knbd-recv recv_work [nbd]
RIP: 0010:refcount_sub_and_test_checked+0x53/0x60
Call Trace:
 blk_mq_free_request+0xb7/0xf0
 blk_mq_complete_request+0x62/0xf0
 recv_work+0x29/0xa1 [nbd]
 process_one_work+0x1f5/0x3f0
 worker_thread+0x2d/0x3d0
 ? rescuer_thread+0x340/0x340
 kthread+0x111/0x130
 ? kthread_create_on_node+0x60/0x60
 ret_from_fork+0x1f/0x30
---[ end trace b079c3c67f98bb7c ]---

This was preceded by us timing out everything and shutting down the
sockets for the device.  The problem is we had a request in the queue at
the same time, so we completed the request twice.  This can actually
happen in a lot of cases, we fail to get a ref on our config, we only
have one connection and just error out the command, etc.

Fix this by checking cmd->status in nbd_read_stat.  We only change this
under the cmd->lock, so we are safe to check this here and see if we've
already error'ed this command out, which would indicate that we've
completed it as well.

Reviewed-by: Mike Christie <mchristi@redhat.com>
Signed-off-by: Josef Bacik <josef@toxicpanda.com>

Signed-off-by: Jens Axboe <axboe@kernel.dk>
Signed-off-by: Sasha Levin <sashal@kernel.org>
---
 drivers/block/nbd.c | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
index da6a36d14f4cf..867841c56a6da 100644
--- a/drivers/block/nbd.c
+++ b/drivers/block/nbd.c
@@ -663,6 +663,12 @@ static struct nbd_cmd *nbd_read_stat(struct nbd_device *nbd, int index)
 		ret = -ENOENT;
 		goto out;
 	}
+	if (cmd->status != BLK_STS_OK) {
+		dev_err(disk_to_dev(nbd->disk), "Command already handled %p\n",
+			req);
+		ret = -ENOENT;
+		goto out;
+	}
 	if (test_bit(NBD_CMD_REQUEUED, &cmd->flags)) {
 		dev_err(disk_to_dev(nbd->disk), "Raced with timeout on req %p\n",
 			req);
-- 
2.20.1




  parent reply index

Thread overview: 83+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2019-11-08 18:49 [PATCH 4.19 00/79] 4.19.83-stable review Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 01/79] kbuild: add -fcf-protection=none when using retpoline flags Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 02/79] regulator: of: fix suspend-min/max-voltage parsing Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 03/79] ASoC: wm8994: Do not register inapplicable controls for WM1811 Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 04/79] arm64: dts: allwinner: a64: pine64-plus: Add PHY regulator delay Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 05/79] arm64: dts: allwinner: a64: sopine-baseboard: " Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 06/79] arm64: dts: Fix gpio to pinmux mapping Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 07/79] regulator: ti-abb: Fix timeout in ti_abb_wait_txdone/ti_abb_clear_all_txdone Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 08/79] ASoC: rt5682: add NULL handler to set_jack function Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 09/79] regulator: pfuze100-regulator: Variable "val" in pfuze100_regulator_probe() could be uninitialized Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 10/79] ASoC: wm_adsp: Dont generate kcontrols without READ flags Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 11/79] ASoc: rockchip: i2s: Fix RPM imbalance Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 12/79] ARM: dts: logicpd-torpedo-som: Remove twl_keypad Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 13/79] pinctrl: ns2: Fix off by one bugs in ns2_pinmux_enable() Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 14/79] ARM: mm: fix alignment handler faults under memory pressure Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 15/79] scsi: qla2xxx: fix a potential NULL pointer dereference Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 16/79] scsi: scsi_dh_alua: handle RTPG sense code correctly during state transitions Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 17/79] scsi: sni_53c710: fix compilation error Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 18/79] scsi: fix kconfig dependency warning related to 53C700_LE_ON_BE Greg Kroah-Hartman
2019-11-08 18:49 ` [PATCH 4.19 19/79] ARM: dts: imx7s: Correct GPTs ipg clock source Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 20/79] perf c2c: Fix memory leak in build_cl_output() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 21/79] 8250-men-mcb: fix error checking when get_num_ports returns -ENODEV Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 22/79] perf kmem: Fix memory leak in compact_gfp_flags() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 23/79] ARM: davinci: dm365: Fix McBSP dma_slave_map entry Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 24/79] drm/amdgpu: fix potential VM faults Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 25/79] scsi: target: core: Do not overwrite CDB byte 1 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 26/79] tracing: Fix "gfp_t" format for synthetic events Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 27/79] ARM: 8926/1: v7m: remove register save to stack before svc Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 28/79] of: unittest: fix memory leak in unittest_data_add Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 29/79] MIPS: bmips: mark exception vectors as char arrays Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 30/79] irqchip/gic-v3-its: Use the exact ITSList for VMOVP Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 31/79] i2c: stm32f7: fix first byte to send in slave mode Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 32/79] i2c: stm32f7: fix a race in slave mode with arbitration loss irq Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 33/79] i2c: stm32f7: remove warning when compiling with W=1 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 34/79] cifs: Fix cifsInodeInfo lock_sem deadlock when reconnect occurs Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 35/79] nbd: protect cmd->status with cmd->lock Greg Kroah-Hartman
2019-11-08 18:50 ` Greg Kroah-Hartman [this message]
2019-11-08 18:50 ` [PATCH 4.19 37/79] cxgb4: fix panic when attaching to ULD fail Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 38/79] dccp: do not leak jiffies on the wire Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 39/79] erspan: fix the tun_info options_len check for erspan Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 40/79] inet: stop leaking jiffies on the wire Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 41/79] net: annotate accesses to sk->sk_incoming_cpu Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 42/79] net: annotate lockless accesses to sk->sk_napi_id Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 43/79] net: dsa: bcm_sf2: Fix IMP setup for port different than 8 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 44/79] net: ethernet: ftgmac100: Fix DMA coherency issue with SW checksum Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 45/79] net: fix sk_page_frag() recursion from memory reclaim Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 46/79] net: hisilicon: Fix ping latency when deal with high throughput Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 47/79] net/mlx4_core: Dynamically set guaranteed amount of counters per VF Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 48/79] netns: fix GFP flags in rtnl_net_notifyid() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 49/79] net: usb: lan78xx: Disable interrupts before calling generic_handle_irq() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 50/79] net: Zeroing the structure ethtool_wolinfo in ethtool_get_wol() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 51/79] selftests: net: reuseport_dualstack: fix uninitalized parameter Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 52/79] udp: fix data-race in udp_set_dev_scratch() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 53/79] vxlan: check tun_info options_len properly Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 54/79] net: add skb_queue_empty_lockless() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 55/79] udp: use skb_queue_empty_lockless() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 56/79] net: use skb_queue_empty_lockless() in poll() handlers Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 57/79] net: use skb_queue_empty_lockless() in busy poll contexts Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 58/79] net: add READ_ONCE() annotation in __skb_wait_for_more_packets() Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 59/79] ipv4: fix route update on metric change Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 60/79] selftests: fib_tests: add more tests for metric update Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 61/79] net/mlx5e: Fix handling of compressed CQEs in case of low NAPI budget Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 62/79] r8169: fix wrong PHY ID issue with RTL8168dp Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 63/79] net/mlx5e: Fix ethtool self test: link speed Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 64/79] net: dsa: b53: Do not clear existing mirrored port mask Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 65/79] net: bcmgenet: dont set phydev->link from MAC Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 66/79] net: phy: bcm7xxx: define soft_reset for 40nm EPHY Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 67/79] net: bcmgenet: reset 40nm EPHY on energy detect Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 68/79] net: usb: lan78xx: Connect PHY before registering MAC Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 69/79] net: dsa: fix switch tree list Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 70/79] r8152: add device id for Lenovo ThinkPad USB-C Dock Gen 2 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 71/79] net/flow_dissector: switch to siphash Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 72/79] wireless: Skip directory when generating certificates Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 73/79] platform/x86: pmc_atom: Add Siemens SIMATIC IPC227E to critclk_systems DMI table Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 74/79] powerpc/mm: Fixup tlbie vs mtpidr/mtlpidr ordering issue on POWER9 Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 75/79] selftests/powerpc: Add test case for tlbie vs mtpidr ordering issue Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 76/79] selftests/powerpc: Fix compile error on tlbie_test due to newer gcc Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 77/79] ASoC: pcm3168a: The codec does not support S32_LE Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 78/79] arm64: dts: ti: k3-am65-main: Fix gic-its node unit-address Greg Kroah-Hartman
2019-11-08 18:50 ` [PATCH 4.19 79/79] usb: gadget: udc: core: Fix segfault if udc_bind_to_driver() for pending driver fails Greg Kroah-Hartman
2019-11-08 23:57 ` [PATCH 4.19 00/79] 4.19.83-stable review kernelci.org bot
2019-11-09 10:46 ` Naresh Kamboju
2019-11-09 15:40 ` Guenter Roeck

Reply instructions:

You may reply publically to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20191108174806.174452403@linuxfoundation.org \
    --to=gregkh@linuxfoundation.org \
    --cc=axboe@kernel.dk \
    --cc=josef@toxicpanda.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=mchristi@redhat.com \
    --cc=sashal@kernel.org \
    --cc=stable@vger.kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Stable Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/stable/0 stable/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 stable stable/ https://lore.kernel.org/stable \
		stable@vger.kernel.org
	public-inbox-index stable

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.stable


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git