From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-13.0 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 01DCDC433E1 for ; Thu, 6 Aug 2020 19:11:45 +0000 (UTC) Received: from merlin.infradead.org (merlin.infradead.org [205.233.59.134]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 229AF221E3 for ; Thu, 6 Aug 2020 19:11:46 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=lists.infradead.org header.i=@lists.infradead.org header.b="wI0gtkH9" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 229AF221E3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=grimberg.me Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=merlin.20170209; h=Sender:Content-Transfer-Encoding: Content-Type:List-Subscribe:List-Help:List-Post:List-Archive:List-Unsubscribe :List-Id:MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:To:From: Reply-To:Cc:Content-ID:Content-Description:Resent-Date:Resent-From: Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:List-Owner; bh=/2kZyevh0uKVbidpzfJzwYGGewP76LgWaSxcvUFFlt0=; b=wI0gtkH98ayXzhUParAhP3UfTW gVVpIdlYQv+CsrHZKLi1F5zNhAfG9ONCUv5FEd+2jb400pl6rBsYXAaVUBaqESnb094wFyN7AAbdV /U9qveaTmvgezBO6uWH69FQSw1+1mCZKpRFkYmsW0yaYhJH7u1TU+zvQnJb1u/S94k+QozT1DuskB Tu6xadVPUOOkFYyGOyjTuHZhNXIvIgHgpJMppG1KSib3sGqC8V4eHR/uCS7k5AJs3Th+E+6V3QU0k s1sfPObiggw3rclXzU7AvJjk/WkTpMUAm6FSTQ6JP6/2Q64T1YDt5SAg6CZz2B7yw2ADUZKxcDaZu L7Ur9kAQ==; Received: from localhost ([::1] helo=merlin.infradead.org) by merlin.infradead.org with esmtp (Exim 4.92.3 #3 (Red Hat Linux)) id 1k3lIj-0008CC-Lj; Thu, 06 Aug 2020 19:11:41 +0000 Received: from mail-pl1-f193.google.com ([209.85.214.193]) by merlin.infradead.org with esmtps (Exim 4.92.3 #3 (Red Hat Linux)) id 1k3lIe-000892-6O for linux-nvme@lists.infradead.org; Thu, 06 Aug 2020 19:11:38 +0000 Received: by mail-pl1-f193.google.com with SMTP id p1so28273971pls.4 for ; Thu, 06 Aug 2020 12:11:35 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=4C62l9Rezo5SIMRgmYcUvCuRya6oBs9UIR+aXhpVUFA=; b=eCo2PVN/XlP2AzsFumjjs9/74sstVg6NO+TUFqzGRLuFH+dkiFLGD4pje0JVOMSsRD JZaGjTtFjIU8bkW9MNWYEZfUMyiEyINEYpp6v7V4IO6DiHSBAa3LWiT0jtCuSShGsD5g x07w8x8NpoAphELa804GvabV3t7k9myj/XoqWeOoVod8QL0H+uJYlC5SXt1bkWCRX7An doE36TI/h0V/GeqWrCy1GO2EI2rvS30AnBRMpZ/xDRBkHTBvXQH8Kj/BHwLmZnXp0PWC mqwVUO4hKKLoejL5VHY8BDIFoK+0VqAdNHfKoXfPPCrj565kWBUjA+2ab72ZjDk1HLhQ ac0Q== X-Gm-Message-State: AOAM532ZhGzngbW9NUhl9ZXgrC8vn1zGWyvptq1f8wRmm4XmANMWTIrA loPEaD4F2D4U/Wqurw8w8R94ddky X-Google-Smtp-Source: ABdhPJwIcmx2ut1Qasbqy/TmnluBpQao68vwKrh8+NrsLC23hIwTknyEuuXTUqjpPxHIu7GnLY4L+Q== X-Received: by 2002:a17:90a:718c:: with SMTP id i12mr9947022pjk.101.1596741094634; Thu, 06 Aug 2020 12:11:34 -0700 (PDT) Received: from localhost.localdomain ([2601:647:4802:9070:d88d:857c:b14c:519a]) by smtp.gmail.com with ESMTPSA id gn13sm7801348pjb.17.2020.08.06.12.11.33 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Aug 2020 12:11:33 -0700 (PDT) From: Sagi Grimberg To: linux-nvme@lists.infradead.org, Christoph Hellwig , Keith Busch , James Smart Subject: [PATCH v2 3/8] nvme-tcp: serialize controller teardown double completion Date: Thu, 6 Aug 2020 12:11:22 -0700 Message-Id: <20200806191127.592062-4-sagi@grimberg.me> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200806191127.592062-1-sagi@grimberg.me> References: <20200806191127.592062-1-sagi@grimberg.me> MIME-Version: 1.0 X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20200806_151136_290552_EFCD642B X-CRM114-Status: GOOD ( 16.26 ) X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org In the timeout handler we may need to complete a request because the request that timed out may be an I/O that is a part of a serial sequence of controller teardown or initialization. In order to complete the request, we need to fence any other context that may compete with us and complete the request that is timing out. In this case, we could have a potential double completion in case a hard-irq or a different competing context triggered error recovery and is running inflight request cancellation concurrently with the timeout handler. Protect using a ctrl teardown_lock to serialize contexts that may complete a cancelled request due to error recovery or a reset. Signed-off-by: Sagi Grimberg --- drivers/nvme/host/tcp.c | 11 +++++++++-- 1 file changed, 9 insertions(+), 2 deletions(-) diff --git a/drivers/nvme/host/tcp.c b/drivers/nvme/host/tcp.c index 62fbaecdc960..80af1b287cd0 100644 --- a/drivers/nvme/host/tcp.c +++ b/drivers/nvme/host/tcp.c @@ -124,6 +124,7 @@ struct nvme_tcp_ctrl { struct sockaddr_storage src_addr; struct nvme_ctrl ctrl; + struct mutex teardown_lock; struct work_struct err_work; struct delayed_work connect_work; struct nvme_tcp_request async_req; @@ -1527,7 +1528,6 @@ static void nvme_tcp_stop_queue(struct nvme_ctrl *nctrl, int qid) if (!test_and_clear_bit(NVME_TCP_Q_LIVE, &queue->flags)) return; - __nvme_tcp_stop_queue(queue); } @@ -1875,6 +1875,7 @@ static int nvme_tcp_configure_admin_queue(struct nvme_ctrl *ctrl, bool new) static void nvme_tcp_teardown_admin_queue(struct nvme_ctrl *ctrl, bool remove) { + mutex_lock(&to_tcp_ctrl(ctrl)->teardown_lock); blk_mq_quiesce_queue(ctrl->admin_q); nvme_tcp_stop_queue(ctrl, 0); if (ctrl->admin_tagset) { @@ -1885,13 +1886,16 @@ static void nvme_tcp_teardown_admin_queue(struct nvme_ctrl *ctrl, if (remove) blk_mq_unquiesce_queue(ctrl->admin_q); nvme_tcp_destroy_admin_queue(ctrl, remove); + mutex_unlock(&to_tcp_ctrl(ctrl)->teardown_lock); } static void nvme_tcp_teardown_io_queues(struct nvme_ctrl *ctrl, bool remove) { + mutex_lock(&to_tcp_ctrl(ctrl)->teardown_lock); if (ctrl->queue_count <= 1) - return; + goto out; + blk_mq_quiesce_queue(ctrl->admin_q); nvme_start_freeze(ctrl); nvme_stop_queues(ctrl); nvme_tcp_stop_io_queues(ctrl); @@ -1903,6 +1907,8 @@ static void nvme_tcp_teardown_io_queues(struct nvme_ctrl *ctrl, if (remove) nvme_start_queues(ctrl); nvme_tcp_destroy_io_queues(ctrl, remove); +out: + mutex_unlock(&to_tcp_ctrl(ctrl)->teardown_lock); } static void nvme_tcp_reconnect_or_remove(struct nvme_ctrl *ctrl) @@ -2423,6 +2429,7 @@ static struct nvme_ctrl *nvme_tcp_create_ctrl(struct device *dev, nvme_tcp_reconnect_ctrl_work); INIT_WORK(&ctrl->err_work, nvme_tcp_error_recovery_work); INIT_WORK(&ctrl->ctrl.reset_work, nvme_reset_ctrl_work); + mutex_init(&ctrl->teardown_lock); if (!(opts->mask & NVMF_OPT_TRSVCID)) { opts->trsvcid = -- 2.25.1 _______________________________________________ Linux-nvme mailing list Linux-nvme@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-nvme