From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.9 required=3.0 tests=DKIMWL_WL_HIGH,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8D07FC3F2D1 for ; Mon, 2 Mar 2020 18:46:39 +0000 (UTC) Received: from vger.kernel.org (vger.kernel.org [209.132.180.67]) by mail.kernel.org (Postfix) with ESMTP id 599C02072A for ; Mon, 2 Mar 2020 18:46:39 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=amazon.com header.i=@amazon.com header.b="QQhiJ//B" Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1727481AbgCBSqi (ORCPT ); Mon, 2 Mar 2020 13:46:38 -0500 Received: from smtp-fw-33001.amazon.com ([207.171.190.10]:21256 "EHLO smtp-fw-33001.amazon.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1727439AbgCBSqi (ORCPT ); Mon, 2 Mar 2020 13:46:38 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=amazon.com; i=@amazon.com; q=dns/txt; s=amazon201209; t=1583174797; x=1614710797; h=from:to:cc:subject:date:message-id:in-reply-to: references:mime-version:content-transfer-encoding; bh=+xq6lveZ+prywBAcKUivfYzgKg22lx6AhvILIKrWiGg=; b=QQhiJ//BlWDktyspSr4EhPMqYDQnPbg7r15qz+hvkLTwggfuHzxPpkpc imBGGJdCdJmClaMzDIUbAzTpHa17UZzJv1J9TlrpQH69Z7Sx0dH8rHqIt O27+lUtdwVRYDvfx6dErQNwmsXQg3u7tpjLzjiMNsWl6Gs+3LwJmkG+Vj 4=; IronPort-SDR: KH34hA9bGk7InqJD/pmZSs9iwTKAeIjEaMou9EBY1q0unkHGGYAdD3lB/dlqion3mAJ4kNISJg 0dkQuKVAuhRA== X-IronPort-AV: E=Sophos;i="5.70,507,1574121600"; d="scan'208";a="30074795" Received: from sea32-co-svc-lb4-vlan3.sea.corp.amazon.com (HELO email-inbound-relay-2c-2225282c.us-west-2.amazon.com) ([10.47.23.38]) by smtp-border-fw-out-33001.sea14.amazon.com with ESMTP; 02 Mar 2020 18:46:35 +0000 Received: from EX13MTAUEA002.ant.amazon.com (pdx4-ws-svc-p6-lb7-vlan3.pdx.amazon.com [10.170.41.166]) by email-inbound-relay-2c-2225282c.us-west-2.amazon.com (Postfix) with ESMTPS id 0221BA3608; Mon, 2 Mar 2020 18:46:33 +0000 (UTC) Received: from EX13D12EUC001.ant.amazon.com (10.43.164.45) by EX13MTAUEA002.ant.amazon.com (10.43.61.77) with Microsoft SMTP Server (TLS) id 15.0.1236.3; Mon, 2 Mar 2020 18:46:23 +0000 Received: from EX13MTAUWB001.ant.amazon.com (10.43.161.207) by EX13D12EUC001.ant.amazon.com (10.43.164.45) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Mon, 2 Mar 2020 18:46:21 +0000 Received: from u961addbe640f56.ant.amazon.com (10.28.84.111) by mail-relay.amazon.com (10.43.161.249) with Microsoft SMTP Server id 15.0.1367.3 via Frontend Transport; Mon, 2 Mar 2020 18:46:18 +0000 From: Stanislav Spassov To: CC: Stanislav Spassov , Bjorn Helgaas , Thomas Gleixner , Andrew Morton , =?UTF-8?q?Jan=20H=20=2E=20Sch=C3=B6nherr?= , Jonathan Corbet , Ashok Raj , Alex Williamson , "Sinan Kaya" , Rajat Jain Subject: [PATCH v2 16/17] PCI: Add CRS handling to pci_dev_wait() Date: Mon, 2 Mar 2020 19:44:28 +0100 Message-ID: <20200302184429.12880-17-stanspas@amazon.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20200302184429.12880-1-stanspas@amazon.com> References: <20200302184429.12880-1-stanspas@amazon.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: linux-pci-owner@vger.kernel.org Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org From: Stanislav Spassov The PCI Express specification dictates minimal amounts of time that the host needs to wait after triggering different kinds of resets before it is allowed to attempt accessing the device. After this waiting period, devices are required to be responsive to Configuration Space reads. However, if a device needs more time to actually complete the reset operation internally, it may respond to the read with a Completion Request Retry Status (CRS), and keep doing so on subsequent reads for as long as necessary. If the device is broken, it may even keep responding with CRS indefinitely. The specification also mandates that any Root Port that supports CRS and has CRS Software Visibility (CRS SV) enabled will synthesize the special value 0x0001 for the Vendor ID and set any other bits to 1 upon receiving a CRS Completion for a Configuration Read Request that includes both bytes of the Vendor ID (offset 0). IF CRS is supported by Root Port but CRS SV is not enabled, the request is retried autonomosly by the Root Port. Platform-specific configuration registers may exist to limit the number of or time taken by such retries. If CRS is not supported, or a different register (not Vendor ID) is polled, or the device is responding with CA/UR Completions (rather than CRS), the behavior is platform-dependent, but generally the Root Port synthesizes ~0 to complete the software read. Previously, pci_dev_wait() avoided taking advantage of CRS. However, on platforms where no limit/timeout can be configured as explained above, a device responding with CRS for too long (e.g. because it is stuck and cannot complete its reset) may trigger more severe error conditions (e.g. TOR timeout, 3-strike CPU CATERR), because the Root Port never reports back to the lower-level component requesting the transaction. This patch introduces special handling when CRS is available, and otherwise falls back to the previous behavior of polling COMMAND. Signed-off-by: Stanislav Spassov --- drivers/pci/pci.c | 52 +++++++++++++++++++++++++++++++++++++++-------- 1 file changed, 44 insertions(+), 8 deletions(-) diff --git a/drivers/pci/pci.c b/drivers/pci/pci.c index f1ba931b0ead..1a504419e0de 100644 --- a/drivers/pci/pci.c +++ b/drivers/pci/pci.c @@ -1081,18 +1081,54 @@ static int pci_dev_wait(struct pci_dev *dev, enum pci_init_event event) { const char *event_name = pci_init_event_name(event); int timeout = dev->reset_ready_poll_ms; + int waited = 0; + int rc = 0; + /* * After reset, the device should not silently discard config * requests, but it may still indicate that it needs more time by - * responding to them with CRS completions. The Root Port will - * generally synthesize ~0 data to complete the read (except when - * CRS SV is enabled and the read was for the Vendor ID; in that - * case it synthesizes 0x0001 data). - * - * Wait for the device to return a non-CRS completion. Read the - * Command register instead of Vendor ID so we don't have to - * contend with the CRS SV value. + * responding to them with CRS completions. For such completions: + * - If CRS SV is enabled on the Root Port, and the read request + * covers both bytes of the Vendor ID register, the Root Port + * will synthesize the value 0x0001 (and set any extra requested + * bytes to 0xff) + * - If CRS SV is not enabled on the Root Port, the Root Port must + * re-issue the Configuration Request as a new Request. + * Depending on platform-specific Root Complex configurations, + * the Root Port may stop retrying after a set number of attempts, + * or a configured timeout is hit, or continue indefinitely + * (ultimately resulting in non-PCI-specific platform errors, such as + * a TOR timeout). + */ + if (dev->crssv_enabled) { + u32 id; + + rc = pci_dev_poll_until_not_equal(dev, PCI_VENDOR_ID, 0xffff, + 0x0001, event_name, timeout, + &waited, &id); + if (rc) + return rc; + + /* + * If Vendor/Device ID is valid, the device must be ready. + * Note: SR-IOV VFs return ~0 for reads to Vendor/Device + * ID and will not be recognized as ready by this check. + */ + if (id != 0x0000ffff && id != 0xffff0000 && + id != 0x00000000 && id != 0xffffffff) + return 0; + } + + /* + * Root Ports will generally indicate error scenarios (e.g. + * internal timeouts, or received Completion with CA/UR) by + * synthesizing an 'all bits set' value (~0). + * In case CRS is not supported/enabled, as well as for SR-IOV VFs, + * fall back to polling a different register that cannot validly + * contain ~0. As of PCIe 5.0, bits 11-15 of COMMAND are still RsvdP + * and must return 0 when read. + * XXX: These bits might become meaningful in the future */ return pci_dev_poll_until_not_equal(dev, PCI_COMMAND, ~0, ~0, event_name, timeout, NULL, -- 2.25.1 Amazon Development Center Germany GmbH Krausenstr. 38 10117 Berlin Geschaeftsfuehrung: Christian Schlaeger, Jonathan Weiss Eingetragen am Amtsgericht Charlottenburg unter HRB 149173 B Sitz: Berlin Ust-ID: DE 289 237 879