From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752404AbdHBRyC (ORCPT ); Wed, 2 Aug 2017 13:54:02 -0400 Received: from mail-by2nam01on0108.outbound.protection.outlook.com ([104.47.34.108]:51840 "EHLO NAM01-BY2-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751699AbdHBRx7 (ORCPT ); Wed, 2 Aug 2017 13:53:59 -0400 From: Casey Leedom To: Ding Tianhong , Alexander Duyck CC: Alex Williamson , Sinan Kaya , "ashok.raj@intel.com" , "bhelgaas@google.com" , "helgaas@kernel.org" , Michael Werner , Ganesh GR , "asit.k.mallick@intel.com" , "patrick.j.cramer@intel.com" , "Suravee.Suthikulpanit@amd.com" , "Bob.Shaw@amd.com" , "l.stach@pengutronix.de" , "amira@mellanox.com" , "gabriele.paoloni@huawei.com" , "David.Laight@aculab.com" , "jeffrey.t.kirsher@intel.com" , "catalin.marinas@arm.com" , "will.deacon@arm.com" , "mark.rutland@arm.com" , "robin.murphy@arm.com" , "davem@davemloft.net" , "linux-arm-kernel@lists.infradead.org" , "netdev@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linuxarm@huawei.com" Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Thread-Topic: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Thread-Index: AQHS++OWrGKhofvtIUCheprjRIXqU6JSQPmAgABHnoCAANEhgIAL8fcAgAPZDQCAA1q+dYAAcmoAgAEXs4CAAJoQgIAI088D Date: Wed, 2 Aug 2017 17:53:52 +0000 Message-ID: References: <1499955692-26556-1-git-send-email-dingtianhong@huawei.com> <1499955692-26556-3-git-send-email-dingtianhong@huawei.com> <0260e398-bd8e-6615-6d5c-1f7c07b6fc09@huawei.com> <67be791f-e0cf-8284-9229-17174dc741ef@codeaurora.org> <5f9b8bfb-41a8-a17c-6fea-581aec1d5573@huawei.com> <20170724090516.2e0f0d2a@w520.home> <75213fca-4522-2297-3cb8-338e643d3552@huawei.com> ,<14218972-553d-eb60-0207-460ac7f4b064@huawei.com> In-Reply-To: <14218972-553d-eb60-0207-460ac7f4b064@huawei.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=leedom@chelsio.com; x-originating-ip: [12.32.117.8] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;MWHPR12MB1439;7:4JoNg7hSERFnFyifwBH4B8vE58er7tHSovI5D3gnb9JdEbjF4sk8oDJ8+8mNpjiWyqMDFUet4Re/YkJDQHxJsIhmulIOkR1T4M3y99DA/08HuZuo8hv4CylXMOjCcA8aqLABUmjE22gDffIimynu+1V+gW9iP9RqUNQ9Pozgs/f6qOZ5O5kXVUaL9+MzGA6uHauw3Mz3wQGFqac/+vYJurmVDR9iYQgj9VlO5+3b1MZlpoqtgOeY866Nz1aJHdEPQTOp0pG7nPpGcE2QQGhN1+3JFb6Z8zS9PUfrYs/b+q5scBUGNV3McBbrQaXWyBRTJVthsxPjnhU0/BIo/5XDEVQ2F4ojEShkf0nwxjn0vfv8w2usDww6/7kCgYP1dTC3UYfaMGkAH6BjVjssVTx4XqUNC/k/F8gUTaxeFF4HL/Re12pdmH4rnUxyUmD5ggCwGvfJB6YVLrPzuUwXtRKdhjUphPki7uEj3+h3oUzy93XqozDJwVHaEYKwXBBMDI9Cl/X/plpWxNk8EtHk4DhpjrKrGjzST9/maircPf55dtMoRcD+wgebhl3ShnsaYXzEFKvyHR/OBlLcFeQVduK9VKD/yiOlWlRz8Xz4fep8xQqdDQ0dY3jIDUyPSj/UzeBx16o2hCkaRRctPKFRKR9161je0yxabBV0nRcEA9r5SRt1D3T9MqfkuEKQrCR9BbKtHrmPnt7rqyZCFsWimn5hvOSR9EAGLf7W5VOTYmw6aCUUajT+3KtMGabVDo6BKQ1sXPAluzvnJXi7wTIcR3ZqXPoV28d8h5WxkQb72yz//So= x-ms-office365-filtering-correlation-id: 0d2e73c2-cbcc-47fd-76c3-08d4d9cf7044 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(2017082002075)(300000503095)(300135400095)(2017052603031)(201703131423075)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:MWHPR12MB1439; x-ms-traffictypediagnostic: MWHPR12MB1439: x-exchange-antispam-report-test: UriScan:; x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(10201501046)(100000703101)(100105400095)(3002001)(93006095)(93001095)(6041248)(20161123562025)(20161123558100)(20161123560025)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(2016111802025)(20161123564025)(6072148)(6043046)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:MWHPR12MB1439;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:MWHPR12MB1439; x-forefront-prvs: 0387D64A71 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(39410400002)(39450400003)(39840400002)(39400400002)(199003)(189002)(4326008)(7696004)(39060400002)(53936002)(74316002)(25786009)(189998001)(38730400002)(2900100001)(105586002)(5890100001)(229853002)(8936002)(575784001)(86362001)(77096006)(6246003)(106356001)(68736007)(6506006)(97736004)(93886004)(3660700001)(5660300001)(54356999)(76176999)(3846002)(50986999)(6116002)(102836003)(8676002)(99286003)(55016002)(66066001)(9686003)(8666007)(33656002)(81166006)(81156014)(54906002)(305945005)(478600001)(551934003)(2906002)(14454004)(7416002)(3280700002)(101416001)(7736002)(6436002)(2950100002);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR12MB1439;H:MWHPR12MB1600.namprd12.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 X-OriginatorOrg: chelsio.com X-MS-Exchange-CrossTenant-originalarrivaltime: 02 Aug 2017 17:53:52.8262 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 065db76d-a7ae-4c60-b78a-501e8fc17095 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1439 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id v72Hs6ue008931   Okay, here you go.  As you can tell, it's almost a trivial copy of the cxgb4 patch.     By the way, I realized that we have yet another hole which is likely not to be fixable.  If we're dealing with a problematic Root Complex, and we instantiate Virtual Functions and attach them to a Virtual Machine along with an NVMe device which can deal with Relaxed Ordering TLPs, the VF driver in the VM will be able to tell that it shouldn't attempt to send RO TLPs to the RC because it will see the state of its own PCIe Capability Device Control[Relaxed Ordering Enable] (a copy of the setting in the VF's corresponding PF), but it won't be able to change that and send non-RO TLPs to the RC, and RO TLPs to the NVMe device.  Oh well.   I sure wish that the Intel guys would pop up with a hidden register change for these problematic Intel RCs that perform poorly with RO TLPs.  Their silence has been frustrating. Casey ---------- cxgb4vf Ethernet driver now queries PCIe configuration space to determine if it can send TLPs to it with the Relaxed Ordering Attribute set. diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h b/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h index 109bc63..08c6ddb 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h +++ b/drivers/net/ethernet/chelsio/cxgb4vf/adapter.h @@ -408,6 +408,7 @@ enum { /* adapter flags */ USING_MSI = (1UL << 1), USING_MSIX = (1UL << 2), QUEUES_BOUND = (1UL << 3), + ROOT_NO_RELAXED_ORDERING = (1UL << 4), }; /* diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c index ac7a150..59e7639 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c +++ b/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf_main.c @@ -2888,6 +2888,24 @@ static int cxgb4vf_pci_probe(struct pci_dev *pdev, */ adapter->name = pci_name(pdev); adapter->msg_enable = DFLT_MSG_ENABLE; + + /* If possible, we use PCIe Relaxed Ordering Attribute to deliver + * Ingress Packet Data to Free List Buffers in order to allow for + * chipset performance optimizations between the Root Complex and + * Memory Controllers. (Messages to the associated Ingress Queue + * notifying new Packet Placement in the Free Lists Buffers will be + * send without the Relaxed Ordering Attribute thus guaranteeing that + * all preceding PCIe Transaction Layer Packets will be processed + * first.) But some Root Complexes have various issues with Upstream + * Transaction Layer Packets with the Relaxed Ordering Attribute set. + * The PCIe devices which under the Root Complexes will be cleared the + * Relaxed Ordering bit in the configuration space, So we check our + * PCIe configuration space to see if it's flagged with advice against + * using Relaxed Ordering. + */ + if (!pcie_relaxed_ordering_supported(pdev)) + adapter->flags |= ROOT_NO_RELAXED_ORDERING; + err = adap_init0(adapter); if (err) goto err_unmap_bar; diff --git a/drivers/net/ethernet/chelsio/cxgb4vf/sge.c b/drivers/net/ethernet/chelsio/cxgb4vf/sge.c index e37dde2..05498e7 100644 --- a/drivers/net/ethernet/chelsio/cxgb4vf/sge.c +++ b/drivers/net/ethernet/chelsio/cxgb4vf/sge.c @@ -2205,6 +2205,7 @@ int t4vf_sge_alloc_rxq(struct adapter *adapter, struct sge_rspq *rspq, struct port_info *pi = netdev_priv(dev); struct fw_iq_cmd cmd, rpl; int ret, iqandst, flsz = 0; + int relaxed = !(adapter->flags & ROOT_NO_RELAXED_ORDERING); /* * If we're using MSI interrupts and we're not initializing the @@ -2300,6 +2301,8 @@ int t4vf_sge_alloc_rxq(struct adapter *adapter, struct sge_rspq *rspq, cpu_to_be32( FW_IQ_CMD_FL0HOSTFCMODE_V(SGE_HOSTFCMODE_NONE) | FW_IQ_CMD_FL0PACKEN_F | + FW_IQ_CMD_FL0FETCHRO_V(relaxed) | + FW_IQ_CMD_FL0DATARO_V(relaxed) | FW_IQ_CMD_FL0PADEN_F); /* In T6, for egress queue type FL there is internal overhead