From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752097AbdHDUUn (ORCPT ); Fri, 4 Aug 2017 16:20:43 -0400 Received: from mail-sn1nam01on0110.outbound.protection.outlook.com ([104.47.32.110]:23734 "EHLO NAM01-SN1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1751208AbdHDUUl (ORCPT ); Fri, 4 Aug 2017 16:20:41 -0400 From: Casey Leedom To: "Raj, Ashok" CC: Ding Tianhong , Alexander Duyck , Alex Williamson , Sinan Kaya , "bhelgaas@google.com" , "helgaas@kernel.org" , "Michael Werner" , Ganesh GR , "asit.k.mallick@intel.com" , "patrick.j.cramer@intel.com" , "Suravee.Suthikulpanit@amd.com" , "Bob.Shaw@amd.com" , "l.stach@pengutronix.de" , "amira@mellanox.com" , "gabriele.paoloni@huawei.com" , "David.Laight@aculab.com" , "jeffrey.t.kirsher@intel.com" , "catalin.marinas@arm.com" , "will.deacon@arm.com" , "mark.rutland@arm.com" , "robin.murphy@arm.com" , "davem@davemloft.net" , "linux-arm-kernel@lists.infradead.org" , "netdev@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linuxarm@huawei.com" Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Thread-Topic: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Thread-Index: AQHS++OWrGKhofvtIUCheprjRIXqU6JSQPmAgABHnoCAANEhgIAL8fcAgAPZDQCAA1q+dYAAcmoAgAEXs4CAAJoQgIAI088DgAD2rYCAAjftEw== Date: Fri, 4 Aug 2017 20:20:37 +0000 Message-ID: References: <0260e398-bd8e-6615-6d5c-1f7c07b6fc09@huawei.com> <67be791f-e0cf-8284-9229-17174dc741ef@codeaurora.org> <5f9b8bfb-41a8-a17c-6fea-581aec1d5573@huawei.com> <20170724090516.2e0f0d2a@w520.home> <75213fca-4522-2297-3cb8-338e643d3552@huawei.com> <14218972-553d-eb60-0207-460ac7f4b064@huawei.com> ,<20170803083153.GB4883@otc-nc-03> In-Reply-To: <20170803083153.GB4883@otc-nc-03> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=leedom@chelsio.com; x-originating-ip: [12.32.117.8] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;MWHPR12MB1440;6:NC8+fhloK+eU+2Cjm5ZcSPR93UyhB7O6oV1K5Mt2GS/ka7qts/h2gKkwe9FXSWgAG2UQlvyhKd3KqsZMcn3YFvobmdIinkQGP4xl1gXDuCCoKI3YPccU5ebG5yki+02Fpb7DmkV4xWc8HlWUGSDMJ9LDm+pl7qMLZEvnQi6Yn4BVqtzWjyUXfdoTDfOZak+b7fynfrLJ5YuMxQb1dUVBgRz6t7ZAx/Iq8fOr/Ue32qkpqSQQulOP9fuxeHtJvaCBspWCCv/jZWSsGQPsKU59xQ1j5ByvZt0esIVQphGluBSYXmj4i3fHqx9yOFutHEvzX9Bb+5LDGIYkmCf0MMSj3w==;5:5YgjnMUAzihDZ56DYXFbemw9J0B5iUmzZ4ph0EgyYSQvYWtDPioFtkpyA2V2+ofA375DaLUwD790P03f5qmp272kQAKnu/7a8Nqm1aW5046UgGKj30FA4D9DQJCoTu6/tI5rHBaHz68cclXnEVZ00A==;24:x59jd4XjJKJ1cjFZCkajdCQ4M3K7Z2yaaLwsgp8OtUy+yvAW736KFAP2PUjcaUkOgHZRHZC+TEmKYhPf4OM22fTkHilMkOaOKQY6PkPvpa8=;7:MqNyM5YrGbkVvYOgwh/U4gA5iAS9nRWtY097vO7h6XL6ZXMKHpr+3dHmYfS5bzBPco47JeMm+j9QzwP21rPo8g2vTFZAnf6GP4kdsBGARD57wJ+c3ZRoJeQMO43AxPGj/oc3FP+gTlzPxsRYsZusNoIxo5kPR72lFc8UBcEDa4KkVMlPJ7P/M7/9z62PX+ejNIpdy1ksh2eOZ7cfADvdTtskZNyi8wOKwXG4+ggi+gI= x-ms-office365-filtering-correlation-id: 9252558c-69b5-4365-4820-08d4db764507 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(2017082002075)(300000503095)(300135400095)(2017052603031)(201703131423075)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:MWHPR12MB1440; x-ms-traffictypediagnostic: MWHPR12MB1440: x-exchange-antispam-report-test: UriScan:(278428928389397)(228905959029699); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(5005006)(8121501046)(100000703101)(100105400095)(3002001)(10201501046)(93006095)(93001095)(6041248)(2016111802025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123558100)(20161123564025)(20161123562025)(20161123560025)(6072148)(6043046)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:MWHPR12MB1440;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:MWHPR12MB1440; x-forefront-prvs: 0389EDA07F x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(39840400002)(39400400002)(39410400002)(39450400003)(189002)(199003)(377454003)(2900100001)(86362001)(105586002)(97736004)(81156014)(101416001)(93886004)(189998001)(66066001)(54356999)(76176999)(50986999)(102836003)(74316002)(106356001)(33656002)(81166006)(2906002)(8676002)(3846002)(6116002)(6246003)(14454004)(6506006)(39060400002)(110136004)(54906002)(5660300001)(68736007)(55016002)(8666007)(99286003)(9686003)(3660700001)(7696004)(77096006)(38730400002)(53936002)(3280700002)(305945005)(7416002)(2950100002)(229853002)(25786009)(8936002)(6916009)(4326008)(7736002)(6436002)(478600001);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR12MB1440;H:MWHPR12MB1600.namprd12.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 X-OriginatorOrg: chelsio.com X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Aug 2017 20:20:37.6030 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 065db76d-a7ae-4c60-b78a-501e8fc17095 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1440 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id v74KKsZp000702 | From: Raj, Ashok | Sent: Thursday, August 3, 2017 1:31 AM | | I don't understand this completely.. So your driver would know not to send | RO TLP's to root complex. But you want to send RO to the NVMe device? This | is the peer-2-peer case correct? Yes, this is the "heavy hammer" issue which you alluded to later. There are applications where a device will want to send TLPs to a Root Complex without Relaxed Ordering set, but will want to use it when sending TLPs to a Peer device (say, an NVMe storage device). The current approach doesn't make that easy ... and in fact, I still don't kow how to code a solution for this with the proposed APIs. This means that we may be trading off one performance problem for another and that Relaxed Ordering may be doomed for use under Linux for the foreseeable future. As I've noted a number of times, it would be great if the Intel Hardware Engineers who attempted to implement the Relaxed Ordering semantics in the current generation of Root Complexes had left the ability to turn off the logic which is obviously not working. If there was a way to disable the logic via an undocumented register, then we could have the Linux PCI Quirk do that. Since Relaxed Ordering is just a hint, it's completely legitimate to completely ignore it. Casey From mboxrd@z Thu Jan 1 00:00:00 1970 From: Casey Leedom Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Date: Fri, 4 Aug 2017 20:20:37 +0000 Message-ID: References: <0260e398-bd8e-6615-6d5c-1f7c07b6fc09@huawei.com> <67be791f-e0cf-8284-9229-17174dc741ef@codeaurora.org> <5f9b8bfb-41a8-a17c-6fea-581aec1d5573@huawei.com> <20170724090516.2e0f0d2a@w520.home> <75213fca-4522-2297-3cb8-338e643d3552@huawei.com> <14218972-553d-eb60-0207-460ac7f4b064@huawei.com> ,<20170803083153.GB4883@otc-nc-03> Mime-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Cc: Ding Tianhong , Alexander Duyck , Alex Williamson , Sinan Kaya , "bhelgaas@google.com" , "helgaas@kernel.org" , "Michael Werner" , Ganesh GR , "asit.k.mallick@intel.com" , "patrick.j.cramer@intel.com" , "Suravee.Suthikulpanit@amd.com" , "Bob.Shaw@amd.com" , "l.stach@pengutronix.de" , "amira@mellanox.com" , "gabriele.paoloni@huawei.com" , "David.Laight@aculab.com" , "jeffrey.t.kir To: "Raj, Ashok" Return-path: In-Reply-To: <20170803083153.GB4883@otc-nc-03> Content-Language: en-US Sender: linux-kernel-owner@vger.kernel.org List-Id: netdev.vger.kernel.org | From: Raj, Ashok | Sent: Thursday, August 3, 2017 1:31 AM | | I don't understand this completely.. So your driver would know not to sen= d | RO TLP's to root complex. But you want to send RO to the NVMe device? Thi= s | is the peer-2-peer case correct? Yes, this is the "heavy hammer" issue which you alluded to later. There ar= e applications where a device will want to send TLPs to a Root Complex withou= t Relaxed Ordering set, but will want to use it when sending TLPs to a Peer device (say, an NVMe storage device). The current approach doesn't make that easy ... and in fact, I still don't kow how to code a solution for thi= s with the proposed APIs. This means that we may be trading off one performance problem for another and that Relaxed Ordering may be doomed for use under Linux for the foreseeable future. As I've noted a number of times, it would be great if the Intel Hardware Engineers who attempted to implement the Relaxed Ordering semantics in the current generation of Root Complexes had left the ability to turn off the logic which is obviously not working. If there was a way to disable the logic via an undocumented register, then we could have the Linux PCI Quirk do that. Since Relaxed Ordering is just a hint, it's completely legitimate to completely ignore it. Casey From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Return-Path: From: Casey Leedom To: "Raj, Ashok" Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Date: Fri, 4 Aug 2017 20:20:37 +0000 Message-ID: References: <0260e398-bd8e-6615-6d5c-1f7c07b6fc09@huawei.com> <67be791f-e0cf-8284-9229-17174dc741ef@codeaurora.org> <5f9b8bfb-41a8-a17c-6fea-581aec1d5573@huawei.com> <20170724090516.2e0f0d2a@w520.home> <75213fca-4522-2297-3cb8-338e643d3552@huawei.com> <14218972-553d-eb60-0207-460ac7f4b064@huawei.com> , <20170803083153.GB4883@otc-nc-03> In-Reply-To: <20170803083153.GB4883@otc-nc-03> MIME-Version: 1.0 List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: "mark.rutland@arm.com" , "gabriele.paoloni@huawei.com" , "asit.k.mallick@intel.com" , "catalin.marinas@arm.com" , "will.deacon@arm.com" , "linuxarm@huawei.com" , Alexander Duyck , Sinan Kaya , "amira@mellanox.com" , "helgaas@kernel.org" , "jeffrey.t.kirsher@intel.com" , Ding Tianhong , Ganesh GR , "Bob.Shaw@amd.com" , "patrick.j.cramer@intel.com" , Alex Williamson , "bhelgaas@google.com" , Michael Werner , "linux-arm-kernel@lists.infradead.org" , "linux-pci@vger.kernel.org" , "netdev@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "David.Laight@aculab.com" , "Suravee.Suthikulpanit@amd.com" , "robin.murphy@arm.com" , "davem@davemloft.net" , "l.stach@pengutronix.de" Content-Type: text/plain; charset="us-ascii" Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+bjorn=helgaas.com@lists.infradead.org List-ID: | From: Raj, Ashok | Sent: Thursday, August 3, 2017 1:31 AM | | I don't understand this completely.. So your driver would know not to send | RO TLP's to root complex. But you want to send RO to the NVMe device? This | is the peer-2-peer case correct? Yes, this is the "heavy hammer" issue which you alluded to later. There are applications where a device will want to send TLPs to a Root Complex without Relaxed Ordering set, but will want to use it when sending TLPs to a Peer device (say, an NVMe storage device). The current approach doesn't make that easy ... and in fact, I still don't kow how to code a solution for this with the proposed APIs. This means that we may be trading off one performance problem for another and that Relaxed Ordering may be doomed for use under Linux for the foreseeable future. As I've noted a number of times, it would be great if the Intel Hardware Engineers who attempted to implement the Relaxed Ordering semantics in the current generation of Root Complexes had left the ability to turn off the logic which is obviously not working. If there was a way to disable the logic via an undocumented register, then we could have the Linux PCI Quirk do that. Since Relaxed Ordering is just a hint, it's completely legitimate to completely ignore it. Casey _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel From mboxrd@z Thu Jan 1 00:00:00 1970 From: leedom@chelsio.com (Casey Leedom) Date: Fri, 4 Aug 2017 20:20:37 +0000 Subject: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported In-Reply-To: <20170803083153.GB4883@otc-nc-03> References: <0260e398-bd8e-6615-6d5c-1f7c07b6fc09@huawei.com> <67be791f-e0cf-8284-9229-17174dc741ef@codeaurora.org> <5f9b8bfb-41a8-a17c-6fea-581aec1d5573@huawei.com> <20170724090516.2e0f0d2a@w520.home> <75213fca-4522-2297-3cb8-338e643d3552@huawei.com> <14218972-553d-eb60-0207-460ac7f4b064@huawei.com> , <20170803083153.GB4883@otc-nc-03> Message-ID: To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org | From: Raj, Ashok | Sent: Thursday, August 3, 2017 1:31 AM | | I don't understand this completely.. So your driver would know not to send | RO TLP's to root complex. But you want to send RO to the NVMe device? This | is the peer-2-peer case correct? Yes, this is the "heavy hammer" issue which you alluded to later. There are applications where a device will want to send TLPs to a Root Complex without Relaxed Ordering set, but will want to use it when sending TLPs to a Peer device (say, an NVMe storage device). The current approach doesn't make that easy ... and in fact, I still don't kow how to code a solution for this with the proposed APIs. This means that we may be trading off one performance problem for another and that Relaxed Ordering may be doomed for use under Linux for the foreseeable future. As I've noted a number of times, it would be great if the Intel Hardware Engineers who attempted to implement the Relaxed Ordering semantics in the current generation of Root Complexes had left the ability to turn off the logic which is obviously not working. If there was a way to disable the logic via an undocumented register, then we could have the Linux PCI Quirk do that. Since Relaxed Ordering is just a hint, it's completely legitimate to completely ignore it. Casey