From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752487AbdHDUsn (ORCPT ); Fri, 4 Aug 2017 16:48:43 -0400 Received: from mail-cys01nam02on0124.outbound.protection.outlook.com ([104.47.37.124]:63744 "EHLO NAM02-CY1-obe.outbound.protection.outlook.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752227AbdHDUsi (ORCPT ); Fri, 4 Aug 2017 16:48:38 -0400 From: Casey Leedom To: "Raj, Ashok" CC: Ding Tianhong , Alexander Duyck , Alex Williamson , Sinan Kaya , "bhelgaas@google.com" , "helgaas@kernel.org" , "Michael Werner" , Ganesh GR , "asit.k.mallick@intel.com" , "patrick.j.cramer@intel.com" , "Suravee.Suthikulpanit@amd.com" , "Bob.Shaw@amd.com" , "l.stach@pengutronix.de" , "amira@mellanox.com" , "gabriele.paoloni@huawei.com" , "David.Laight@aculab.com" , "jeffrey.t.kirsher@intel.com" , "catalin.marinas@arm.com" , "will.deacon@arm.com" , "mark.rutland@arm.com" , "robin.murphy@arm.com" , "davem@davemloft.net" , "linux-arm-kernel@lists.infradead.org" , "netdev@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linuxarm@huawei.com" Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Thread-Topic: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported Thread-Index: AQHS++OWrGKhofvtIUCheprjRIXqU6JSQPmAgABHnoCAANEhgIAL8fcAgAPZDQCAA1q+dYAAcmoAgAEXs4CAAJoQgIAI088DgAD2rYCAAjftE4AAILmAgAACTSI= Date: Fri, 4 Aug 2017 20:48:34 +0000 Message-ID: References: <67be791f-e0cf-8284-9229-17174dc741ef@codeaurora.org> <5f9b8bfb-41a8-a17c-6fea-581aec1d5573@huawei.com> <20170724090516.2e0f0d2a@w520.home> <75213fca-4522-2297-3cb8-338e643d3552@huawei.com> <14218972-553d-eb60-0207-460ac7f4b064@huawei.com> <20170803083153.GB4883@otc-nc-03> ,<20170804202141.GA39054@otc-nc-03> In-Reply-To: <20170804202141.GA39054@otc-nc-03> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=leedom@chelsio.com; x-originating-ip: [12.32.117.8] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1;MWHPR12MB1440;6:TGktabRg3Wcywi2sJPgbT1gb/UpCzfFeQDXQgrxTnQ3mbawsHhqBYexcd4vHPax54L4lnSu85jS6yNYPDWCLwx4pzWj8N+gnNeOh1ec0ojURTnpVrTWJsb3yOVTK9+6r1sM9RUaHsm40XMLOPqi3RelFdkUsRG1OT8yEQdA5aUQ0wDcXr2/kw7pQeWgoL9wvHW23Z0Rviz6xsq1ZPfjyvOu2lb8QmmTRd8U37ygoJc+63OaBC1VG2aKCiNPjTOZuCFuceXIFdOdS56MadXmJAKxVVuq0COL1CoBbGHQr6lxnwpk5T0e9SevRoyiBT7Vy1KTIzMWg4kLuij/23+USUA==;5:cd1vsRbHAGHgCa73YVRpGI9rDvinmMxDqAs6YGwglHITVLzVfMbzgh1G06XjOaYPFEa4cE4EAdJGAsN4zYui25HCV3l9OUF7yGPc9WBrqLjBYkd+RsUUEIBsZwG59CcfZ7ih7cQnI9vdiX3jMQjOHw==;24:5TZNPRk3XyRLBR/+sCNBkwGyMx6ynPW/uZUivuCIQ0vPj43mZ0QjSiijsIWQZno5pU98qHM+0E20Qx1isBmOHRlmIGeCWm0zgopEjmDLUqo=;7:zAGZVaaN/gLPKooAyn1euMN7A47xYxmmIRH/i/CNVNOH5hjbFHfuj8GqOPxXMTOoTYf10KI79UDJwppBaKxNomkhj710B1veB09CYvSZDHaz9144jCD5eghREsdq7oTTE08PEBZSIiELZ1gNK2hpNZVB9y2M63bLwKQ08CpuwkCReQBiK3eHjzxiuTrlmN6vlb/7A1SVKERyVPO0fL3AavGU6ABZGgV6iqTiB7FNVxk= x-ms-office365-filtering-correlation-id: effb92de-0a28-49b2-7c79-08d4db7a2cb8 x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(2017082002075)(300000503095)(300135400095)(2017052603031)(201703131423075)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095);SRVR:MWHPR12MB1440; x-ms-traffictypediagnostic: MWHPR12MB1440: x-exchange-antispam-report-test: UriScan:(228905959029699); x-microsoft-antispam-prvs: x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(601004)(2401047)(8121501046)(5005006)(100000703101)(100105400095)(93006095)(93001095)(10201501046)(3002001)(6041248)(2016111802025)(20161123560025)(20161123562025)(20161123564025)(20161123558100)(20161123555025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(6072148)(6043046)(100000704101)(100105200095)(100000705101)(100105500095);SRVR:MWHPR12MB1440;BCL:0;PCL:0;RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095);SRVR:MWHPR12MB1440; x-forefront-prvs: 0389EDA07F x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(39410400002)(39400400002)(39840400002)(39450400003)(189002)(50834003)(199003)(377454003)(24454002)(2900100001)(86362001)(105586002)(551934003)(97736004)(81156014)(101416001)(93886004)(189998001)(66066001)(76176999)(54356999)(50986999)(102836003)(74316002)(106356001)(81166006)(33656002)(2906002)(8676002)(3846002)(6116002)(14454004)(6246003)(6506006)(39060400002)(110136004)(54906002)(5660300001)(68736007)(55016002)(8666007)(99286003)(9686003)(3660700001)(7696004)(77096006)(38730400002)(53936002)(3280700002)(305945005)(7416002)(2950100002)(229853002)(25786009)(8936002)(6916009)(4326008)(7736002)(6436002)(478600001);DIR:OUT;SFP:1102;SCL:1;SRVR:MWHPR12MB1440;H:MWHPR12MB1600.namprd12.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="iso-8859-1" MIME-Version: 1.0 X-OriginatorOrg: chelsio.com X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Aug 2017 20:48:34.6988 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 065db76d-a7ae-4c60-b78a-501e8fc17095 X-MS-Exchange-Transport-CrossTenantHeadersStamped: MWHPR12MB1440 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by nfs id v74Kmoda004378 | From: Raj, Ashok | Sent: Friday, August 4, 2017 1:21 PM | | On Fri, Aug 04, 2017 at 08:20:37PM +0000, Casey Leedom wrote: | > ... | > As I've noted a number of times, it would be great if the Intel Hardware | > Engineers who attempted to implement the Relaxed Ordering semantics in the | > current generation of Root Complexes had left the ability to turn off the | > logic which is obviously not working. If there was a way to disable the | > logic via an undocumented register, then we could have the Linux PCI Quirk | > do that. Since Relaxed Ordering is just a hint, it's completely legitimate | > to completely ignore it. | | Suppose you are looking for the existence of a chicken bit to instruct the | port to ignore RO traffic. So all we would do is turn the chicken bit on | but would permit p2p traffic to be allowed since we won't turn off the | capability as currently proposed. | | Let me look into that keep you posted. Huh, I'd never heard it called a "chicken bit" before, but yes, that's what I'm talking about. Whenever our Hardware Designers implement new functionality in our hardware, they almost always put in A. several "knobs" which can control fundamental parameters of the new Hardware Feature, and B. a mechanism of completely disabling it if necessary. This stems from the incredibly long Design -> Deployment cyle for Hardware (as opposed to the edit->compile->run cycle for s! It's obvious that handling Relaxed Ordering is a new Hardware Feature for Intel's Root Complexes since previous versions simply ignored it (because that's legal[1]). If I was a Hardware Engineer tasked with implementing Relaxed Ordering semantics for a device, I would certainly have also implemented a switch to turn it off in case there were unintended consequences (performance in this case). And if there is such a mechanism to simply disable processing of Relaxed Ordering semantics in the Root Complex, that would be a far easier "fix" for this problem ... and leave the code in place to continue sending Relaxed Ordering TLPs for a future Root Complex implementation which got it right ... Casey [1] One can't ~quite~ just ignore the Relaxed Ordering Attribute on an incoming Transaction Layer Packet Request: PCIe completion rules (see section 2.2.9 of the PCIe 3.0 specificatin) require that the Relaxed Ordering and No Snoop Attributes in a Request TLP be reflected back verbatim in any corresponding Response TLP. (The rules for ID-Based Ordering are more complex.)