Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported

From: Casey Leedom <leedom@chelsio.com>
To: "Raj, Ashok" <ashok.raj@intel.com>
Cc: Ding Tianhong <dingtianhong@huawei.com>,
	Alexander Duyck <alexander.duyck@gmail.com>,
	Alex Williamson <alex.williamson@redhat.com>,
	Sinan Kaya <okaya@codeaurora.org>,
	"bhelgaas@google.com" <bhelgaas@google.com>,
	"helgaas@kernel.org" <helgaas@kernel.org>,
	"Michael Werner" <werner@chelsio.com>,
	Ganesh GR <ganeshgr@chelsio.com>,
	"asit.k.mallick@intel.com" <asit.k.mallick@intel.com>,
	"patrick.j.cramer@intel.com" <patrick.j.cramer@intel.com>,
	"Suravee.Suthikulpanit@amd.com" <Suravee.Suthikulpanit@amd.com>,
	"Bob.Shaw@amd.com" <Bob.Shaw@amd.com>,
	"l.stach@pengutronix.de" <l.stach@pengutronix.de>,
	"amira@mellanox.com" <amira@mellanox.com>,
	"gabriele.paoloni@huawei.com" <gabriele.paoloni@huawei.com>,
	"David.Laight@aculab.com" <David.Laight@aculab.com>,
	"jeffrey.t.kirsher@intel.com" <jeffrey.t.kirsher@intel.com>,
	"catalin.marinas@arm.com" <catalin.marinas@arm.com>,
	"will.deacon@arm.com" <will.deacon@arm.com>,
	"mark.rutland@arm.com" <mark.rutland@arm.com>,
	"robin.murphy@arm.com" <robin.murphy@arm.com>,
	"davem@davemloft.net" <davem@davemloft.net>,
	"linux-arm-kernel@lists.infradead.org" 
	<linux-arm-kernel@lists.infradead.org>,
	"netdev@vger.kernel.org" <netdev@vger.kernel.org>,
	"linux-pci@vger.kernel.org" <linux-pci@vger.kernel.org>,
	"linux-kernel@vger.kernel.org" <linux-kernel@vger.kernel.org>,
	"linuxarm@huawei.com" <linuxarm@huawei.com>
Subject: Re: [PATCH v7 2/3] PCI: Enable PCIe Relaxed Ordering if supported
Date: Fri, 4 Aug 2017 20:20:37 +0000	[thread overview]
Message-ID: <MWHPR12MB1600400EC7DEE92E4A3ADC5BC8B60@MWHPR12MB1600.namprd12.prod.outlook.com> (raw)
In-Reply-To: <20170803083153.GB4883@otc-nc-03>

| From: Raj, Ashok <ashok.raj@intel.com>
| Sent: Thursday, August 3, 2017 1:31 AM
|
| I don't understand this completely.. So your driver would know not to send
| RO TLP's to root complex. But you want to send RO to the NVMe device? This
| is the peer-2-peer case correct?

Yes, this is the "heavy hammer" issue which you alluded to later.  There are
applications where a device will want to send TLPs to a Root Complex without
Relaxed Ordering set, but will want to use it when sending TLPs to a Peer
device (say, an NVMe storage device).  The current approach doesn't make
that easy ... and in fact, I still don't kow how to code a solution for this
with the proposed APIs.  This means that we may be trading off one
performance problem for another and that Relaxed Ordering may be doomed for
use under Linux for the foreseeable future.

As I've noted a number of times, it would be great if the Intel Hardware
Engineers who attempted to implement the Relaxed Ordering semantics in the
current generation of Root Complexes had left the ability to turn off the
logic which is obviously not working.  If there was a way to disable the
logic via an undocumented register, then we could have the Linux PCI Quirk
do that.  Since Relaxed Ordering is just a hint, it's completely legitimate
to completely ignore it.

Casey