From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1756220AbdGLJsr (ORCPT ); Wed, 12 Jul 2017 05:48:47 -0400 Received: from szxga01-in.huawei.com ([45.249.212.187]:9743 "EHLO szxga01-in.huawei.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1751447AbdGLJso (ORCPT ); Wed, 12 Jul 2017 05:48:44 -0400 Subject: Re: [PATCH v6 0/3] Add new PCI_DEV_FLAGS_NO_RELAXED_ORDERING flag To: Casey Leedom , Alexander Duyck References: <1498133721-21152-1-git-send-email-dingtianhong@huawei.com> CC: "ashok.raj@intel.com" , "helgaas@kernel.org" , Michael Werner , Ganesh GR , "asit.k.mallick@intel.com" , "patrick.j.cramer@intel.com" , "Suravee.Suthikulpanit@amd.com" , "Bob.Shaw@amd.com" , "l.stach@pengutronix.de" , "amira@mellanox.com" , "gabriele.paoloni@huawei.com" , "David.Laight@aculab.com" , "jeffrey.t.kirsher@intel.com" , "catalin.marinas@arm.com" , "will.deacon@arm.com" , "mark.rutland@arm.com" , "robin.murphy@arm.com" , "davem@davemloft.net" , "linux-arm-kernel@lists.infradead.org" , "netdev@vger.kernel.org" , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" From: Ding Tianhong Message-ID: <16d18af6-f578-ef8a-a52e-46ec975e59e0@huawei.com> Date: Wed, 12 Jul 2017 17:46:40 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset="windows-1252" Content-Transfer-Encoding: 7bit X-Originating-IP: [10.177.23.32] X-CFilter-Loop: Reflected X-Mirapoint-Virus-RAPID-Raw: score=unknown(0), refid=str=0001.0A020202.5965F02E.00F6,ss=1,re=0.000,recu=0.000,reip=0.000,cl=1,cld=1,fgs=0, ip=0.0.0.0, so=2014-11-16 11:51:01, dmn=2013-03-21 17:37:32 X-Mirapoint-Loop-Id: 4ff835e08fba1b589b094d65d695f4ae Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 2017/7/11 8:01, Casey Leedom wrote: > > Hey Alexander, > > Okay, I understand your point regarding the "most likely scenario" being > TLPs directed upstream to the Root Complex. But I'd still like to make sure > that we have an agreed upon API/methodology for doing Peer-to-Peer with > Relaxed Ordering and no Relaxed Ordering to the Root Complex. I don't see > how the proposed APIs can be used in that fashion. > > Right now the proposed change for cxgb4 is for it to test its own PCIe > Capability Device Control[Relaxed Ordering Enable] in order to use that > information to program the Chelsio Hardware to emit/not emit upstream TLPs > with the Relaxed Ordering Attribute set. But if we're going to have the > mixed mode situation I describe, the PCIe Capability Device Control[Relaxed > Ordering Enable] will have to be set which means that we'll be programming > the Chelsio Hardware to send upstream TLPs with Relaxed Ordering Enable to > the Root Complex which is what we were trying to avoid in the first place ... > > [[ And, as I noted on Friday evening, the currect cxgb4 Driver hardwires > the Relaxed Ordering Enable on early dureing device probe, so that > would minimally need to be addressed even if we decide that we don't > ever want to support mixed mode Relaxed Ordering. ]] > > We need some method of telling the Chelsio Driver that it should/shouldn't > use Relaxed Ordering with TLPs directed at the Root Complex. And the same > is true for a Peer PCIe Device. > > It may be that we should approach this from the completely opposite > direction and instead of having quirks which identify problematic devices, > have quirks which identify devices which would benefit from the use of > Relaxed Ordering (if the sending device supports that). That is, assume the > using Relaxed Ordering shouldn't be done unless the target device says "I > love Relaxed Ordering TLPs" ... In such a world, an NVMe or a Graphics > device might declare love of Relaxed Ordering and the same for a SPARC Root > Complex (I think that was your example). > > By the way, the sole example of Data Corruption with Relaxed Ordering is > the AMD A1100 ARM SoC and AMD appears to have given up on that almost as > soon as it was released. So what we're left with currently is a performance > problem on modern Intel CPUs ... (And hopefully we'll get a Technical > Publication on that issue fairly soon.) > > Casey > Hi Casey: After the long discuss, I think If the PCIe Capability Device Control[Relaxed Ordering Enable] to be cleared when the platform's RC has some problematic for RO didn't break anything in your driver, I think you could choose to check the (!pci_dev_should_disable_relaxed_ordering(root)) in the code to to enable ROOT_NO_RELAXED_ORDERING for your adapter, and enable the PCIe Capability Device Control [Relaxed Ordering Enable] bit when you need it, I think we don't have much gap here. And we could leave the pear-to-pear situation to be fixed later. Thanks Ding > . >