From mboxrd@z Thu Jan 1 00:00:00 1970 From: Jason Gunthorpe Subject: Re: RFC on writel and writel_relaxed Date: Wed, 28 Mar 2018 10:57:32 -0600 Message-ID: <20180328165732.GA4546@ziepe.ca> References: <20180327110258.GF2464@arm.com> <20180327143628.GA10642@arm.com> <1522186185.7364.59.camel@kernel.crashing.org> <20180328085338.GA28871@arm.com> <1522230616.21446.1.camel@kernel.crashing.org> <1522231287.21446.9.camel@kernel.crashing.org> <20180328101345.GA30850@arm.com> Mime-Version: 1.0 Content-Type: text/plain; charset=us-ascii Return-path: Content-Disposition: inline In-Reply-To: <20180328101345.GA30850@arm.com> List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: linuxppc-dev-bounces+glppe-linuxppc-embedded-2=m.gmane.org@lists.ozlabs.org Sender: "Linuxppc-dev" To: Will Deacon Cc: Arnd Bergmann , Jonathan Corbet , "linux-rdma@vger.kernel.org" , Sinan Kaya , Peter Zijlstra , David Laight , Oliver , "Paul E. McKenney" , "open list:LINUX FOR POWERPC (32-BIT AND 64-BIT)" , Ingo Molnar List-Id: linux-rdma@vger.kernel.org On Wed, Mar 28, 2018 at 11:13:45AM +0100, Will Deacon wrote: > On Wed, Mar 28, 2018 at 09:01:27PM +1100, Benjamin Herrenschmidt wrote: > > On Wed, 2018-03-28 at 11:55 +0200, Arnd Bergmann wrote: > > > > powerpc and ARM can't quite make them synchronous I think, but at least > > > > they should have the same semantics as writel. > > > > > > One thing that ARM does IIRC is that it only guarantees to order writel() within > > > one device, and the memory mapped PCI I/O space window almost certainly > > > counts as a separate device to the CPU. > > > > That sounds bogus. > > To elaborate, if you do the following on arm: > > writel(DEVICE_FOO); > writel(DEVICE_BAR); > > we generally cannot guarantee in which order those accesses will hit the > devices even if we add every barrier under the sun. You'd need something > in between, specific to DEVICE_FOO (probably a read-back) to really push > the first write out. This doesn't sound like it would be that uncommon to > me. The PCI posted write does not require the above to execute 'in order' only that any bus segment shared by the two devices have the writes issued in CPU order. ie at a shared PCI root port for instance. If I recall this is very similar to the ordering that ARM's on-chip AXI interconnect is supposed to provide.. So I'd be very surprised if a modern ARM64 has an meaningful difference from x86 here. When talking about ordering between the devices, the relevant question is what happens if the writel(DEVICE_BAR) triggers DEVICE_BAR to DMA from the DEVICE_FOO. 'ordered' means that in this case writel(DEVICE_FOO) must be presented to FOO before anything generated by BAR. Jason From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-x234.google.com (mail-wm0-x234.google.com [IPv6:2a00:1450:400c:c09::234]) (using TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 (128/128 bits)) (No client certificate requested) by lists.ozlabs.org (Postfix) with ESMTPS id 40BDZt6s5hzF2LS for ; Thu, 29 Mar 2018 03:57:41 +1100 (AEDT) Received: by mail-wm0-x234.google.com with SMTP id x4so6583033wmh.5 for ; Wed, 28 Mar 2018 09:57:41 -0700 (PDT) Date: Wed, 28 Mar 2018 10:57:32 -0600 From: Jason Gunthorpe To: Will Deacon Cc: Benjamin Herrenschmidt , Arnd Bergmann , Sinan Kaya , David Laight , Oliver , "open list:LINUX FOR POWERPC (32-BIT AND 64-BIT)" , "linux-rdma@vger.kernel.org" , "Paul E. McKenney" , Peter Zijlstra , Ingo Molnar , Jonathan Corbet Subject: Re: RFC on writel and writel_relaxed Message-ID: <20180328165732.GA4546@ziepe.ca> References: <20180327110258.GF2464@arm.com> <20180327143628.GA10642@arm.com> <1522186185.7364.59.camel@kernel.crashing.org> <20180328085338.GA28871@arm.com> <1522230616.21446.1.camel@kernel.crashing.org> <1522231287.21446.9.camel@kernel.crashing.org> <20180328101345.GA30850@arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii In-Reply-To: <20180328101345.GA30850@arm.com> List-Id: Linux on PowerPC Developers Mail List List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , On Wed, Mar 28, 2018 at 11:13:45AM +0100, Will Deacon wrote: > On Wed, Mar 28, 2018 at 09:01:27PM +1100, Benjamin Herrenschmidt wrote: > > On Wed, 2018-03-28 at 11:55 +0200, Arnd Bergmann wrote: > > > > powerpc and ARM can't quite make them synchronous I think, but at least > > > > they should have the same semantics as writel. > > > > > > One thing that ARM does IIRC is that it only guarantees to order writel() within > > > one device, and the memory mapped PCI I/O space window almost certainly > > > counts as a separate device to the CPU. > > > > That sounds bogus. > > To elaborate, if you do the following on arm: > > writel(DEVICE_FOO); > writel(DEVICE_BAR); > > we generally cannot guarantee in which order those accesses will hit the > devices even if we add every barrier under the sun. You'd need something > in between, specific to DEVICE_FOO (probably a read-back) to really push > the first write out. This doesn't sound like it would be that uncommon to > me. The PCI posted write does not require the above to execute 'in order' only that any bus segment shared by the two devices have the writes issued in CPU order. ie at a shared PCI root port for instance. If I recall this is very similar to the ordering that ARM's on-chip AXI interconnect is supposed to provide.. So I'd be very surprised if a modern ARM64 has an meaningful difference from x86 here. When talking about ordering between the devices, the relevant question is what happens if the writel(DEVICE_BAR) triggers DEVICE_BAR to DMA from the DEVICE_FOO. 'ordered' means that in this case writel(DEVICE_FOO) must be presented to FOO before anything generated by BAR. Jason