From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752337AbbFZQZj (ORCPT ); Fri, 26 Jun 2015 12:25:39 -0400 Received: from stargate.chelsio.com ([67.207.112.58]:13254 "EHLO stargate.chelsio.com" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752235AbbFZQZW convert rfc822-to-8bit (ORCPT ); Fri, 26 Jun 2015 12:25:22 -0400 From: Casey Leedom To: Benjamin Herrenschmidt CC: Arnd Bergmann , "Luis R. Rodriguez" , "Michael S. Tsirkin" , Bjorn Helgaas , Toshi Kani , Andy Lutomirski , Juergen Gross , Tomi Valkeinen , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xensource.com" , linux-fbdev , Suresh Siddha , "Ingo Molnar" , Thomas Gleixner , Daniel Vetter , Dave Airlie , Antonino Daplas , Jean-Christophe Plagniol-Villard , Dave Hansen , "venkatesh.pallipadi@intel.com" , Stefan Bader , "ville.syrjala@linux.intel.com" , David Vrabel , "Jan Beulich" , =?iso-8859-1?Q?Roger_Pau_Monn=E9?= Subject: RE: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants Thread-Topic: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants Thread-Index: AQHQqt5XUp1Z32lNNkq4qonErcNaKZ27K1mAgAEsoQD//+0Av4AAiD+AgACGiOmAANs3gP//mDW7gADAn4CAAHlsIg== Date: Fri, 26 Jun 2015 16:24:27 +0000 Message-ID: <4985EFDD773FCB459EF7915D2A3621ADC03621@nice.asicdesigners.com> References: <1434751712-24333-1-git-send-email-mcgrof@do-not-panic.com> <1435189081.3790.24.camel@kernel.crashing.org> <4985EFDD773FCB459EF7915D2A3621ADC02F10@nice.asicdesigners.com> ,<6806026.xb91q6Ad7G@wuerfel> <4985EFDD773FCB459EF7915D2A3621ADC031F8@nice.asicdesigners.com>,<1435284123.3822.24.camel@kernel.crashing.org> In-Reply-To: <1435284123.3822.24.camel@kernel.crashing.org> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [67.207.112.58] Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Thanks for looking into this Ben. As it stands now, it seems as if Write Combined mappings simply aren't supported and/or all driver writers trying to utilize Write Combined mappings have to "hand roll" their own solutions which really isn't supportable. One thing that might be considered is simply to treat the desire to utilize the Write Combining hardware as a separate issue and develop writel_wc(), writeq_wc(), etc. Those could be defined as legal only for Write Combined Mappings and would "do the right thing" for each architecture. The initial implementations could simply be writel(), writeq(), etc. till each architecture'si ssues were investigated and understood. Additionally, it would be very useful to have some sort of predicate which could be used to determine if an architecture supported Write Combining. Our drivers would use this to eliminate a wasteful attempt to use write combining on those architectures where it didn't work. Casey ________________________________________ From: Benjamin Herrenschmidt [benh@kernel.crashing.org] Sent: Thursday, June 25, 2015 7:02 PM To: Casey Leedom Cc: Arnd Bergmann; Luis R. Rodriguez; Michael S. Tsirkin; Bjorn Helgaas; Toshi Kani; Andy Lutomirski; Juergen Gross; Tomi Valkeinen; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; xen-devel@lists.xensource.com; linux-fbdev; Suresh Siddha; Ingo Molnar; Thomas Gleixner; Daniel Vetter; Dave Airlie; Antonino Daplas; Jean-Christophe Plagniol-Villard; Dave Hansen; venkatesh.pallipadi@intel.com; Stefan Bader; ville.syrjala@linux.intel.com; David Vrabel; Jan Beulich; Roger Pau Monné Subject: Re: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants On Thu, 2015-06-25 at 21:40 +0000, Casey Leedom wrote: > > Ah, thanks. I see now that the __raw_*() APIs don't do any of the > Endian Swizzling. Unfortunately the *_relaxed() APIs on PowerPC > are just defined as the normal *() routines. From > arch/powerpc/include/asm/io.h: > > /* > * We don't do relaxed operations yet, at least not with this > semantic > */ Yes so I was looking at this but there are some difficulties. Architecturally, even with I=1 G=1 mappings (normal ioremap), we have no guarantee of ordering of load vs. store unless I misunderstood something. I think all current implementations provide some of that but without barriers in the accessors, we aren't architecturally correct. However, having those barriers will cause issues with G=0 (write combine). It's unclear whether eieio() will provide the required ordering for I=1 G=0 mappings and it will probably break write combine. I'm looking into it with our HW guys and will try to come up with a solution for power, but it doesn't help that our memory model conflates write combining with other relaxations and that all our barriers also prevent write combine. Maybe we can bias the relaxed accessors toward write, by having no barriers in it, and putting extra ones in reads. Cheers, Ben. From mboxrd@z Thu Jan 1 00:00:00 1970 From: Casey Leedom Subject: RE: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants Date: Fri, 26 Jun 2015 16:24:27 +0000 Message-ID: <4985EFDD773FCB459EF7915D2A3621ADC03621@nice.asicdesigners.com> References: <1434751712-24333-1-git-send-email-mcgrof@do-not-panic.com> <1435189081.3790.24.camel@kernel.crashing.org> <4985EFDD773FCB459EF7915D2A3621ADC02F10@nice.asicdesigners.com> ,<6806026.xb91q6Ad7G@wuerfel> <4985EFDD773FCB459EF7915D2A3621ADC031F8@nice.asicdesigners.com>,<1435284123.3822.24.camel@kernel.crashing.org> Mime-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: In-Reply-To: <1435284123.3822.24.camel@kernel.crashing.org> Content-Language: en-US Sender: linux-pci-owner@vger.kernel.org To: Benjamin Herrenschmidt Cc: Arnd Bergmann , "Luis R. Rodriguez" , "Michael S. Tsirkin" , Bjorn Helgaas , Toshi Kani , Andy Lutomirski , Juergen Gross , Tomi Valkeinen , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xensource.com" , linux-fbdev , Suresh Siddha , Ingo Molnar , Thomas Gleixner , Daniel Vetter , Dave Airlie , Antonino Daplas , Jean-Christophe Plagniol-Villard , Dave Hansen , "venkatesh.pallipadi@intel.com" List-Id: xen-devel@lists.xenproject.org Thanks for looking into this Ben. As it stands now, it seems as if Write Combined mappings simply aren't supported and/or all driver writers trying to utilize Write Combined mappings have to "hand roll" their own solutions which really isn't supportable. One thing that might be considered is simply to treat the desire to utilize the Write Combining hardware as a separate issue and develop writel_wc(), writeq_wc(), etc. Those could be defined as legal only for Write Combined Mappings and would "do the right thing" for each architecture. The initial implementations could simply be writel(), writeq(), etc. till each architecture'si ssues were investigated and understood. Additionally, it would be very useful to have some sort of predicate which could be used to determine if an architecture supported Write Combining. Our drivers would use this to eliminate a wasteful attempt to use write combining on those architectures where it didn't work. Casey ________________________________________ =46rom: Benjamin Herrenschmidt [benh@kernel.crashing.org] Sent: Thursday, June 25, 2015 7:02 PM To: Casey Leedom Cc: Arnd Bergmann; Luis R. Rodriguez; Michael S. Tsirkin; Bjorn Helgaas= ; Toshi Kani; Andy Lutomirski; Juergen Gross; Tomi Valkeinen; linux-pci= @vger.kernel.org; linux-kernel@vger.kernel.org; xen-devel@lists.xensour= ce.com; linux-fbdev; Suresh Siddha; Ingo Molnar; Thomas Gleixner; Danie= l Vetter; Dave Airlie; Antonino Daplas; Jean-Christophe Plagniol-Villar= d; Dave Hansen; venkatesh.pallipadi@intel.com; Stefan Bader; ville.syrj= ala@linux.intel.com; David Vrabel; Jan Beulich; Roger Pau Monn=E9 Subject: Re: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants On Thu, 2015-06-25 at 21:40 +0000, Casey Leedom wrote: > > Ah, thanks. I see now that the __raw_*() APIs don't do any of the > Endian Swizzling. Unfortunately the *_relaxed() APIs on PowerPC > are just defined as the normal *() routines. From > arch/powerpc/include/asm/io.h: > > /* > * We don't do relaxed operations yet, at least not with this > semantic > */ Yes so I was looking at this but there are some difficulties. Architecturally, even with I=3D1 G=3D1 mappings (normal ioremap), we ha= ve no guarantee of ordering of load vs. store unless I misunderstood something. I think all current implementations provide some of that but without barriers in the accessors, we aren't architecturally correct. However, having those barriers will cause issues with G=3D0 (write combine). It's unclear whether eieio() will provide the required ordering for I=3D1 G=3D0 mappings and it will probably break write comb= ine. I'm looking into it with our HW guys and will try to come up with a solution for power, but it doesn't help that our memory model conflates write combining with other relaxations and that all our barriers also prevent write combine. Maybe we can bias the relaxed accessors toward write, by having no barriers in it, and putting extra ones in reads. Cheers, Ben.