From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752895AbbFZWCZ (ORCPT ); Fri, 26 Jun 2015 18:02:25 -0400 Received: from gate.crashing.org ([63.228.1.57]:59627 "EHLO gate.crashing.org" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752164AbbFZWCR (ORCPT ); Fri, 26 Jun 2015 18:02:17 -0400 Message-ID: <1435356048.26815.9.camel@kernel.crashing.org> Subject: Re: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants From: Benjamin Herrenschmidt To: Casey Leedom Cc: Arnd Bergmann , "Luis R. Rodriguez" , "Michael S. Tsirkin" , Bjorn Helgaas , Toshi Kani , Andy Lutomirski , Juergen Gross , Tomi Valkeinen , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xensource.com" , linux-fbdev , Suresh Siddha , Ingo Molnar , Thomas Gleixner , Daniel Vetter , Dave Airlie , Antonino Daplas , Jean-Christophe Plagniol-Villard , Dave Hansen , "venkatesh.pallipadi@intel.com" , Stefan Bader , "ville.syrjala@linux.intel.com" , David Vrabel , Jan Beulich , Roger Pau =?ISO-8859-1?Q?Monn=E9?= Date: Sat, 27 Jun 2015 08:00:48 +1000 In-Reply-To: <4985EFDD773FCB459EF7915D2A3621ADC03621@nice.asicdesigners.com> References: <1434751712-24333-1-git-send-email-mcgrof@do-not-panic.com> <1435189081.3790.24.camel@kernel.crashing.org> <4985EFDD773FCB459EF7915D2A3621ADC02F10@nice.asicdesigners.com> ,<6806026.xb91q6Ad7G@wuerfel> <4985EFDD773FCB459EF7915D2A3621ADC031F8@nice.asicdesigners.com> ,<1435284123.3822.24.camel@kernel.crashing.org> <4985EFDD773FCB459EF7915D2A3621ADC03621@nice.asicdesigners.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.12.11-0ubuntu3 Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Fri, 2015-06-26 at 16:24 +0000, Casey Leedom wrote: > Thanks for looking into this Ben. As it stands now, it seems as > if Write Combined mappings simply aren't supported and/or all > driver writers trying to utilize Write Combined mappings have to > "hand roll" their own solutions which really isn't supportable. > > One thing that might be considered is simply to treat the desire > to utilize the Write Combining hardware as a separate issue and > develop writel_wc(), writeq_wc(), etc. Those could be defined > as legal only for Write Combined Mappings and would "do the > right thing" for each architecture. The question then is what is "the right thing". In the powerpc case, we'll have a non-garded mapping, which means we also get no ordering between load and stores. > The initial implementations > could simply be writel(), writeq(), etc. till each architecture'si > ssues were investigated and understood. > > Additionally, it would be very useful to have some sort of > predicate which could be used to determine if an architecture > supported Write Combining. Our drivers would use this to > eliminate a wasteful attempt to use write combining on those > architectures where it didn't work. > > Casey > > ________________________________________ > From: Benjamin Herrenschmidt [benh@kernel.crashing.org] > Sent: Thursday, June 25, 2015 7:02 PM > To: Casey Leedom > Cc: Arnd Bergmann; Luis R. Rodriguez; Michael S. Tsirkin; Bjorn Helgaas; Toshi Kani; Andy Lutomirski; Juergen Gross; Tomi Valkeinen; linux-pci@vger.kernel.org; linux-kernel@vger.kernel.org; xen-devel@lists.xensource.com; linux-fbdev; Suresh Siddha; Ingo Molnar; Thomas Gleixner; Daniel Vetter; Dave Airlie; Antonino Daplas; Jean-Christophe Plagniol-Villard; Dave Hansen; venkatesh.pallipadi@intel.com; Stefan Bader; ville.syrjala@linux.intel.com; David Vrabel; Jan Beulich; Roger Pau Monné > Subject: Re: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants > > On Thu, 2015-06-25 at 21:40 +0000, Casey Leedom wrote: > > > > Ah, thanks. I see now that the __raw_*() APIs don't do any of the > > Endian Swizzling. Unfortunately the *_relaxed() APIs on PowerPC > > are just defined as the normal *() routines. From > > arch/powerpc/include/asm/io.h: > > > > /* > > * We don't do relaxed operations yet, at least not with this > > semantic > > */ > > Yes so I was looking at this but there are some difficulties. > Architecturally, even with I=1 G=1 mappings (normal ioremap), we have no > guarantee of ordering of load vs. store unless I misunderstood > something. I think all current implementations provide some of that but > without barriers in the accessors, we aren't architecturally correct. > > However, having those barriers will cause issues with G=0 (write > combine). It's unclear whether eieio() will provide the required > ordering for I=1 G=0 mappings and it will probably break write combine. > > I'm looking into it with our HW guys and will try to come up with a > solution for power, but it doesn't help that our memory model conflates > write combining with other relaxations and that all our barriers also > prevent write combine. > > Maybe we can bias the relaxed accessors toward write, by having no > barriers in it, and putting extra ones in reads. > > Cheers, > Ben. > > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 From: Benjamin Herrenschmidt Date: Fri, 26 Jun 2015 22:00:48 +0000 Subject: Re: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants Message-Id: <1435356048.26815.9.camel@kernel.crashing.org> List-Id: References: <1434751712-24333-1-git-send-email-mcgrof@do-not-panic.com> <1435189081.3790.24.camel@kernel.crashing.org> <4985EFDD773FCB459EF7915D2A3621ADC02F10@nice.asicdesigners.com> ,<6806026.xb91q6Ad7G@wuerfel> <4985EFDD773FCB459EF7915D2A3621ADC031F8@nice.asicdesigners.com> ,<1435284123.3822.24.camel@kernel.crashing.org> <4985EFDD773FCB459EF7915D2A3621ADC03621@nice.asicdesigners.com> In-Reply-To: <4985EFDD773FCB459EF7915D2A3621ADC03621@nice.asicdesigners.com> MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable To: Casey Leedom Cc: Arnd Bergmann , "Luis R. Rodriguez" , "Michael S. Tsirkin" , Bjorn Helgaas , Toshi Kani , Andy Lutomirski , Juergen Gross , Tomi Valkeinen , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xensource.com" , linux-fbdev , Suresh Siddha , Ingo Molnar , Thomas Gleixner , Daniel Vetter , Dave Airlie , Antonino Daplas , Jean-Christophe Plagniol-Villard , Dave Hansen , "venkatesh.pallipadi@intel.com" , Stefan Bader , "ville.syrjala@linux.intel.com" , David Vrabel , Jan Beulich , Roger Pau =?ISO-8859-1?Q?Monn=E9?= On Fri, 2015-06-26 at 16:24 +0000, Casey Leedom wrote: > Thanks for looking into this Ben. As it stands now, it seems as > if Write Combined mappings simply aren't supported and/or all > driver writers trying to utilize Write Combined mappings have to > "hand roll" their own solutions which really isn't supportable. >=20 > One thing that might be considered is simply to treat the desire > to utilize the Write Combining hardware as a separate issue and > develop writel_wc(), writeq_wc(), etc. Those could be defined > as legal only for Write Combined Mappings and would "do the > right thing" for each architecture. =20 The question then is what is "the right thing". In the powerpc case, we'll have a non-garded mapping, which means we also get no ordering between load and stores. > The initial implementations > could simply be writel(), writeq(), etc. till each architecture'si > ssues were investigated and understood. >=20 > Additionally, it would be very useful to have some sort of > predicate which could be used to determine if an architecture > supported Write Combining. Our drivers would use this to > eliminate a wasteful attempt to use write combining on those > architectures where it didn't work. >=20 > Casey >=20 > ________________________________________ > From: Benjamin Herrenschmidt [benh@kernel.crashing.org] > Sent: Thursday, June 25, 2015 7:02 PM > To: Casey Leedom > Cc: Arnd Bergmann; Luis R. Rodriguez; Michael S. Tsirkin; Bjorn Helgaas; = Toshi Kani; Andy Lutomirski; Juergen Gross; Tomi Valkeinen; linux-pci@vger.= kernel.org; linux-kernel@vger.kernel.org; xen-devel@lists.xensource.com; li= nux-fbdev; Suresh Siddha; Ingo Molnar; Thomas Gleixner; Daniel Vetter; Dave= Airlie; Antonino Daplas; Jean-Christophe Plagniol-Villard; Dave Hansen; ve= nkatesh.pallipadi@intel.com; Stefan Bader; ville.syrjala@linux.intel.com; D= avid Vrabel; Jan Beulich; Roger Pau Monn=C3=A9 > Subject: Re: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants >=20 > On Thu, 2015-06-25 at 21:40 +0000, Casey Leedom wrote: > > > > Ah, thanks. I see now that the __raw_*() APIs don't do any of the > > Endian Swizzling. Unfortunately the *_relaxed() APIs on PowerPC > > are just defined as the normal *() routines. From > > arch/powerpc/include/asm/io.h: > > > > /* > > * We don't do relaxed operations yet, at least not with this > > semantic > > */ >=20 > Yes so I was looking at this but there are some difficulties. > Architecturally, even with I=3D1 G=3D1 mappings (normal ioremap), we have= no > guarantee of ordering of load vs. store unless I misunderstood > something. I think all current implementations provide some of that but > without barriers in the accessors, we aren't architecturally correct. >=20 > However, having those barriers will cause issues with G=3D0 (write > combine). It's unclear whether eieio() will provide the required > ordering for I=3D1 G=3D0 mappings and it will probably break write combin= e. >=20 > I'm looking into it with our HW guys and will try to come up with a > solution for power, but it doesn't help that our memory model conflates > write combining with other relaxations and that all our barriers also > prevent write combine. >=20 > Maybe we can bias the relaxed accessors toward write, by having no > barriers in it, and putting extra ones in reads. >=20 > Cheers, > Ben. >=20 > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html From mboxrd@z Thu Jan 1 00:00:00 1970 From: Benjamin Herrenschmidt Subject: Re: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants Date: Sat, 27 Jun 2015 08:00:48 +1000 Message-ID: <1435356048.26815.9.camel@kernel.crashing.org> References: <1434751712-24333-1-git-send-email-mcgrof@do-not-panic.com> <1435189081.3790.24.camel@kernel.crashing.org> <4985EFDD773FCB459EF7915D2A3621ADC02F10@nice.asicdesigners.com> ,<6806026.xb91q6Ad7G@wuerfel> <4985EFDD773FCB459EF7915D2A3621ADC031F8@nice.asicdesigners.com> ,<1435284123.3822.24.camel@kernel.crashing.org> <4985EFDD773FCB459EF7915D2A3621ADC03621@nice.asicdesigners.com> Mime-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: QUOTED-PRINTABLE Return-path: In-Reply-To: <4985EFDD773FCB459EF7915D2A3621ADC03621@nice.asicdesigners.com> Sender: linux-kernel-owner@vger.kernel.org To: Casey Leedom Cc: Arnd Bergmann , "Luis R. Rodriguez" , "Michael S. Tsirkin" , Bjorn Helgaas , Toshi Kani , Andy Lutomirski , Juergen Gross , Tomi Valkeinen , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "xen-devel@lists.xensource.com" , linux-fbdev , Suresh Siddha , Ingo Molnar , Thomas Gleixner , Daniel Vetter , Dave Airlie , Antonino Daplas , Jean-Christophe Plagniol-Villard , Dave Hansen , "venkatesh.pallipadi@intel.com" List-Id: xen-devel@lists.xenproject.org On Fri, 2015-06-26 at 16:24 +0000, Casey Leedom wrote: > Thanks for looking into this Ben. As it stands now, it seems as > if Write Combined mappings simply aren't supported and/or all > driver writers trying to utilize Write Combined mappings have to > "hand roll" their own solutions which really isn't supportable. >=20 > One thing that might be considered is simply to treat the desire > to utilize the Write Combining hardware as a separate issue and > develop writel_wc(), writeq_wc(), etc. Those could be defined > as legal only for Write Combined Mappings and would "do the > right thing" for each architecture. =20 The question then is what is "the right thing". In the powerpc case, we'll have a non-garded mapping, which means we also get no ordering between load and stores. > The initial implementations > could simply be writel(), writeq(), etc. till each architecture'si > ssues were investigated and understood. >=20 > Additionally, it would be very useful to have some sort of > predicate which could be used to determine if an architecture > supported Write Combining. Our drivers would use this to > eliminate a wasteful attempt to use write combining on those > architectures where it didn't work. >=20 > Casey >=20 > ________________________________________ > From: Benjamin Herrenschmidt [benh@kernel.crashing.org] > Sent: Thursday, June 25, 2015 7:02 PM > To: Casey Leedom > Cc: Arnd Bergmann; Luis R. Rodriguez; Michael S. Tsirkin; Bjorn Helga= as; Toshi Kani; Andy Lutomirski; Juergen Gross; Tomi Valkeinen; linux-p= ci@vger.kernel.org; linux-kernel@vger.kernel.org; xen-devel@lists.xenso= urce.com; linux-fbdev; Suresh Siddha; Ingo Molnar; Thomas Gleixner; Dan= iel Vetter; Dave Airlie; Antonino Daplas; Jean-Christophe Plagniol-Vill= ard; Dave Hansen; venkatesh.pallipadi@intel.com; Stefan Bader; ville.sy= rjala@linux.intel.com; David Vrabel; Jan Beulich; Roger Pau Monn=C3=A9 > Subject: Re: [PATCH v7 5/9] PCI: Add pci_iomap_wc() variants >=20 > On Thu, 2015-06-25 at 21:40 +0000, Casey Leedom wrote: > > > > Ah, thanks. I see now that the __raw_*() APIs don't do any of the > > Endian Swizzling. Unfortunately the *_relaxed() APIs on PowerPC > > are just defined as the normal *() routines. From > > arch/powerpc/include/asm/io.h: > > > > /* > > * We don't do relaxed operations yet, at least not with this > > semantic > > */ >=20 > Yes so I was looking at this but there are some difficulties. > Architecturally, even with I=3D1 G=3D1 mappings (normal ioremap), we = have no > guarantee of ordering of load vs. store unless I misunderstood > something. I think all current implementations provide some of that b= ut > without barriers in the accessors, we aren't architecturally correct. >=20 > However, having those barriers will cause issues with G=3D0 (write > combine). It's unclear whether eieio() will provide the required > ordering for I=3D1 G=3D0 mappings and it will probably break write co= mbine. >=20 > I'm looking into it with our HW guys and will try to come up with a > solution for power, but it doesn't help that our memory model conflat= es > write combining with other relaxations and that all our barriers also > prevent write combine. >=20 > Maybe we can bias the relaxed accessors toward write, by having no > barriers in it, and putting extra ones in reads. >=20 > Cheers, > Ben. >=20 > -- > To unsubscribe from this list: send the line "unsubscribe linux-pci" = in > the body of a message to majordomo@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html