From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S965609AbbKDPTV (ORCPT ); Wed, 4 Nov 2015 10:19:21 -0500 Received: from relmlor3.renesas.com ([210.160.252.173]:33554 "EHLO relmlie2.idc.renesas.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S932373AbbKDPTU (ORCPT ); Wed, 4 Nov 2015 10:19:20 -0500 X-IronPort-AV: E=Sophos;i="5.20,243,1444662000"; d="scan'208";a="197841831" From: Phil Edworthy To: "Liviu.Dudau@arm.com" CC: "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , Bjorn Helgaas , Arnd Bergmann , Lorenzo Pieralisi , Magnus Subject: RE: PCIe host controller behind IOMMU on ARM Thread-Topic: PCIe host controller behind IOMMU on ARM Thread-Index: AdEXBXpsqoonZ+nqRTu1yvYRg99bGAABv/qAAAApWPAAASaHgAAAPSUw Date: Wed, 4 Nov 2015 15:19:13 +0000 Message-ID: References: <20151104142412.GS963@e106497-lin.cambridge.arm.com> <20151104150147.GT963@e106497-lin.cambridge.arm.com> In-Reply-To: <20151104150147.GT963@e106497-lin.cambridge.arm.com> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=phil.edworthy@renesas.com; x-originating-ip: [193.141.220.21] x-microsoft-exchange-diagnostics: 1;PS1PR06MB1178;5:yQbvbnZoDFQ/d6Wed38CmzFXxFpKk03nDRrD/D+pJ3UYjTu3kxUSTtzWwND199buVYNbJ0L5V5vZ1yXRAJ7tWKXtrAXldEi+M/1zs1xkiRImzBJ3eDd6mN5TG73d+a0wJolObRGiAIdxm4pmxqvP1A==;24:eSe4pPSD67w8dClFrSgzR3k+hQHDgVzMez4pYPUDmYVDuoGnBB7PVlivewJR2aPIhwPwtKGnITs01c4pHwSMiN9Rs9EewwgFvxTJ+3xyfng=;20:N8PSUElTUBQkguEC61gaMM/OUAEZz02++UntxXWSiicr8rPCr+207suRnxXmrwN7oIp+p/OE4PRQhWqT8aq0AhLsivQ3g/jgXI56Ea+hJDmGit0A8bOvzz4iceE5fGoTyNA0jKkb3L/u93w0iUKK3bVdUTB+CwLxyAKUZ3J1e8k= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:PS1PR06MB1178; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(601004)(2401047)(520078)(8121501046)(5005006)(10201501046)(3002001);SRVR:PS1PR06MB1178;BCL:0;PCL:0;RULEID:;SRVR:PS1PR06MB1178; x-forefront-prvs: 0750463DC9 x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(189002)(199003)(24454002)(92566002)(11100500001)(2900100001)(10400500002)(50986999)(76176999)(66066001)(76576001)(101416001)(110136002)(5001960100002)(5002640100001)(105586002)(189998001)(40100003)(106356001)(19580395003)(74316001)(97736004)(2351001)(5007970100001)(77096005)(93886004)(87936001)(54356999)(81156007)(5004730100002)(5003600100002)(122556002)(33656002)(2501003)(86362001)(5008740100001)(2950100001)(102836002)(5890100001);DIR:OUT;SFP:1102;SCL:1;SRVR:PS1PR06MB1178;H:PS1PR06MB1180.apcprd06.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;MX:1;A:1;LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 X-OriginatorOrg: renesas.com X-MS-Exchange-CrossTenant-originalarrivaltime: 04 Nov 2015 15:19:13.6259 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 53d82571-da19-47e4-9cb4-625a166a4a2a X-MS-Exchange-Transport-CrossTenantHeadersStamped: PS1PR06MB1178 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from base64 to 8bit by mail.home.local id tA4FJRuV014033 Hi Liviu, On 04 November 2015 15:02, Liviu wrote: > On Wed, Nov 04, 2015 at 02:48:38PM +0000, Phil Edworthy wrote: > > Hi Liviu, > > > > On 04 November 2015 14:24, Liviu wrote: > > > On Wed, Nov 04, 2015 at 01:57:48PM +0000, Phil Edworthy wrote: > > > > Hi, > > > > > > > > I am trying to hook up a PCIe host controller that sits behind an IOMMU, > > > > but having some problems. > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine without > > > > the IOMMU, and I can attach the IOMMU to the controller such that any > calls > > > > to dma_alloc_coherent made by the controller driver uses the iommu_ops > > > > version of dma_ops. > > > > > > > > However, I can't see how to make the endpoints to utilise the dma_ops that > > > > the controller uses. Shouldn't the endpoints inherit the dma_ops from the > > > > controller? > > > > > > No, not directly. > > > > > > > Any pointers for this? > > > > > > You need to understand the process through which a driver for endpoint get > > > an address to be passed down to the device. Have a look at > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > > (Hint: EP driver needs to call dma_map_single). > > > > > > Also, you need to make sure that the bus address that ends up being set into > > > the endpoint gets translated correctly by the host controller into an address > > > that the IOMMU can then translate into physical address. > > Sure, though since this is bog standard Intel PCIe ethernet card which works > > fine when the IOMMU is effectively unused, I don’t think there is a problem > > with that. > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. when I > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > __iommu_alloc_buffer() and __alloc_iova(). > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > Why do you think that? Remember that the only thing attached to the IOMMU is > the > host controller. The endpoint is on the PCIe bus, which gets a different > translation > that the IOMMU knows nothing about. If it helps you to visualise it better, think > of the host controller as another IOMMU device. It's the ops of the host > controller > that should be invoked, not the IOMMU's. Ok, that makes sense. I'll have a think and poke it a bit more... Thanks for your comments Phil {.n++%ݶw{.n+{G{ayʇڙ,jfhz_(階ݢj"mG?&~iOzv^m ?I From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: From: Phil Edworthy To: "Liviu.Dudau@arm.com" CC: "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , Bjorn Helgaas , Arnd Bergmann , Lorenzo Pieralisi , Magnus Subject: RE: PCIe host controller behind IOMMU on ARM Date: Wed, 4 Nov 2015 15:19:13 +0000 Message-ID: References: <20151104142412.GS963@e106497-lin.cambridge.arm.com> <20151104150147.GT963@e106497-lin.cambridge.arm.com> In-Reply-To: <20151104150147.GT963@e106497-lin.cambridge.arm.com> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Sender: linux-kernel-owner@vger.kernel.org List-ID: SGkgTGl2aXUsDQoNCk9uIDA0IE5vdmVtYmVyIDIwMTUgMTU6MDIsIExpdml1IHdyb3RlOg0KPiBP biBXZWQsIE5vdiAwNCwgMjAxNSBhdCAwMjo0ODozOFBNICswMDAwLCBQaGlsIEVkd29ydGh5IHdy b3RlOg0KPiA+IEhpIExpdml1LA0KPiA+DQo+ID4gT24gMDQgTm92ZW1iZXIgMjAxNSAxNDoyNCwg TGl2aXUgd3JvdGU6DQo+ID4gPiBPbiBXZWQsIE5vdiAwNCwgMjAxNSBhdCAwMTo1Nzo0OFBNICsw MDAwLCBQaGlsIEVkd29ydGh5IHdyb3RlOg0KPiA+ID4gPiBIaSwNCj4gPiA+ID4NCj4gPiA+ID4g SSBhbSB0cnlpbmcgdG8gaG9vayB1cCBhIFBDSWUgaG9zdCBjb250cm9sbGVyIHRoYXQgc2l0cyBi ZWhpbmQgYW4gSU9NTVUsDQo+ID4gPiA+IGJ1dCBoYXZpbmcgc29tZSBwcm9ibGVtcy4NCj4gPiA+ ID4NCj4gPiA+ID4gSSdtIHVzaW5nIHRoZSBwY2llLXJjYXIgUENJZSBob3N0IGNvbnRyb2xsZXIg YW5kIGl0IHdvcmtzIGZpbmUgd2l0aG91dA0KPiA+ID4gPiB0aGUgSU9NTVUsIGFuZCBJIGNhbiBh dHRhY2ggdGhlIElPTU1VIHRvIHRoZSBjb250cm9sbGVyIHN1Y2ggdGhhdCBhbnkNCj4gY2FsbHMN Cj4gPiA+ID4gdG8gZG1hX2FsbG9jX2NvaGVyZW50IG1hZGUgYnkgdGhlIGNvbnRyb2xsZXIgZHJp dmVyIHVzZXMgdGhlIGlvbW11X29wcw0KPiA+ID4gPiB2ZXJzaW9uIG9mIGRtYV9vcHMuDQo+ID4g PiA+DQo+ID4gPiA+IEhvd2V2ZXIsIEkgY2FuJ3Qgc2VlIGhvdyB0byBtYWtlIHRoZSBlbmRwb2lu dHMgdG8gdXRpbGlzZSB0aGUgZG1hX29wcyB0aGF0DQo+ID4gPiA+IHRoZSBjb250cm9sbGVyIHVz ZXMuIFNob3VsZG4ndCB0aGUgZW5kcG9pbnRzIGluaGVyaXQgdGhlIGRtYV9vcHMgZnJvbSB0aGUN Cj4gPiA+ID4gY29udHJvbGxlcj8NCj4gPiA+DQo+ID4gPiBObywgbm90IGRpcmVjdGx5Lg0KPiA+ ID4NCj4gPiA+ID4gQW55IHBvaW50ZXJzIGZvciB0aGlzPw0KPiA+ID4NCj4gPiA+IFlvdSBuZWVk IHRvIHVuZGVyc3RhbmQgdGhlIHByb2Nlc3MgdGhyb3VnaCB3aGljaCBhIGRyaXZlciBmb3IgZW5k cG9pbnQgZ2V0DQo+ID4gPiBhbiBhZGRyZXNzIHRvIGJlIHBhc3NlZCBkb3duIHRvIHRoZSBkZXZp Y2UuIEhhdmUgYSBsb29rIGF0DQo+ID4gPiBEb2N1bWVudGF0aW9uL0RNQS1BUEktSE9XVE8udHh0 LCB0aGVyZSBpcyBhIG5pY2UgZXhwbGFuYXRpb24gdGhlcmUuDQo+ID4gPiAoSGludDogRVAgZHJp dmVyIG5lZWRzIHRvIGNhbGwgZG1hX21hcF9zaW5nbGUpLg0KPiA+ID4NCj4gPiA+IEFsc28sIHlv dSBuZWVkIHRvIG1ha2Ugc3VyZSB0aGF0IHRoZSBidXMgYWRkcmVzcyB0aGF0IGVuZHMgdXAgYmVp bmcgc2V0IGludG8NCj4gPiA+IHRoZSBlbmRwb2ludCBnZXRzIHRyYW5zbGF0ZWQgY29ycmVjdGx5 IGJ5IHRoZSBob3N0IGNvbnRyb2xsZXIgaW50byBhbiBhZGRyZXNzDQo+ID4gPiB0aGF0IHRoZSBJ T01NVSBjYW4gdGhlbiB0cmFuc2xhdGUgaW50byBwaHlzaWNhbCBhZGRyZXNzLg0KPiA+IFN1cmUs IHRob3VnaCBzaW5jZSB0aGlzIGlzIGJvZyBzdGFuZGFyZCBJbnRlbCBQQ0llIGV0aGVybmV0IGNh cmQgd2hpY2ggd29ya3MNCj4gPiBmaW5lIHdoZW4gdGhlIElPTU1VIGlzIGVmZmVjdGl2ZWx5IHVu dXNlZCwgSSBkb27igJl0IHRoaW5rIHRoZXJlIGlzIGEgcHJvYmxlbQ0KPiA+IHdpdGggdGhhdC4N Cj4gPg0KPiA+IFRoZSBkcml2ZXIgZm9yIHRoZSBQQ0llIGNvbnRyb2xsZXIgc2V0cyB1cCB0aGUg SU9NTVUgbWFwcGluZyBvayB3aGVuIEkNCj4gPiBkbyBhIHRlc3QgY2FsbCB0byBkbWFfYWxsb2Nf Y29oZXJlbnQoKSBpbiB0aGUgY29udHJvbGxlcidzIGRyaXZlci4gaS5lLiB3aGVuIEkNCj4gPiBk byB0aGlzLCBpdCBlbmRzIHVwIGluIGFybV9pb21tdV9hbGxvY19hdHRycygpLCB3aGljaCBjYWxs cw0KPiA+IF9faW9tbXVfYWxsb2NfYnVmZmVyKCkgYW5kIF9fYWxsb2NfaW92YSgpLg0KPiA+DQo+ ID4gV2hlbiBhbiBlbmRwb2ludCBkcml2ZXIgYWxsb2NhdGVzIGFuZCBtYXBzIGEgZG1hIGNvaGVy ZW50IGJ1ZmZlciBpdA0KPiA+IGFsc28gbmVlZHMgdG8gZW5kIHVwIGluIGFybV9pb21tdV9hbGxv Y19hdHRycygpLCBidXQgaXQgZG9lc24ndC4NCj4gDQo+IFdoeSBkbyB5b3UgdGhpbmsgdGhhdD8g UmVtZW1iZXIgdGhhdCB0aGUgb25seSB0aGluZyBhdHRhY2hlZCB0byB0aGUgSU9NTVUgaXMNCj4g dGhlDQo+IGhvc3QgY29udHJvbGxlci4gVGhlIGVuZHBvaW50IGlzIG9uIHRoZSBQQ0llIGJ1cywg d2hpY2ggZ2V0cyBhIGRpZmZlcmVudA0KPiB0cmFuc2xhdGlvbg0KPiB0aGF0IHRoZSBJT01NVSBr bm93cyBub3RoaW5nIGFib3V0LiBJZiBpdCBoZWxwcyB5b3UgdG8gdmlzdWFsaXNlIGl0IGJldHRl ciwgdGhpbmsNCj4gb2YgdGhlIGhvc3QgY29udHJvbGxlciBhcyBhbm90aGVyIElPTU1VIGRldmlj ZS4gSXQncyB0aGUgb3BzIG9mIHRoZSBob3N0DQo+IGNvbnRyb2xsZXINCj4gdGhhdCBzaG91bGQg YmUgaW52b2tlZCwgbm90IHRoZSBJT01NVSdzLg0KT2ssIHRoYXQgbWFrZXMgc2Vuc2UuIEknbGwg aGF2ZSBhIHRoaW5rIGFuZCBwb2tlIGl0IGEgYml0IG1vcmUuLi4NCg0KVGhhbmtzIGZvciB5b3Vy IGNvbW1lbnRzDQpQaGlsDQo= From mboxrd@z Thu Jan 1 00:00:00 1970 From: phil.edworthy@renesas.com (Phil Edworthy) Date: Wed, 4 Nov 2015 15:19:13 +0000 Subject: PCIe host controller behind IOMMU on ARM In-Reply-To: <20151104150147.GT963@e106497-lin.cambridge.arm.com> References: <20151104142412.GS963@e106497-lin.cambridge.arm.com> <20151104150147.GT963@e106497-lin.cambridge.arm.com> Message-ID: To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Hi Liviu, On 04 November 2015 15:02, Liviu wrote: > On Wed, Nov 04, 2015 at 02:48:38PM +0000, Phil Edworthy wrote: > > Hi Liviu, > > > > On 04 November 2015 14:24, Liviu wrote: > > > On Wed, Nov 04, 2015 at 01:57:48PM +0000, Phil Edworthy wrote: > > > > Hi, > > > > > > > > I am trying to hook up a PCIe host controller that sits behind an IOMMU, > > > > but having some problems. > > > > > > > > I'm using the pcie-rcar PCIe host controller and it works fine without > > > > the IOMMU, and I can attach the IOMMU to the controller such that any > calls > > > > to dma_alloc_coherent made by the controller driver uses the iommu_ops > > > > version of dma_ops. > > > > > > > > However, I can't see how to make the endpoints to utilise the dma_ops that > > > > the controller uses. Shouldn't the endpoints inherit the dma_ops from the > > > > controller? > > > > > > No, not directly. > > > > > > > Any pointers for this? > > > > > > You need to understand the process through which a driver for endpoint get > > > an address to be passed down to the device. Have a look at > > > Documentation/DMA-API-HOWTO.txt, there is a nice explanation there. > > > (Hint: EP driver needs to call dma_map_single). > > > > > > Also, you need to make sure that the bus address that ends up being set into > > > the endpoint gets translated correctly by the host controller into an address > > > that the IOMMU can then translate into physical address. > > Sure, though since this is bog standard Intel PCIe ethernet card which works > > fine when the IOMMU is effectively unused, I don?t think there is a problem > > with that. > > > > The driver for the PCIe controller sets up the IOMMU mapping ok when I > > do a test call to dma_alloc_coherent() in the controller's driver. i.e. when I > > do this, it ends up in arm_iommu_alloc_attrs(), which calls > > __iommu_alloc_buffer() and __alloc_iova(). > > > > When an endpoint driver allocates and maps a dma coherent buffer it > > also needs to end up in arm_iommu_alloc_attrs(), but it doesn't. > > Why do you think that? Remember that the only thing attached to the IOMMU is > the > host controller. The endpoint is on the PCIe bus, which gets a different > translation > that the IOMMU knows nothing about. If it helps you to visualise it better, think > of the host controller as another IOMMU device. It's the ops of the host > controller > that should be invoked, not the IOMMU's. Ok, that makes sense. I'll have a think and poke it a bit more... Thanks for your comments Phil