From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754624AbbKMOL4 (ORCPT ); Fri, 13 Nov 2015 09:11:56 -0500 Received: from relmlor3.renesas.com ([210.160.252.173]:65191 "EHLO relmlie2.idc.renesas.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752126AbbKMOLy convert rfc822-to-8bit (ORCPT ); Fri, 13 Nov 2015 09:11:54 -0500 X-IronPort-AV: E=Sophos;i="5.20,287,1444662000"; d="scan'208";a="199717328" From: Phil Edworthy To: Arnd Bergmann , "linux-arm-kernel@lists.infradead.org" CC: Lorenzo Pieralisi , Magnus , Will Deacon , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , Bjorn Helgaas , "Liviu.Dudau@arm.com" Subject: RE: PCIe host controller behind IOMMU on ARM Thread-Topic: PCIe host controller behind IOMMU on ARM Thread-Index: AdEXBXpsqoonZ+nqRTu1yvYRg99bGAABv/qAAAApWPAAASaHgAAAPSUwAPRcWiAAcokIAAAe/r+AAAFLFIAACM5fkAAEt8OAACffjcAABaIIAAAAQVpw Date: Fri, 13 Nov 2015 14:11:47 +0000 Message-ID: References: <4169020.aC5VXkILQm@wuerfel> <4744053.Iz6Y8QJQYF@wuerfel> In-Reply-To: <4744053.Iz6Y8QJQYF@wuerfel> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=phil.edworthy@renesas.com; x-originating-ip: [193.141.220.21] x-microsoft-exchange-diagnostics: 1;PS1PR06MB1178;5:MzS1c2zSxK0FgZKwf65PWX5imvaSZXozrnnZsmkexuBqXMXMqoJJzB8alzKSD3FNPb1ATznrB/l4S0eqgYPdUtHpUpEAHTlFt7QTDsXwaQOGsy+mi043qAC8R/6a+Ol3TzPXtLKoAeArpx9JndmDNA==;24:xlJHjiOdCri/hFMTkqe/Oe4S+vSv/mLsTk7jw7olVXQ/XFSAfmWJ0ebQnpjR816AqZm6IYMJo6rSosUCzRpvGsOv3wcEz6VgDb1GNWXX3XM=;20:RjTzjcC2Ii/sfEFrK7mjTOWG7QJskl48aD+SsVSVjxslfxF7cpaU0j1cNI9sy0iXi5zHZgo5JaRQcmgV/KbfxE+oXn/DwazSPfqrtU0XY6nOgsn6Kva4GpeNMMBArZiHKVI3rEie6gSplZUf2gzlbKHxbnS4EnMkbvONNSiaWOo= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:PS1PR06MB1178; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(601004)(2401047)(5005006)(520078)(8121501046)(10201501046)(3002001);SRVR:PS1PR06MB1178;BCL:0;PCL:0;RULEID:;SRVR:PS1PR06MB1178; x-forefront-prvs: 0759F7A50A x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(199003)(24454002)(189002)(40764003)(50986999)(5003600100002)(5008740100001)(86362001)(102836002)(101416001)(11100500001)(5002640100001)(92566002)(93886004)(76176999)(77096005)(10400500002)(54356999)(74316001)(40100003)(122556002)(5001770100001)(5001920100001)(189998001)(33656002)(87936001)(81156007)(5007970100001)(76576001)(97736004)(106356001)(2950100001)(2501003)(105586002)(5004730100002)(5001960100002)(66066001)(2900100001);DIR:OUT;SFP:1102;SCL:1;SRVR:PS1PR06MB1178;H:PS1PR06MB1180.apcprd06.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 X-OriginatorOrg: renesas.com X-MS-Exchange-CrossTenant-originalarrivaltime: 13 Nov 2015 14:11:47.7697 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 53d82571-da19-47e4-9cb4-625a166a4a2a X-MS-Exchange-Transport-CrossTenantHeadersStamped: PS1PR06MB1178 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On 13 November 2015 14:00, Arnd Bergmann wrote: > On Friday 13 November 2015 13:03:11 Phil Edworthy wrote: > > > > > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB > before > > > > calling of_pci_dma_configure(). I assume it does this on the basis that this is > a > > > > good default for PCI drivers that don't call dma_set_mask(). > > > > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll > hit > > > > this mask. > > > > > > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() > > > does this before calling arch_setup_dma_ops(). The PCI devices start out > > > with the 32-bit mask, but the limit should be whatever PCI host uses. > > Ok, so of_dma_configure() could walk up the tree and restrict the dma > > mask to whatever parents limit it to. Then it could be overridden by > > a dma-ranges entry in the DT node, right? > > No, the dma-ranges properties tell you what the allowed masks are, > this is what of_dma_configure() looks at. Ok, I understand now. > > If so, one problem I can see is PCI controllers already use the > > dma-ranges binding but with 3 address cells since it also specifies > > the PCI address range. > > > > I noticed that of_dma_get_range() skips straight to the parent node. > > Shouldn't it attempt to get the dma-ranges for the device's node > > first? > > No, the dma-ranges explain the capabilities of the bus, this is > what you have to look at. The device itself may have additional > restrictions, but those are what the driver knows based on the > compatibility value when it passes the device specific mask into > dma_set_mask() Ok, this is making sense now. > > I mean most hardware is limited by the peripheral's > > capabilities, not the bus. If fact, of_dma_get_range() gets the number > > of address and size cells from the device node, but gets the dma-ranges > > from the parent. That seems a little odd to me. > > of_dma_get_range() calls of_n_addr_cells()/of_n_size_cells(), which get > the #address-cells/#size-cells property from the parent device (except > for the root, which is special). Right, I should have checked what of_n_addr/size_cell() actually did. > > The only other problem I can see is that currently all PCI drivers can > > try to set their dma mask to 64 bits. At the moment that succeeds > > because there are no checks. > > Right, this is the main bug we need to fix. Yep. > > Until devices using them have their DTs > > updated with dma-ranges, we would be limiting them to a 32 bit mask. I > > guess that's not much of an issue in practice. > > Correct. I've tried to tell everyone about this when they added device > nodes for DMA capable devices. In most cases, they want 32-bit masks > anyway. Thanks for your help & patience, much appreciated! Phil From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from relmlor3.renesas.com ([210.160.252.173]:65191 "EHLO relmlie2.idc.renesas.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1752126AbbKMOLy convert rfc822-to-8bit (ORCPT ); Fri, 13 Nov 2015 09:11:54 -0500 From: Phil Edworthy To: Arnd Bergmann , "linux-arm-kernel@lists.infradead.org" CC: Lorenzo Pieralisi , Magnus , Will Deacon , "linux-kernel@vger.kernel.org" , "linux-pci@vger.kernel.org" , Bjorn Helgaas , "Liviu.Dudau@arm.com" Subject: RE: PCIe host controller behind IOMMU on ARM Date: Fri, 13 Nov 2015 14:11:47 +0000 Message-ID: References: <4169020.aC5VXkILQm@wuerfel> <4744053.Iz6Y8QJQYF@wuerfel> In-Reply-To: <4744053.Iz6Y8QJQYF@wuerfel> Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org List-ID: On 13 November 2015 14:00, Arnd Bergmann wrote: > On Friday 13 November 2015 13:03:11 Phil Edworthy wrote: > > > > > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB > before > > > > calling of_pci_dma_configure(). I assume it does this on the basis that this is > a > > > > good default for PCI drivers that don't call dma_set_mask(). > > > > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll > hit > > > > this mask. > > > > > > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() > > > does this before calling arch_setup_dma_ops(). The PCI devices start out > > > with the 32-bit mask, but the limit should be whatever PCI host uses. > > Ok, so of_dma_configure() could walk up the tree and restrict the dma > > mask to whatever parents limit it to. Then it could be overridden by > > a dma-ranges entry in the DT node, right? > > No, the dma-ranges properties tell you what the allowed masks are, > this is what of_dma_configure() looks at. Ok, I understand now. > > If so, one problem I can see is PCI controllers already use the > > dma-ranges binding but with 3 address cells since it also specifies > > the PCI address range. > > > > I noticed that of_dma_get_range() skips straight to the parent node. > > Shouldn't it attempt to get the dma-ranges for the device's node > > first? > > No, the dma-ranges explain the capabilities of the bus, this is > what you have to look at. The device itself may have additional > restrictions, but those are what the driver knows based on the > compatibility value when it passes the device specific mask into > dma_set_mask() Ok, this is making sense now. > > I mean most hardware is limited by the peripheral's > > capabilities, not the bus. If fact, of_dma_get_range() gets the number > > of address and size cells from the device node, but gets the dma-ranges > > from the parent. That seems a little odd to me. > > of_dma_get_range() calls of_n_addr_cells()/of_n_size_cells(), which get > the #address-cells/#size-cells property from the parent device (except > for the root, which is special). Right, I should have checked what of_n_addr/size_cell() actually did. > > The only other problem I can see is that currently all PCI drivers can > > try to set their dma mask to 64 bits. At the moment that succeeds > > because there are no checks. > > Right, this is the main bug we need to fix. Yep. > > Until devices using them have their DTs > > updated with dma-ranges, we would be limiting them to a 32 bit mask. I > > guess that's not much of an issue in practice. > > Correct. I've tried to tell everyone about this when they added device > nodes for DMA capable devices. In most cases, they want 32-bit masks > anyway. Thanks for your help & patience, much appreciated! Phil From mboxrd@z Thu Jan 1 00:00:00 1970 From: phil.edworthy@renesas.com (Phil Edworthy) Date: Fri, 13 Nov 2015 14:11:47 +0000 Subject: PCIe host controller behind IOMMU on ARM In-Reply-To: <4744053.Iz6Y8QJQYF@wuerfel> References: <4169020.aC5VXkILQm@wuerfel> <4744053.Iz6Y8QJQYF@wuerfel> Message-ID: To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On 13 November 2015 14:00, Arnd Bergmann wrote: > On Friday 13 November 2015 13:03:11 Phil Edworthy wrote: > > > > > > Then pci_device_add() sets the devices coherent_dma_mask to 4GiB > before > > > > calling of_pci_dma_configure(). I assume it does this on the basis that this is > a > > > > good default for PCI drivers that don't call dma_set_mask(). > > > > So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll > hit > > > > this mask. > > > > > > arch_setup_dma_ops() does not walk up the hierarchy, of_dma_configure() > > > does this before calling arch_setup_dma_ops(). The PCI devices start out > > > with the 32-bit mask, but the limit should be whatever PCI host uses. > > Ok, so of_dma_configure() could walk up the tree and restrict the dma > > mask to whatever parents limit it to. Then it could be overridden by > > a dma-ranges entry in the DT node, right? > > No, the dma-ranges properties tell you what the allowed masks are, > this is what of_dma_configure() looks at. Ok, I understand now. > > If so, one problem I can see is PCI controllers already use the > > dma-ranges binding but with 3 address cells since it also specifies > > the PCI address range. > > > > I noticed that of_dma_get_range() skips straight to the parent node. > > Shouldn't it attempt to get the dma-ranges for the device's node > > first? > > No, the dma-ranges explain the capabilities of the bus, this is > what you have to look at. The device itself may have additional > restrictions, but those are what the driver knows based on the > compatibility value when it passes the device specific mask into > dma_set_mask() Ok, this is making sense now. > > I mean most hardware is limited by the peripheral's > > capabilities, not the bus. If fact, of_dma_get_range() gets the number > > of address and size cells from the device node, but gets the dma-ranges > > from the parent. That seems a little odd to me. > > of_dma_get_range() calls of_n_addr_cells()/of_n_size_cells(), which get > the #address-cells/#size-cells property from the parent device (except > for the root, which is special). Right, I should have checked what of_n_addr/size_cell() actually did. > > The only other problem I can see is that currently all PCI drivers can > > try to set their dma mask to 64 bits. At the moment that succeeds > > because there are no checks. > > Right, this is the main bug we need to fix. Yep. > > Until devices using them have their DTs > > updated with dma-ranges, we would be limiting them to a 32 bit mask. I > > guess that's not much of an issue in practice. > > Correct. I've tried to tell everyone about this when they added device > nodes for DMA capable devices. In most cases, they want 32-bit masks > anyway. Thanks for your help & patience, much appreciated! Phil