From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1754838AbbKLPdv (ORCPT ); Thu, 12 Nov 2015 10:33:51 -0500 Received: from relmlor4.renesas.com ([210.160.252.174]:64668 "EHLO relmlie3.idc.renesas.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754752AbbKLPds convert rfc822-to-8bit (ORCPT ); Thu, 12 Nov 2015 10:33:48 -0500 X-IronPort-AV: E=Sophos;i="5.20,282,1444662000"; d="scan'208";a="198403453" From: Phil Edworthy To: Arnd Bergmann , "linux-arm-kernel@lists.infradead.org" CC: "Liviu.Dudau@arm.com" , Lorenzo Pieralisi , Magnus , "linux-pci@vger.kernel.org" , Will Deacon , "linux-kernel@vger.kernel.org" , Bjorn Helgaas Subject: RE: PCIe host controller behind IOMMU on ARM Thread-Topic: PCIe host controller behind IOMMU on ARM Thread-Index: AdEXBXpsqoonZ+nqRTu1yvYRg99bGAABv/qAAAApWPAAASaHgAAAPSUwAPRcWiAAcokIAAAe/r+AAAFLFIAACM5fkA== Date: Thu, 12 Nov 2015 15:33:41 +0000 Message-ID: References: <20151111182456.GV963@e106497-lin.cambridge.arm.com> <6255198.JqytBn7T9E@wuerfel> In-Reply-To: <6255198.JqytBn7T9E@wuerfel> Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=phil.edworthy@renesas.com; x-originating-ip: [193.141.220.21] x-microsoft-exchange-diagnostics: 1;PS1PR06MB1180;5:hrabtG8iQ9dKCLxVNpqPRLuMrKKlehdRZKvmHytzGyjTYAx+6X/QLeUOUxL4vfxBTT0nAX/MTa56gs6LPY4WvazzwVp7NO0UEpDIpE1Uwyib1wXmqIcCYrrw6dqXlC1jGVuCE6TgwivbUY214e0+lw==;24:1gZmV/4N784aD8m5aIi1MzOENxP3oM9K/v2eAGRN5uSH8Q7t3AzQIJve/S+1/Il7BrBJWVWku9NqZaIP3rppFVmTYxiJIExy9czK/zps59Q=;20:TZ0qeHQSJLb/xVYWfisiD37P7XsRKTjaDsVf3C1DAkMX8raIQKRA7TMYF7ovB+63zwtzk1vwoyXRPJv/NZv9QydN+2hYR1w4D78tRXNWNpKhj/9GAE9jhhS8rpLsQjH7gtJaNDku8DHw0YwVfr0IPB58EkL1qLzGSFQWAwCFY1E= x-microsoft-antispam: UriScan:;BCL:0;PCL:0;RULEID:;SRVR:PS1PR06MB1180; x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:; x-exchange-antispam-report-cfa-test: BCL:0;PCL:0;RULEID:(601004)(2401047)(520078)(8121501046)(5005006)(3002001)(10201501046);SRVR:PS1PR06MB1180;BCL:0;PCL:0;RULEID:;SRVR:PS1PR06MB1180; x-forefront-prvs: 07584EDBCD x-forefront-antispam-report: SFV:NSPM;SFS:(10019020)(6009001)(24454002)(199003)(189002)(5002640100001)(33656002)(87936001)(2501003)(11100500001)(5001960100002)(2950100001)(5004730100002)(77096005)(5001770100001)(81156007)(106356001)(2900100001)(102836002)(105586002)(97736004)(76576001)(189998001)(92566002)(5008740100001)(5007970100001)(122556002)(74316001)(10400500002)(93886004)(76176999)(101416001)(66066001)(86362001)(54356999)(5001920100001)(40100003)(50986999)(5003600100002);DIR:OUT;SFP:1102;SCL:1;SRVR:PS1PR06MB1180;H:PS1PR06MB1180.apcprd06.prod.outlook.com;FPR:;SPF:None;PTR:InfoNoRecords;A:1;MX:1;LANG:en; spamdiagnosticoutput: 1:23 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 8BIT MIME-Version: 1.0 X-OriginatorOrg: renesas.com X-MS-Exchange-CrossTenant-originalarrivaltime: 12 Nov 2015 15:33:41.5728 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 53d82571-da19-47e4-9cb4-625a166a4a2a X-MS-Exchange-Transport-CrossTenantHeadersStamped: PS1PR06MB1180 Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org Hi Arnd, On 12 November 2015 09:49, Arnd Bergmann wrote: > On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > > On 11 November 2015 18:25, LIviu wrote: > > > On Mon, Nov 09, 2015 at 12:32:13PM +0000, Phil Edworthy wrote: > > > > I think you're mixing things a bit or not explaining them very well. Having the > > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > > > carry 64-bit addresses. It depends on how they get translated by the host > bridge > > > or its associated ATS block. I can't see why you can't have a setup where > > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > > You just have to be careful on how you setup your mem64 ranges so that > they > > > don't > > > overlap with the 32-bit ranges when translated. > > From a HW point of view I agree that we can setup the PCI host bridge such that > > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice doesn't > > this mean that the dma ops used by card drivers has to be provided by our PCI > > host bridge driver so we can apply the translation to those PCI addresses? > > This comes back to my point below about how to do this. Adding a bus notifier > > to do this may be too late, and arm64 doesn't implement set_dma_ops(). > > > > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless > the > > > card is not capable of supporting more than 32-bit addresses. > > If there was infrastructure that checked all parents dma-ranges when the > > dma_set_mask() function is called as Arnd pointed out, this would nicely solve > > the problem. > > of_dma_configure calls of_dma_get_range to do all this for the PCIe host, > and then calls arch_setup_dma_ops() so the architecture specific code can > enforce the limits in dma_set_mask and pick an appropriate set of dma > operations. The missing part is in the implementation of arch_setup_dma_ops, > which currently happily ignores the base and limit. I don't think it's as simple as that, though I could be wrong! First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB. This default is set for the 'platform soc' device. For my own testing I increased this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes boot failure that I haven't looked into. Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before calling of_pci_dma_configure(). I assume it does this on the basis that this is a good default for PCI drivers that don't call dma_set_mask(). So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll hit this mask. Finally, dma_set_mask_and_coherent() is called from the PCI card driver but it doesn't check the parents dma masks either. Thanks Phil From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from relmlor4.renesas.com ([210.160.252.174]:64668 "EHLO relmlie3.idc.renesas.com" rhost-flags-OK-OK-OK-FAIL) by vger.kernel.org with ESMTP id S1754752AbbKLPds convert rfc822-to-8bit (ORCPT ); Thu, 12 Nov 2015 10:33:48 -0500 From: Phil Edworthy To: Arnd Bergmann , "linux-arm-kernel@lists.infradead.org" CC: "Liviu.Dudau@arm.com" , Lorenzo Pieralisi , Magnus , "linux-pci@vger.kernel.org" , Will Deacon , "linux-kernel@vger.kernel.org" , Bjorn Helgaas Subject: RE: PCIe host controller behind IOMMU on ARM Date: Thu, 12 Nov 2015 15:33:41 +0000 Message-ID: References: <20151111182456.GV963@e106497-lin.cambridge.arm.com> <6255198.JqytBn7T9E@wuerfel> In-Reply-To: <6255198.JqytBn7T9E@wuerfel> Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Sender: linux-pci-owner@vger.kernel.org List-ID: Hi Arnd, On 12 November 2015 09:49, Arnd Bergmann wrote: > On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > > On 11 November 2015 18:25, LIviu wrote: > > > On Mon, Nov 09, 2015 at 12:32:13PM +0000, Phil Edworthy wrote: > > > > I think you're mixing things a bit or not explaining them very well. Having the > > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > > > carry 64-bit addresses. It depends on how they get translated by the host > bridge > > > or its associated ATS block. I can't see why you can't have a setup where > > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > > You just have to be careful on how you setup your mem64 ranges so that > they > > > don't > > > overlap with the 32-bit ranges when translated. > > From a HW point of view I agree that we can setup the PCI host bridge such that > > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice doesn't > > this mean that the dma ops used by card drivers has to be provided by our PCI > > host bridge driver so we can apply the translation to those PCI addresses? > > This comes back to my point below about how to do this. Adding a bus notifier > > to do this may be too late, and arm64 doesn't implement set_dma_ops(). > > > > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless > the > > > card is not capable of supporting more than 32-bit addresses. > > If there was infrastructure that checked all parents dma-ranges when the > > dma_set_mask() function is called as Arnd pointed out, this would nicely solve > > the problem. > > of_dma_configure calls of_dma_get_range to do all this for the PCIe host, > and then calls arch_setup_dma_ops() so the architecture specific code can > enforce the limits in dma_set_mask and pick an appropriate set of dma > operations. The missing part is in the implementation of arch_setup_dma_ops, > which currently happily ignores the base and limit. I don't think it's as simple as that, though I could be wrong! First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB. This default is set for the 'platform soc' device. For my own testing I increased this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes boot failure that I haven't looked into. Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before calling of_pci_dma_configure(). I assume it does this on the basis that this is a good default for PCI drivers that don't call dma_set_mask(). So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll hit this mask. Finally, dma_set_mask_and_coherent() is called from the PCI card driver but it doesn't check the parents dma masks either. Thanks Phil From mboxrd@z Thu Jan 1 00:00:00 1970 From: phil.edworthy@renesas.com (Phil Edworthy) Date: Thu, 12 Nov 2015 15:33:41 +0000 Subject: PCIe host controller behind IOMMU on ARM In-Reply-To: <6255198.JqytBn7T9E@wuerfel> References: <20151111182456.GV963@e106497-lin.cambridge.arm.com> <6255198.JqytBn7T9E@wuerfel> Message-ID: To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org Hi Arnd, On 12 November 2015 09:49, Arnd Bergmann wrote: > On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > > On 11 November 2015 18:25, LIviu wrote: > > > On Mon, Nov 09, 2015 at 12:32:13PM +0000, Phil Edworthy wrote: > > > > I think you're mixing things a bit or not explaining them very well. Having the > > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > > > carry 64-bit addresses. It depends on how they get translated by the host > bridge > > > or its associated ATS block. I can't see why you can't have a setup where > > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > > You just have to be careful on how you setup your mem64 ranges so that > they > > > don't > > > overlap with the 32-bit ranges when translated. > > From a HW point of view I agree that we can setup the PCI host bridge such that > > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice doesn't > > this mean that the dma ops used by card drivers has to be provided by our PCI > > host bridge driver so we can apply the translation to those PCI addresses? > > This comes back to my point below about how to do this. Adding a bus notifier > > to do this may be too late, and arm64 doesn't implement set_dma_ops(). > > > > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless > the > > > card is not capable of supporting more than 32-bit addresses. > > If there was infrastructure that checked all parents dma-ranges when the > > dma_set_mask() function is called as Arnd pointed out, this would nicely solve > > the problem. > > of_dma_configure calls of_dma_get_range to do all this for the PCIe host, > and then calls arch_setup_dma_ops() so the architecture specific code can > enforce the limits in dma_set_mask and pick an appropriate set of dma > operations. The missing part is in the implementation of arch_setup_dma_ops, > which currently happily ignores the base and limit. I don't think it's as simple as that, though I could be wrong! First off, of_dma_configure() sets a default coherent_dma_mask to 4GiB. This default is set for the 'platform soc' device. For my own testing I increased this to DMA_BIT_MASK(63). Note that setting it to DMA_BIT_MASK(64) causes boot failure that I haven't looked into. Then pci_device_add() sets the devices coherent_dma_mask to 4GiB before calling of_pci_dma_configure(). I assume it does this on the basis that this is a good default for PCI drivers that don't call dma_set_mask(). So if arch_setup_dma_ops() walks up the parents to limit the mask, you'll hit this mask. Finally, dma_set_mask_and_coherent() is called from the PCI card driver but it doesn't check the parents dma masks either. Thanks Phil