From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1752734AbbKKU2j (ORCPT ); Wed, 11 Nov 2015 15:28:39 -0500 Received: from mout.kundenserver.de ([212.227.17.10]:64091 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752046AbbKKU2h (ORCPT ); Wed, 11 Nov 2015 15:28:37 -0500 From: Arnd Bergmann To: "Liviu.Dudau@arm.com" Cc: Phil Edworthy , Will Deacon , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , Bjorn Helgaas , Lorenzo Pieralisi , Magnus Subject: Re: PCIe host controller behind IOMMU on ARM Date: Wed, 11 Nov 2015 21:22:35 +0100 Message-ID: <4372968.dQOnUSIksK@wuerfel> User-Agent: KMail/4.11.5 (Linux/3.16.0-10-generic; KDE/4.11.5; x86_64; ; ) In-Reply-To: <20151111182456.GV963@e106497-lin.cambridge.arm.com> References: <20151111182456.GV963@e106497-lin.cambridge.arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Provags-ID: V03:K0:zGucZlWKGRS+oPjDofTM1vvP4JRIcS/1HZr6hv7YugsxvoUpN30 +M39gKHFEOT9LyVMzvuB3kEScSCU7IO5GcpN/VnfmV9uThaw4BEms5UiJ+KP4khpFoeV2aI cb05IOTslOeNnsKrZTtrx1FaVDQqmixRrnyrlhGbq27afpPGDcGSFPYn2sQ/O/8YqmyfYWG U/p+9k74uJkigRMJkj0ow== X-UI-Out-Filterresults: notjunk:1;V01:K0:2pAThxMa7xU=:AAfZMPBcRTM//kXGkPW1LS MrVrjvi37nrqyXSBU7jR7Tqmaxq6nQm1G4+s71ZcpWjaTzvPw6GEkfVmi7UZ3XJ7MUBWfqngr ikqW6ggHYsFYQAfjrRHNO1CmCDlfoMGH2Y7RKmL/mMnK8vJ1wnn0v59D8+o/mYUNzJdSTpOAh EO+zX5BWKU9UrtPESPQXuJC+UK5UEjFT3rUwYk/qMUY1Q6C2tBZG7D0W/jpYuFAEQFOSHvH3q tACY//oagNdhonIJaKd7xW1hTrhZ583QJrL7PmCoqBUyU6Ci+XZa/WT1QjZaxmzMJw0mTYQ0i HsRZq0Mcu8+OzjdtLlv0iugZXULdIktT/xUSwLNd+Dtk4xz1Ov4GL2Nl0A4O5fDC36PMODLRX Aya/hC1HPRGZpPtk+mFd8M9g3oDykoqk15KhGngfZew1377gAGdw0kGpydbMozh32tCFSXrC0 vbDt97KpRlVcSA2h5lcj0CQZ0Btgs4iZTajnPnCNjS/sWL00DdAHdPrruFl6/A1wLYeeOYQw4 VK3j64u0+lvgLP5YRgzUeaBH7R36eKIODNvXkg+aUyuFrY5lWrxW+nPtcl44nGOOT+yJyr4EB hj37qAPqEOgQOSJ1GiFpiHnAYslpB+rEqfnGs1LMVHS04j+9VptWMmd3JFzxZfGdrvyI8NwhN CEvTIOMa8MJr+7oXFnnM0gqCkgRxmeccKxgKkfH6zw/00nW1uqQHXnjrquM0Xap4EtA/zs/Cj tvRY8LUpB3jiYQtK Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Wednesday 11 November 2015 18:24:56 Liviu.Dudau@arm.com wrote: > > > Somewhat related to this, since our PCIe controller HW is limited to > > 32-bit AXI address range, before trying to hook up the IOMMU I have > > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > > reason being that Linux uses a 1 to 1 mapping between PCI addresses > > and cpu (phys) addresses when there isn't an IOMMU involved, so I > > think that we need to limit the PCI address space used. > > I think you're mixing things a bit or not explaining them very well. Having the > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > carry 64-bit addresses. It depends on how they get translated by the host bridge > or its associated ATS block. I can't see why you can't have a setup where > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > You just have to be careful on how you setup your mem64 ranges so that they don't > overlap with the 32-bit ranges when translated. > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless the > card is not capable of supporting more than 32-bit addresses. I think we are missing one crucial bit of infrastructure on ARM64 at the moment: the dma_set_mask() function should fail if a driver asks for a mask that is larger than the dma-ranges property of the parent device (or any device higher up in the hierarchy) allows. Drivers that want a larger mask should try that first, and then fall back to a 32-bit mask, which is guaranteed to work. Arnd From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mout.kundenserver.de ([212.227.17.10]:64091 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1752046AbbKKU2h (ORCPT ); Wed, 11 Nov 2015 15:28:37 -0500 From: Arnd Bergmann To: "Liviu.Dudau@arm.com" Cc: Phil Edworthy , Will Deacon , "linux-pci@vger.kernel.org" , "linux-kernel@vger.kernel.org" , "linux-arm-kernel@lists.infradead.org" , Bjorn Helgaas , Lorenzo Pieralisi , Magnus Subject: Re: PCIe host controller behind IOMMU on ARM Date: Wed, 11 Nov 2015 21:22:35 +0100 Message-ID: <4372968.dQOnUSIksK@wuerfel> In-Reply-To: <20151111182456.GV963@e106497-lin.cambridge.arm.com> References: <20151111182456.GV963@e106497-lin.cambridge.arm.com> MIME-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Sender: linux-pci-owner@vger.kernel.org List-ID: On Wednesday 11 November 2015 18:24:56 Liviu.Dudau@arm.com wrote: > > > Somewhat related to this, since our PCIe controller HW is limited to > > 32-bit AXI address range, before trying to hook up the IOMMU I have > > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > > reason being that Linux uses a 1 to 1 mapping between PCI addresses > > and cpu (phys) addresses when there isn't an IOMMU involved, so I > > think that we need to limit the PCI address space used. > > I think you're mixing things a bit or not explaining them very well. Having the > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > carry 64-bit addresses. It depends on how they get translated by the host bridge > or its associated ATS block. I can't see why you can't have a setup where > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > You just have to be careful on how you setup your mem64 ranges so that they don't > overlap with the 32-bit ranges when translated. > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless the > card is not capable of supporting more than 32-bit addresses. I think we are missing one crucial bit of infrastructure on ARM64 at the moment: the dma_set_mask() function should fail if a driver asks for a mask that is larger than the dma-ranges property of the parent device (or any device higher up in the hierarchy) allows. Drivers that want a larger mask should try that first, and then fall back to a 32-bit mask, which is guaranteed to work. Arnd From mboxrd@z Thu Jan 1 00:00:00 1970 From: arnd@arndb.de (Arnd Bergmann) Date: Wed, 11 Nov 2015 21:22:35 +0100 Subject: PCIe host controller behind IOMMU on ARM In-Reply-To: <20151111182456.GV963@e106497-lin.cambridge.arm.com> References: <20151111182456.GV963@e106497-lin.cambridge.arm.com> Message-ID: <4372968.dQOnUSIksK@wuerfel> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On Wednesday 11 November 2015 18:24:56 Liviu.Dudau at arm.com wrote: > > > Somewhat related to this, since our PCIe controller HW is limited to > > 32-bit AXI address range, before trying to hook up the IOMMU I have > > tried to limit the dma_mask for PCI cards to DMA_BIT_MASK(32). The > > reason being that Linux uses a 1 to 1 mapping between PCI addresses > > and cpu (phys) addresses when there isn't an IOMMU involved, so I > > think that we need to limit the PCI address space used. > > I think you're mixing things a bit or not explaining them very well. Having the > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > carry 64-bit addresses. It depends on how they get translated by the host bridge > or its associated ATS block. I can't see why you can't have a setup where > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > You just have to be careful on how you setup your mem64 ranges so that they don't > overlap with the 32-bit ranges when translated. > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless the > card is not capable of supporting more than 32-bit addresses. I think we are missing one crucial bit of infrastructure on ARM64 at the moment: the dma_set_mask() function should fail if a driver asks for a mask that is larger than the dma-ranges property of the parent device (or any device higher up in the hierarchy) allows. Drivers that want a larger mask should try that first, and then fall back to a 32-bit mask, which is guaranteed to work. Arnd