From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S1753999AbbKLJuG (ORCPT ); Thu, 12 Nov 2015 04:50:06 -0500 Received: from mout.kundenserver.de ([212.227.17.13]:55897 "EHLO mout.kundenserver.de" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S1753500AbbKLJuC (ORCPT ); Thu, 12 Nov 2015 04:50:02 -0500 From: Arnd Bergmann To: linux-arm-kernel@lists.infradead.org Cc: Phil Edworthy , "Liviu.Dudau@arm.com" , Lorenzo Pieralisi , Magnus , "linux-pci@vger.kernel.org" , Will Deacon , "linux-kernel@vger.kernel.org" , Bjorn Helgaas Subject: Re: PCIe host controller behind IOMMU on ARM Date: Thu, 12 Nov 2015 10:49:27 +0100 Message-ID: <6255198.JqytBn7T9E@wuerfel> User-Agent: KMail/4.11.5 (Linux/3.16.0-10-generic; KDE/4.11.5; x86_64; ; ) In-Reply-To: References: <20151111182456.GV963@e106497-lin.cambridge.arm.com> MIME-Version: 1.0 Content-Transfer-Encoding: 7Bit Content-Type: text/plain; charset="us-ascii" X-Provags-ID: V03:K0:ID9E95eclqXzgZ7qQqTPg42JhyeQnuVXZvWHHYdKoUOtOskXAPZ bnjtR9rS1496RF6G6qEC6XWEBOmP81m25hK87uqkeF6wwpPVgE64su7XrP17jrUaM2eQz05 gtSPY/EgHbKo4tz96A1DH1L+mJFhWLAMrIoaZvggvN5HG9H0Tc4gy5IMvMBF+cPok/6PHK2 2DhNtDNjlKGj2JyGZSDZw== X-UI-Out-Filterresults: notjunk:1;V01:K0:MCbYQruKdzM=:rT9k8VYdStKOJFZtqjkCJQ EPQGSUtdisT226THj+lt+Zz242jb0449L5APrCP/R7ac6B35Ip1QBTJlgmpFlPo6/JdvMBVmn gh1CA0uy43aP7HH3GHk44evoanlKias79LNj0NZwBLVaB0l150WnsJz5IB0aHex4sc+JflRkS dBKP77goQ47UKL5dfUw5KxENxRjcY0M4Sk0FkrOu/obgxdHvQ6a1MNnuXdv5c9EazY7lATGpi oIBXZDzNm/oFNnYGfEwyBuCJKY8X3H799ZthUHl3t8Al/N5O+1lxu6Z6znzFdQH5wDR1ewbgu ClU8pwNAzghC/rQawNlCk6JDTar9NhxslKcRsA6QYc4Ek0yTt97rsduvZKBkUJ0S+LUc0J/0x W41ENFAOTKeVnVfUMR2m1+o1P7+ekwTfeYXQ7wyh4G7z+mz3QLDK5L3sVyXN9vlaN8Tvo8cMX ZHXM3Ln858wXlSLaYAsQ3OmzAMh5ur6YdmcKiXauZXPX0C+gxd0rSAftswoJuYRiMQmgwKdYB qEHboaT4CV6a91gD6Yzo5Kh9ydDSZV/JcC+X9KPacuiioy0Nj/tZtZIEtGGi/WCtArpTSzmqe Yys92idrix1i6VxlJY+i/SAPz/wHY14bjuFVhaYRj7vyusCj0i+BhN34oM7u4KgF862YAQFH7 mAH99b+ohVsTQDgVX/uYhmBV9tCdRD/CbqxpwzJINZUErs+h08Jn8WsIcG52192VJOL3zwdEz Nw0km5mqZc2w8sBf Sender: linux-kernel-owner@vger.kernel.org List-ID: X-Mailing-List: linux-kernel@vger.kernel.org On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > On 11 November 2015 18:25, LIviu wrote: > > On Mon, Nov 09, 2015 at 12:32:13PM +0000, Phil Edworthy wrote: > > I think you're mixing things a bit or not explaining them very well. Having the > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > > carry 64-bit addresses. It depends on how they get translated by the host bridge > > or its associated ATS block. I can't see why you can't have a setup where > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > You just have to be careful on how you setup your mem64 ranges so that they > > don't > > overlap with the 32-bit ranges when translated. > From a HW point of view I agree that we can setup the PCI host bridge such that > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice doesn't > this mean that the dma ops used by card drivers has to be provided by our PCI > host bridge driver so we can apply the translation to those PCI addresses? > This comes back to my point below about how to do this. Adding a bus notifier > to do this may be too late, and arm64 doesn't implement set_dma_ops(). > > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless the > > card is not capable of supporting more than 32-bit addresses. > If there was infrastructure that checked all parents dma-ranges when the > dma_set_mask() function is called as Arnd pointed out, this would nicely solve > the problem. of_dma_configure calls of_dma_get_range to do all this for the PCIe host, and then calls arch_setup_dma_ops() so the architecture specific code can enforce the limits in dma_set_mask and pick an appropriate set of dma operations. The missing part is in the implementation of arch_setup_dma_ops, which currently happily ignores the base and limit. Arnd From mboxrd@z Thu Jan 1 00:00:00 1970 From: arnd@arndb.de (Arnd Bergmann) Date: Thu, 12 Nov 2015 10:49:27 +0100 Subject: PCIe host controller behind IOMMU on ARM In-Reply-To: References: <20151111182456.GV963@e106497-lin.cambridge.arm.com> Message-ID: <6255198.JqytBn7T9E@wuerfel> To: linux-arm-kernel@lists.infradead.org List-Id: linux-arm-kernel.lists.infradead.org On Thursday 12 November 2015 09:26:33 Phil Edworthy wrote: > On 11 November 2015 18:25, LIviu wrote: > > On Mon, Nov 09, 2015 at 12:32:13PM +0000, Phil Edworthy wrote: > > I think you're mixing things a bit or not explaining them very well. Having the > > PCIe controller limited to 32-bit AXI does not mean that the PCIe bus cannot > > carry 64-bit addresses. It depends on how they get translated by the host bridge > > or its associated ATS block. I can't see why you can't have a setup where > > the CPU addresses are 32-bit but the PCIe bus addresses are all 64-bit. > > You just have to be careful on how you setup your mem64 ranges so that they > > don't > > overlap with the 32-bit ranges when translated. > From a HW point of view I agree that we can setup the PCI host bridge such that > it uses 64-bit PCI address, with 32-bit cpu addresses. Though in practice doesn't > this mean that the dma ops used by card drivers has to be provided by our PCI > host bridge driver so we can apply the translation to those PCI addresses? > This comes back to my point below about how to do this. Adding a bus notifier > to do this may be too late, and arm64 doesn't implement set_dma_ops(). > > > And no, you should not limit at the card driver the DMA_BIT_MASK() unless the > > card is not capable of supporting more than 32-bit addresses. > If there was infrastructure that checked all parents dma-ranges when the > dma_set_mask() function is called as Arnd pointed out, this would nicely solve > the problem. of_dma_configure calls of_dma_get_range to do all this for the PCIe host, and then calls arch_setup_dma_ops() so the architecture specific code can enforce the limits in dma_set_mask and pick an appropriate set of dma operations. The missing part is in the implementation of arch_setup_dma_ops, which currently happily ignores the base and limit. Arnd