From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from vger.kernel.org (vger.kernel.org [23.128.96.18]) by smtp.lore.kernel.org (Postfix) with ESMTP id 86C1CC433EF for ; Thu, 24 Feb 2022 19:43:23 +0000 (UTC) Received: (majordomo@vger.kernel.org) by vger.kernel.org via listexpand id S232519AbiBXTnv (ORCPT ); Thu, 24 Feb 2022 14:43:51 -0500 Received: from lindbergh.monkeyblade.net ([23.128.96.19]:58680 "EHLO lindbergh.monkeyblade.net" rhost-flags-OK-OK-OK-OK) by vger.kernel.org with ESMTP id S233043AbiBXTnt (ORCPT ); Thu, 24 Feb 2022 14:43:49 -0500 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by lindbergh.monkeyblade.net (Postfix) with ESMTPS id 38DD01B8BD2 for ; Thu, 24 Feb 2022 11:43:17 -0800 (PST) Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 6A95A61756 for ; Thu, 24 Feb 2022 19:43:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8D772C340E9; Thu, 24 Feb 2022 19:43:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1645731796; bh=q8BGb7R1C+px4L+SV0/F/PoXumVpPWGqchxwyNTZCok=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=vJ6fmaMgA8lbvGsL2xbdFrUfMKvKJbyiJkq5TbqNgRuHKoAEYCt8ee0wSXa5UWgKP hCNojDN23anTJ7xwXQhRPrMMfDRjcgX2vcq+4uTHHOss7DQl1kBCX+QHVrmsN2mFAU kXf4hlzXgk0Ad+YFRx+Pk9kNmanxm3OGEIvRmHILVenRr6hJHFnL7IkZDtyPAJ2cgE uEfLuZvjq8i0npC7cCI50udWYYNvzu40ZPLi8+R3CIKOWF1f7oGfFcx8BMsNT1PkhG VOqvX1RQFLSw+F1+uqAYU4Qgzap70iActCfYnsxU+UwIZOZZv/71t63QJQzbY5+V+T wA+mFFVpQVaxw== Date: Thu, 24 Feb 2022 13:43:15 -0600 From: Bjorn Helgaas To: Pali =?iso-8859-1?Q?Roh=E1r?= Cc: Marek =?iso-8859-1?Q?Beh=FAn?= , Marc Zyngier , Lorenzo Pieralisi , linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Rob Herring Subject: Re: [PATCH v2 11/23] PCI: aardvark: Fix setting MSI address Message-ID: <20220224194315.GA278999@bhelgaas> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-1 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <20220224125901.bmfqrby3lrda36p3@pali> Precedence: bulk List-ID: X-Mailing-List: linux-pci@vger.kernel.org [+cc Rob] On Thu, Feb 24, 2022 at 01:59:01PM +0100, Pali Rohár wrote: > On Wednesday 23 February 2022 12:13:12 Bjorn Helgaas wrote: > > On Fri, Feb 18, 2022 at 03:43:29PM +0100, Marek Behún wrote: > > > On Thu, 17 Feb 2022 11:14:52 -0600 > > > Bjorn Helgaas wrote: > > > > > > > > + phys_addr_t msi_addr; > > > > > u32 reg; > > > > > int i; > > > > > > > > > > @@ -561,6 +561,11 @@ static void advk_pcie_setup_hw(struct advk_pcie *pcie) > > > > > reg |= LANE_COUNT_1; > > > > > advk_writel(pcie, reg, PCIE_CORE_CTRL0_REG); > > > > > > > > > > + /* Set MSI address */ > > > > > + msi_addr = virt_to_phys(pcie); > > > > > > > > Strictly speaking, msi_addr should be a pci_bus_addr_t, not a > > > > phys_addr_t, and virt_to_phys() doesn't return a bus address. On second thought, probably a dma_addr_t, not a pci_bus_addr_t. > > > the problem here is that as far as we know currently there is no > > > function that converts a virtual address to pci_bus_addr_t like > > > virt_to_phys() does to convert to phys_addr_t. > > > > > > On Armada 3720 there are PCIe Controller Address Decoder Registers, > > > which such a translating function would need to consult to do the > > > translation. But the default settings of these registers is to map PCIe > > > addresses 1 to 1 to physical addresses, and no driver changes these > > > registers. > > > > The poorly-named pcibios_resource_to_bus() (I think the name is my > > fault) is the way to convert a CPU physical address to a PCI bus > > address. > > But here it is needed to do something different. It is needed to do > inverse mapping of function which converts PCI bus address to CPU > physical address of CPU memory. So to converting CPU physical address of > CPU memory to PCI bus address from PCI bus point of view. > > I think that this information is stored in dma_ranges member of struct > pci_host_bridge. But function pcibios_resource_to_bus() looks at the > ->windows member which converts CPU physical address of PCI memory (not > CPU memory) to PCI bus address, which is something different. So > pcibios_resource_to_bus() would not work here and it may return > incorrect values (as PCI memory may be different from CPU point of view > and PCI bus point of view). Oh, sorry, indeed you are correct and I was completely on the wrong track. pcibios_resource_to_bus() is what you need for doing MMIO -- CPU accesses to things on PCI. MSI is the reverse. From the device's point of view, an MSI is basically a DMA as you allude to above, so I would expect the DMA API to be involved somehow. I do see a couple drivers using the DMA API for this: struct pcie_port { dma_addr_t msi_data; }; dw_pcie_host_init pp->msi_data = dma_map_single_attrs(..., &pp->msi_msg, ...) dw_pcie_setup_rc dw_pcie_msi_init u64 msi_target = (u64)pp->msi_data; dw_pcie_writel_dbi(pci, PCIE_MSI_ADDR_LO, lower_32_bits(msi_target)); dw_pci_setup_msi_msg u64 msi_target = (u64)pp->msi_data; msg->address_lo = lower_32_bits(msi_target); ----------------------------------------------------------- struct tegra_msi { dma_addr_t phys; }; tegra_pcie_probe tegra_pcie_msi_setup msi->virt = dma_alloc_attrs(..., &msi->phys, ...); tegra_pcie_enable_msi afi_writel(pcie, msi->phys, ...); tegra_compose_msi_msg msg->address_low = lower_32_bits(msi->phys); Bjorn From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id E26C9C433EF for ; Thu, 24 Feb 2022 19:44:42 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:References: List-Owner; bh=GToLHBb8cpFS6BjMzb+6NqsPk2tqB/Q6CGAri47pR3M=; b=oJ8hXqdsoL66vP PPRMpVaLZT2t5mC7/tQ1/tGMOI0BQW6CoCquDMc8VynACqyX0Fq/5O8XIVSfeArp8TgZudkuNesRk XaUm6XXSYQfAtwECDQcrBjTGbGD/5nrOQj9L/tEBgpldqQTOgqOos6ZvhxEzPRpZDuS2VFl1YxUTh k7VkX2/l0F6RgKPmZX0RWvIDr2G5+E+1W69HLS52tcuUsXKmmC/rs4EjjvwFJDffMX2H2uSdNkcte bJzvml+0JyoL9sQuoNg+nOg2jRce0pC9c6iRv/iSa7YX90XLvoymBNpFQe7oWRC/dgEN9LFBZzQiH biewSWUJdyehzTaJr8Eg==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nNK1K-002B0v-5C; Thu, 24 Feb 2022 19:43:22 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nNK1G-002B0Y-Gf for linux-arm-kernel@lists.infradead.org; Thu, 24 Feb 2022 19:43:19 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 658C36174D; Thu, 24 Feb 2022 19:43:17 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id 8D772C340E9; Thu, 24 Feb 2022 19:43:16 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1645731796; bh=q8BGb7R1C+px4L+SV0/F/PoXumVpPWGqchxwyNTZCok=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=vJ6fmaMgA8lbvGsL2xbdFrUfMKvKJbyiJkq5TbqNgRuHKoAEYCt8ee0wSXa5UWgKP hCNojDN23anTJ7xwXQhRPrMMfDRjcgX2vcq+4uTHHOss7DQl1kBCX+QHVrmsN2mFAU kXf4hlzXgk0Ad+YFRx+Pk9kNmanxm3OGEIvRmHILVenRr6hJHFnL7IkZDtyPAJ2cgE uEfLuZvjq8i0npC7cCI50udWYYNvzu40ZPLi8+R3CIKOWF1f7oGfFcx8BMsNT1PkhG VOqvX1RQFLSw+F1+uqAYU4Qgzap70iActCfYnsxU+UwIZOZZv/71t63QJQzbY5+V+T wA+mFFVpQVaxw== Date: Thu, 24 Feb 2022 13:43:15 -0600 From: Bjorn Helgaas To: Pali =?iso-8859-1?Q?Roh=E1r?= Cc: Marek =?iso-8859-1?Q?Beh=FAn?= , Marc Zyngier , Lorenzo Pieralisi , linux-pci@vger.kernel.org, linux-arm-kernel@lists.infradead.org, Rob Herring Subject: Re: [PATCH v2 11/23] PCI: aardvark: Fix setting MSI address Message-ID: <20220224194315.GA278999@bhelgaas> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20220224125901.bmfqrby3lrda36p3@pali> X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220224_114318_674701_B172BB2E X-CRM114-Status: GOOD ( 31.31 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="iso-8859-1" Content-Transfer-Encoding: quoted-printable Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org [+cc Rob] On Thu, Feb 24, 2022 at 01:59:01PM +0100, Pali Roh=E1r wrote: > On Wednesday 23 February 2022 12:13:12 Bjorn Helgaas wrote: > > On Fri, Feb 18, 2022 at 03:43:29PM +0100, Marek Beh=FAn wrote: > > > On Thu, 17 Feb 2022 11:14:52 -0600 > > > Bjorn Helgaas wrote: > > > = > > > > > + phys_addr_t msi_addr; > > > > > u32 reg; > > > > > int i; > > > > > = > > > > > @@ -561,6 +561,11 @@ static void advk_pcie_setup_hw(struct advk_p= cie *pcie) > > > > > reg |=3D LANE_COUNT_1; > > > > > advk_writel(pcie, reg, PCIE_CORE_CTRL0_REG); > > > > > = > > > > > + /* Set MSI address */ > > > > > + msi_addr =3D virt_to_phys(pcie); = > > > > = > > > > Strictly speaking, msi_addr should be a pci_bus_addr_t, not a > > > > phys_addr_t, and virt_to_phys() doesn't return a bus address. On second thought, probably a dma_addr_t, not a pci_bus_addr_t. > > > the problem here is that as far as we know currently there is no > > > function that converts a virtual address to pci_bus_addr_t like > > > virt_to_phys() does to convert to phys_addr_t. > > > = > > > On Armada 3720 there are PCIe Controller Address Decoder Registers, > > > which such a translating function would need to consult to do the > > > translation. But the default settings of these registers is to map PC= Ie > > > addresses 1 to 1 to physical addresses, and no driver changes these > > > registers. > > = > > The poorly-named pcibios_resource_to_bus() (I think the name is my > > fault) is the way to convert a CPU physical address to a PCI bus > > address. > = > But here it is needed to do something different. It is needed to do > inverse mapping of function which converts PCI bus address to CPU > physical address of CPU memory. So to converting CPU physical address of > CPU memory to PCI bus address from PCI bus point of view. > > I think that this information is stored in dma_ranges member of struct > pci_host_bridge. But function pcibios_resource_to_bus() looks at the > ->windows member which converts CPU physical address of PCI memory (not > CPU memory) to PCI bus address, which is something different. So > pcibios_resource_to_bus() would not work here and it may return > incorrect values (as PCI memory may be different from CPU point of view > and PCI bus point of view). Oh, sorry, indeed you are correct and I was completely on the wrong track. pcibios_resource_to_bus() is what you need for doing MMIO -- CPU accesses to things on PCI. MSI is the reverse. From the device's point of view, an MSI is basically a DMA as you allude to above, so I would expect the DMA API to be involved somehow. I do see a couple drivers using the DMA API for this: struct pcie_port { dma_addr_t msi_data; }; dw_pcie_host_init pp->msi_data =3D dma_map_single_attrs(..., &pp->msi_msg, ...) dw_pcie_setup_rc dw_pcie_msi_init u64 msi_target =3D (u64)pp->msi_data; dw_pcie_writel_dbi(pci, PCIE_MSI_ADDR_LO, lower_32_bits(msi_target)); dw_pci_setup_msi_msg u64 msi_target =3D (u64)pp->msi_data; msg->address_lo =3D lower_32_bits(msi_target); ----------------------------------------------------------- struct tegra_msi { dma_addr_t phys; }; tegra_pcie_probe tegra_pcie_msi_setup msi->virt =3D dma_alloc_attrs(..., &msi->phys, ...); tegra_pcie_enable_msi afi_writel(pcie, msi->phys, ...); tegra_compose_msi_msg msg->address_low =3D lower_32_bits(msi->phys); Bjorn _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel