From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-14.2 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 6B451C48BE5 for ; Wed, 16 Jun 2021 22:25:49 +0000 (UTC) Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 1C303613BD for ; Wed, 16 Jun 2021 22:25:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 1C303613BD Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=none smtp.mailfrom=linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:In-Reply-To:MIME-Version:Message-ID: Subject:Cc:To:From:Date:Reply-To:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID:References: List-Owner; bh=BbITsoOwDq1h5RueqG4ru3DBHXqy3XpJ+admm/ROTSY=; b=Vl/ZNflVf/3J4A l2EaP5IW8BxGDZpPSNBi/IGqGSVZpPR4DUEf/6rnmx8r+gLZrYqK3p91bt7Htayq3iUXIErYCiF4o mC4iyQG6fM6Z29SaRm0y5d5siaHiG/DBVDlSHL1RsyI89cubsYN30PmGXLx+anKVmYbD093jz0i7a iPHOLQBOmMURWtZDTTtVlD5EPdoTfYCSZg6UALWV0ZQL6hEH3FocRxnvdy+MYWolefZsUxv5Tn6rD VLAkZKtKwhY59XPq1zhfmDQxruWXsr2TI5qbmSjnLufyMZMbF/qdIcpKYE2c5iOUkupqggiMQVQz9 hhHAZQmz2N69y/gA27jw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltdx0-007rtp-0c; Wed, 16 Jun 2021 22:23:58 +0000 Received: from mail.kernel.org ([198.145.29.99]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1ltdwu-007rsY-R9 for linux-arm-kernel@lists.infradead.org; Wed, 16 Jun 2021 22:23:54 +0000 Received: by mail.kernel.org (Postfix) with ESMTPSA id 8433E61159; Wed, 16 Jun 2021 22:23:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1623882231; bh=5jpgqPY1x83qs5b9HXo/R6RwFGHHgxmM+YqIOckLd8A=; h=Date:From:To:Cc:Subject:In-Reply-To:From; b=JeEnLMTEZYCcZSFmUSzgEp7CDOeyaitOmjmyvqTor6jugDkiTMidtJ+z2xBfyf/Iu wu2xCdEwMdFSfZuQWNdFA6CqjIhoO7Mtv+dTAnqFYlVxGSO8D4mVuVzQXHj3uoRRDs EsBr0NcX25W6zW6jS8D1iaDCp5PG5/cjbd4dzPQYHDaWExqJEZeffw0sWVwIA/LGC0 zwTm7obq9+Mdj2DjdrKvDZ9Lf8ghIAoWegmuAaW9tfBrvr+OcETnFms7+2O43VrE4e 5C3hDlgCrXudwxD8b8mg/zWydsxWa1XWXPNx1XuBFRRvXP+nOkDCl6TV4YmNM9eITu Arrqze1rzdo+w== Date: Wed, 16 Jun 2021 17:23:50 -0500 From: Bjorn Helgaas To: Russell King Cc: Bjorn Helgaas , Arnd Bergmann , linux-arm-kernel@lists.infradead.org, linux-pci@vger.kernel.org Subject: Re: [PATCH] PCI: dynamically map ECAM regions Message-ID: <20210616222350.GA3013952@bjorn-Precision-5520> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20210616_152352_985351_1C975E2C X-CRM114-Status: GOOD ( 30.46 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Thu, May 13, 2021 at 03:18:27PM +0100, Russell King wrote: > Attempting to boot 32-bit ARM kernels under QEMU's 3.x virt models > fails when we have more than 512M of RAM in the model as we run out > of vmalloc space for the PCI ECAM regions. This failure will be > silent when running libvirt, as the console in that situation is a > PCI device. > > In this configuration, the kernel maps the whole ECAM, which QEMU > sets up for 256 buses, even when maybe only seven buses are in use. > Each bus uses 1M of ECAM space, and ioremap() adds an additional > guard page between allocations. The kernel vmap allocator will > align these regions to 512K, resulting in each mapping eating 1.5M > of vmalloc space. This means we need 384M of vmalloc space just to > map all of these, which is very wasteful of resources. > > Fix this by only mapping the ECAM for buses we are going to be using. > In my setups, this is around seven buses in most guests, which is > 10.5M of vmalloc space - way smaller than the 384M that would > otherwise be required. This also means that the kernel can boot > without forcing extra RAM into highmem with the vmalloc= argument, > or decreasing the virtual RAM available to the guest. > > Suggested-by: Arnd Bergmann > Signed-off-by: Russell King Applied with Arnd's reviewed-by to pci/enumeration for v5.14, thanks! > --- > drivers/pci/ecam.c | 54 ++++++++++++++++++++++++++++++++++------ > include/linux/pci-ecam.h | 1 + > 2 files changed, 47 insertions(+), 8 deletions(-) > > diff --git a/drivers/pci/ecam.c b/drivers/pci/ecam.c > index d2a1920bb055..1c40d2506aef 100644 > --- a/drivers/pci/ecam.c > +++ b/drivers/pci/ecam.c > @@ -32,7 +32,7 @@ struct pci_config_window *pci_ecam_create(struct device *dev, > struct pci_config_window *cfg; > unsigned int bus_range, bus_range_max, bsz; > struct resource *conflict; > - int i, err; > + int err; > > if (busr->start > busr->end) > return ERR_PTR(-EINVAL); > @@ -50,6 +50,7 @@ struct pci_config_window *pci_ecam_create(struct device *dev, > cfg->busr.start = busr->start; > cfg->busr.end = busr->end; > cfg->busr.flags = IORESOURCE_BUS; > + cfg->bus_shift = bus_shift; > bus_range = resource_size(&cfg->busr); > bus_range_max = resource_size(cfgres) >> bus_shift; > if (bus_range > bus_range_max) { > @@ -77,13 +78,6 @@ struct pci_config_window *pci_ecam_create(struct device *dev, > cfg->winp = kcalloc(bus_range, sizeof(*cfg->winp), GFP_KERNEL); > if (!cfg->winp) > goto err_exit_malloc; > - for (i = 0; i < bus_range; i++) { > - cfg->winp[i] = > - pci_remap_cfgspace(cfgres->start + i * bsz, > - bsz); > - if (!cfg->winp[i]) > - goto err_exit_iomap; > - } > } else { > cfg->win = pci_remap_cfgspace(cfgres->start, bus_range * bsz); > if (!cfg->win) > @@ -129,6 +123,44 @@ void pci_ecam_free(struct pci_config_window *cfg) > } > EXPORT_SYMBOL_GPL(pci_ecam_free); > > +static int pci_ecam_add_bus(struct pci_bus *bus) > +{ > + struct pci_config_window *cfg = bus->sysdata; > + unsigned int bsz = 1 << cfg->bus_shift; > + unsigned int busn = bus->number; > + phys_addr_t start; > + > + if (!per_bus_mapping) > + return 0; > + > + if (busn < cfg->busr.start || busn > cfg->busr.end) > + return -EINVAL; > + > + busn -= cfg->busr.start; > + start = cfg->res.start + busn * bsz; > + > + cfg->winp[busn] = pci_remap_cfgspace(start, bsz); > + if (!cfg->winp[busn]) > + return -ENOMEM; > + > + return 0; > +} > + > +static void pci_ecam_remove_bus(struct pci_bus *bus) > +{ > + struct pci_config_window *cfg = bus->sysdata; > + unsigned int busn = bus->number; > + > + if (!per_bus_mapping || busn < cfg->busr.start || busn > cfg->busr.end) > + return; > + > + busn -= cfg->busr.start; > + if (cfg->winp[busn]) { > + iounmap(cfg->winp[busn]); > + cfg->winp[busn] = NULL; > + } > +} > + > /* > * Function to implement the pci_ops ->map_bus method > */ > @@ -167,6 +199,8 @@ EXPORT_SYMBOL_GPL(pci_ecam_map_bus); > /* ECAM ops */ > const struct pci_ecam_ops pci_generic_ecam_ops = { > .pci_ops = { > + .add_bus = pci_ecam_add_bus, > + .remove_bus = pci_ecam_remove_bus, > .map_bus = pci_ecam_map_bus, > .read = pci_generic_config_read, > .write = pci_generic_config_write, > @@ -178,6 +212,8 @@ EXPORT_SYMBOL_GPL(pci_generic_ecam_ops); > /* ECAM ops for 32-bit access only (non-compliant) */ > const struct pci_ecam_ops pci_32b_ops = { > .pci_ops = { > + .add_bus = pci_ecam_add_bus, > + .remove_bus = pci_ecam_remove_bus, > .map_bus = pci_ecam_map_bus, > .read = pci_generic_config_read32, > .write = pci_generic_config_write32, > @@ -187,6 +223,8 @@ const struct pci_ecam_ops pci_32b_ops = { > /* ECAM ops for 32-bit read only (non-compliant) */ > const struct pci_ecam_ops pci_32b_read_ops = { > .pci_ops = { > + .add_bus = pci_ecam_add_bus, > + .remove_bus = pci_ecam_remove_bus, > .map_bus = pci_ecam_map_bus, > .read = pci_generic_config_read32, > .write = pci_generic_config_write, > diff --git a/include/linux/pci-ecam.h b/include/linux/pci-ecam.h > index 65d3d83015c3..944da75ff25c 100644 > --- a/include/linux/pci-ecam.h > +++ b/include/linux/pci-ecam.h > @@ -55,6 +55,7 @@ struct pci_ecam_ops { > struct pci_config_window { > struct resource res; > struct resource busr; > + unsigned int bus_shift; > void *priv; > const struct pci_ecam_ops *ops; > union { > -- > 2.20.1 > _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel