From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.3 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 70E01C433E0 for ; Fri, 3 Jul 2020 16:21:55 +0000 (UTC) Received: from hemlock.osuosl.org (smtp2.osuosl.org [140.211.166.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 3D98220870 for ; Fri, 3 Jul 2020 16:21:55 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (1024-bit key) header.d=kernel.org header.i=@kernel.org header.b="unrH+mW/" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3D98220870 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=iommu-bounces@lists.linux-foundation.org Received: from localhost (localhost [127.0.0.1]) by hemlock.osuosl.org (Postfix) with ESMTP id 15BA089730; Fri, 3 Jul 2020 16:21:55 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from hemlock.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id 4b9pUCCNnYSU; Fri, 3 Jul 2020 16:21:54 +0000 (UTC) Received: from lists.linuxfoundation.org (lf-lists.osuosl.org [140.211.9.56]) by hemlock.osuosl.org (Postfix) with ESMTP id 5D0A0895A8; Fri, 3 Jul 2020 16:21:54 +0000 (UTC) Received: from lf-lists.osuosl.org (localhost [127.0.0.1]) by lists.linuxfoundation.org (Postfix) with ESMTP id 41B9EC0888; Fri, 3 Jul 2020 16:21:54 +0000 (UTC) Received: from silver.osuosl.org (smtp3.osuosl.org [140.211.166.136]) by lists.linuxfoundation.org (Postfix) with ESMTP id 218AAC0733 for ; Fri, 3 Jul 2020 16:21:53 +0000 (UTC) Received: from localhost (localhost [127.0.0.1]) by silver.osuosl.org (Postfix) with ESMTP id 0853C204E1 for ; Fri, 3 Jul 2020 16:21:53 +0000 (UTC) X-Virus-Scanned: amavisd-new at osuosl.org Received: from silver.osuosl.org ([127.0.0.1]) by localhost (.osuosl.org [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id OE2BhzS7m4QL for ; Fri, 3 Jul 2020 16:21:52 +0000 (UTC) X-Greylist: domain auto-whitelisted by SQLgrey-1.7.6 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by silver.osuosl.org (Postfix) with ESMTPS id 4734E203F1 for ; Fri, 3 Jul 2020 16:21:52 +0000 (UTC) Received: from willie-the-truck (236.31.169.217.in-addr.arpa [217.169.31.236]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPSA id 9DBA7208E4; Fri, 3 Jul 2020 16:21:50 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=default; t=1593793312; bh=HhlNgQjMhz+GYZbL+CIhM+GpUQjJBmxjUQV7y4TEyig=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=unrH+mW//Md4+ho/OF+JI8Ue148MTSN6Upj5bJ7Xy26C5p0iVXFLE/HGSAFfeZlO+ 4szqHNKiY5JVlS3cVC9cC1kT9LnweMdBh9agWlR7WH0PO2aZ1GF3GCRKeHWNtme/NJ XiAaOcGZtY1we2bwxesxFu/uqxGspBD9EjJfZiFU= Date: Fri, 3 Jul 2020 17:21:47 +0100 From: Will Deacon To: Barry Song Subject: Re: [PATCH] iommu/arm-smmu-v3: allocate the memory of queues in local numa node Message-ID: <20200703162147.GB19780@willie-the-truck> References: <20200601113141.69488-1-song.bao.hua@hisilicon.com> MIME-Version: 1.0 Content-Disposition: inline In-Reply-To: <20200601113141.69488-1-song.bao.hua@hisilicon.com> User-Agent: Mutt/1.10.1 (2018-07-13) Cc: linuxarm@huawei.com, iommu@lists.linux-foundation.org, robin.murphy@arm.com, hch@lst.de, linux-arm-kernel@lists.infradead.org X-BeenThere: iommu@lists.linux-foundation.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: Development issues for Linux IOMMU support List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Errors-To: iommu-bounces@lists.linux-foundation.org Sender: "iommu" On Mon, Jun 01, 2020 at 11:31:41PM +1200, Barry Song wrote: > dmam_alloc_coherent() will usually allocate memory from the default CMA. For > a common arm64 defconfig without reserved memory in device tree, there is only > one CMA close to address 0. > dma_alloc_contiguous() will allocate memory without any idea of NUMA and smmu > has no customized per-numa cma_area. > struct page *dma_alloc_contiguous(struct device *dev, size_t size, gfp_t gfp) > { > size_t count = size >> PAGE_SHIFT; > struct page *page = NULL; > struct cma *cma = NULL; > > if (dev && dev->cma_area) > cma = dev->cma_area; > else if (count > 1) > cma = dma_contiguous_default_area; > > ... > return page; > } > > if there are N numa nodes, N-1 nodes will put command/evt queues etc in a > remote node the default CMA belongs to,probably node 0. Tests show, after > sending CMD_SYNC in an empty command queue, > on Node2, without this patch, it takes 550ns to wait for the completion > of CMD_SYNC; with this patch, it takes 250ns to wait for the completion > of CMD_SYNC. > > Signed-off-by: Barry Song > --- > drivers/iommu/arm-smmu-v3.c | 63 ++++++++++++++++++++++++++++--------- > 1 file changed, 48 insertions(+), 15 deletions(-) I would prefer that the coherent DMA allocator learned about NUMA, rather than we bodge drivers to use the streaming API where it doesn't really make sense. I see that you've posted other patches to do that (thanks!), so I'll disregard this series. Cheers, Will _______________________________________________ iommu mailing list iommu@lists.linux-foundation.org https://lists.linuxfoundation.org/mailman/listinfo/iommu