From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id 029BCC433F5 for ; Fri, 15 Apr 2022 06:07:26 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender: Content-Transfer-Encoding:Content-Type:List-Subscribe:List-Help:List-Post: List-Archive:List-Unsubscribe:List-Id:Cc:To:Subject:Message-ID:Date:From: In-Reply-To:References:MIME-Version:Reply-To:Content-ID:Content-Description: Resent-Date:Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID: List-Owner; bh=dbw1f0Vvkj7Q93d7xjol6MFE+YIgRmG/yA/x5NhGzrk=; b=G2ukWAQrP2uWgh a2i+7UaCAMsKpEVA1GR16o1UHyym3qFkl/rwTbpa1ztwA0mYVP0/FbiCd6ICyhnpFXCc0g5pGSfN8 G35NbOC6Y397wV0vWrg/yudnl5v+/KJTYzyk2eMzAiNCQtsOcfvpCkeqs1WCX4EuZ0IItlnveSt4I aBFX86AWDpA/oTmtYOpKyazyD9oYVSGmWaWVB2soAf3yxUvD9P+vgfcOfCS+w0P0GHXgp2U7TWiIE TeM9OTYnrfr7KTJ3OaqJmd/fI2t4PLjo+1vbziaGQFAtu2L8yUzM2v+NODUkWW10zletfP6PV7RdK f/vggCaaZXy+VH5Igh4w==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfF5a-008rmB-70; Fri, 15 Apr 2022 06:05:51 +0000 Received: from dfw.source.kernel.org ([2604:1380:4641:c500::1]) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nfF3x-008qp4-Nd for linux-arm-kernel@lists.infradead.org; Fri, 15 Apr 2022 06:04:11 +0000 Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D923461EFC for ; Fri, 15 Apr 2022 06:04:08 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id C4841C385B0 for ; Fri, 15 Apr 2022 06:04:07 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1650002647; bh=CfFBFrsOrM1D6QuDY5JL6c2VKraxHdnJ8iQxVuMosgw=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=rQq1sZ4J1sg22E9oqdhg4EJn8neJLzquXI+ZjaLeifOERti0Wc+sHK1ca9AEuAedm amy5K2gapC1d/YsBm8lKYZa4FJlvxfvg2uMYXZW+FECTSlN5rRK7n46i4eYJiT8dUP NtiYz25I/4SLS+ajSx8TMqQAykzYQpMyTAcuLWze4mw13G0A4DVOegWZUDW0M2S/tI uzeS54JMWY8wDnTklfomrz11JZpKfoakRJTc9X4u3F2pgMPd0kcfv1uydJuck6C97z CHIcWL2MuEdFh32B9IwZ2OpEt+7syLJujcZiPCe8uAwUQ8XxGWdZKvFcfl9sbse5l2 wrEusSELhVbcw== Received: by mail-oi1-f181.google.com with SMTP id k10so7666240oia.0 for ; Thu, 14 Apr 2022 23:04:07 -0700 (PDT) X-Gm-Message-State: AOAM530MbPd2P4pyG3QUHjOpb9WMy3QbMqyJYIMDIflOKIOeySJzbpHe QLWIAalduN/ijP1wp3LyYVNM6uvr8CPQ1XULdh0= X-Google-Smtp-Source: ABdhPJz2xwpnXQaQcae68WnDv78hot7NwSWZe+t2TtHgLprVUTWzxwQZi9+tdzNS0HtyfFS5Ou/0nwNdizG5ZHLSuiQ= X-Received: by 2002:a05:6808:1596:b0:2f7:5d89:eec7 with SMTP id t22-20020a056808159600b002f75d89eec7mr1017334oiw.228.1650002646826; Thu, 14 Apr 2022 23:04:06 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Ard Biesheuvel Date: Fri, 15 Apr 2022 08:03:54 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH 07/10] crypto: Use ARCH_DMA_MINALIGN instead of ARCH_KMALLOC_MINALIGN To: Linus Torvalds Cc: Catalin Marinas , Herbert Xu , Will Deacon , Marc Zyngier , Arnd Bergmann , Greg Kroah-Hartman , Andrew Morton , Linux Memory Management List , Linux ARM , Linux Kernel Mailing List , "David S. Miller" X-CRM114-Version: 20100106-BlameMichelson ( TRE 0.8.0 (BSD) ) MR-646709E3 X-CRM114-CacheID: sfid-20220414_230409_888688_87C1AA8B X-CRM114-Status: GOOD ( 35.83 ) X-BeenThere: linux-arm-kernel@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Sender: "linux-arm-kernel" Errors-To: linux-arm-kernel-bounces+linux-arm-kernel=archiver.kernel.org@lists.infradead.org On Fri, 15 Apr 2022 at 00:26, Linus Torvalds wrote: > > On Thu, Apr 14, 2022 at 12:49 PM Catalin Marinas > wrote: > > > > It's a lot worse, ARCH_KMALLOC_MINALIGN is currently 128 bytes on arm64. > > I want to at least get it down to 64 with this series while preserving > > the current kmalloc() semantics. > > So here's a thought - maybe we could do the reverse of GFP_DMA, and > add a flag to the places that want small allocations and know they > don't need DMA? > > Because even 64 bytes is _huge_ for those small allocations. And 128 > bytes is just insane. > > Maybe we don't have a ton of them, but I just checked my desktop, and > if my laptop had 17k 8-byte allocation, on my desktop it's currently > sitting at 43k of them. And I've got a lot of 16- and 32-byte ones > too: > > kmalloc-32 51529 51712 32 128 1 : > kmalloc-16 45056 45056 16 256 1 : > kmalloc-8 43008 43008 8 512 1 : > > Don't ask me what they are. It's probably fairly easy to find out, and > it's probably something silly like sysfs private pointer data or > whatever. > > If I did my math right, with a 128-byte minimum allocation, that is > 16MB of wasted memory. > > Yeah, yeah, I've got 64GB or RAM in this thing, and there are things > that take a lot more memory than the above (mem_map etc), and 64MB is > peanuts at just 0.1% of RAM. > Actually, I think the impact on D-cache efficiency is a much bigger concern, although I don't have any data to back that up. On arm64, every 8 byte allocation takes up an entire cacheline, pushing out other data that we could meaningfully keep there. And the doubling to 128 that is unnecessary on most arm64 systems probably has an additional negative effect, as it means those allocations are heavily skewed to use even indexes into the cache sets, causing extra contention. > Maybe nobody cares. But I really feel bad about that kind of egregious > waste. The mem_map[] array may be big, it may use 1.5% of the memory I > have, but at least it *does* something. > > And it could be that if I have 150k of those smallish allocations, a > server with lots of active users might have millions. Not having > looked at where they come from, maybe that isn't the case, but it > *might* be. > > Maybe adding something like a > > static int warn_every_1k = 0; > WARN_ON(size < 32 && (1023 & ++warn_every_1k)); > > to kmalloc() would give us a statistical view of "lots of these small > allocations" thing, and we could add GFP_NODMA to them. There probably > aren't that many places that have those small allocations, and it's > certainly safer to annotate "this is not for DMA" than have the > requirement that all DMA allocations must be marked. > > Then just teach kmalloc() something like > > align = (gfp_flags & __GFP_NODMA) ? 8 : 128; > > on arm. > Sounds like a lot of churn as well, tbh. But perhaps there are a few hot spots that we can fix that would alleviate the bulk of the issue, who knows? _______________________________________________ linux-arm-kernel mailing list linux-arm-kernel@lists.infradead.org http://lists.infradead.org/mailman/listinfo/linux-arm-kernel