From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.2 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 3B8FCC433E0 for ; Wed, 24 Feb 2021 11:55:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 8461A64DF3 for ; Wed, 24 Feb 2021 11:55:14 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 8461A64DF3 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id E54F56B0006; Wed, 24 Feb 2021 06:55:13 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id DDEAB6B006C; Wed, 24 Feb 2021 06:55:13 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CCC0D6B006E; Wed, 24 Feb 2021 06:55:13 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0034.hostedemail.com [216.40.44.34]) by kanga.kvack.org (Postfix) with ESMTP id B29006B0006 for ; Wed, 24 Feb 2021 06:55:13 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 80CE1824934B for ; Wed, 24 Feb 2021 11:55:13 +0000 (UTC) X-FDA: 77853005706.01.2509A0F Received: from outbound-smtp30.blacknight.com (outbound-smtp30.blacknight.com [81.17.249.61]) by imf16.hostedemail.com (Postfix) with ESMTP id 1987180192E2 for ; Wed, 24 Feb 2021 11:55:11 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail01.blacknight.ie [81.17.254.10]) by outbound-smtp30.blacknight.com (Postfix) with ESMTPS id 45F0EBAA8E for ; Wed, 24 Feb 2021 11:55:11 +0000 (GMT) Received: (qmail 8759 invoked from network); 24 Feb 2021 11:55:11 -0000 Received: from unknown (HELO techsingularity.net) (mgorman@techsingularity.net@[84.203.22.4]) by 81.17.254.9 with ESMTPSA (AES256-SHA encrypted, authenticated); 24 Feb 2021 11:55:11 -0000 Date: Wed, 24 Feb 2021 11:55:08 +0000 From: Mel Gorman To: Jesper Dangaard Brouer Cc: Chuck Lever , LKML , Linux-Net , Linux-MM , Linux-NFS Subject: Re: [RFC PATCH 0/3] Introduce a bulk order-0 page allocator for sunrpc Message-ID: <20210224115508.GL3697@techsingularity.net> References: <20210224102603.19524-1-mgorman@techsingularity.net> <20210224122723.15943e95@carbon> MIME-Version: 1.0 Content-Type: text/plain; charset=iso-8859-15 Content-Disposition: inline In-Reply-To: <20210224122723.15943e95@carbon> User-Agent: Mutt/1.10.1 (2018-07-13) X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: 1987180192E2 X-Stat-Signature: s7pqmqijp4681papd8nx16b9f8j1tpmq Received-SPF: none (techsingularity.net>: No applicable sender policy available) receiver=imf16; identity=mailfrom; envelope-from=""; helo=outbound-smtp30.blacknight.com; client-ip=81.17.249.61 X-HE-DKIM-Result: none/none X-HE-Tag: 1614167711-472904 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Feb 24, 2021 at 12:27:23PM +0100, Jesper Dangaard Brouer wrote: > On Wed, 24 Feb 2021 10:26:00 +0000 > Mel Gorman wrote: > > > This is a prototype series that introduces a bulk order-0 page allocator > > with sunrpc being the first user. The implementation is not particularly > > efficient and the intention is to iron out what the semantics of the API > > should be. That said, sunrpc was reported to have reduced allocation > > latency when refilling a pool. > > I also have a use-case in page_pool, and I've been testing with the > earlier patches, results are here[1] > > [1] https://github.com/xdp-project/xdp-project/blob/master/areas/mem/page_pool06_alloc_pages_bulk.org > > Awesome to see this newer patchset! thanks a lot for working on this! > I'll run some new tests based on this. > Thanks and if they get finalised, a patch on top for review would be nice with the results included in the changelog. Obviously any change that would need to be made to the allocator would happen first. > > As a side-note, while the implementation could be more efficient, it > > would require fairly deep surgery in numerous places. The lock scope would > > need to be significantly reduced, particularly as vmstat, per-cpu and the > > buddy allocator have different locking protocol that overal -- e.g. all > > partially depend on irqs being disabled at various points. Secondly, > > the core of the allocator deals with single pages where as both the bulk > > allocator and per-cpu allocator operate in batches. All of that has to > > be reconciled with all the existing users and their constraints (memory > > offline, CMA and cpusets being the trickiest). > > As you can see in[1], I'm getting a significant speedup from this. I > guess that the cost of finding the "zone" is higher than I expected, as > this basically what we/you amortize for the bulk. > The obvious goal would be that if a refactoring did happen that the performance would be at least neutral but hopefully improved. > > In terms of semantics required by new users, my preference is that a pair > > of patches be applied -- the first which adds the required semantic to > > the bulk allocator and the second which adds the new user. > > > > Patch 1 of this series is a cleanup to sunrpc, it could be merged > > separately but is included here for convenience. > > > > Patch 2 is the prototype bulk allocator > > > > Patch 3 is the sunrpc user. Chuck also has a patch which further caches > > pages but is not included in this series. It's not directly > > related to the bulk allocator and as it caches pages, it might > > have other concerns (e.g. does it need a shrinker?) > > > > This has only been lightly tested on a low-end NFS server. It did not break > > but would benefit from an evaluation to see how much, if any, the headline > > performance changes. The biggest concern is that a light test case showed > > that there are a *lot* of bulk requests for 1 page which gets delegated to > > the normal allocator. The same criteria should apply to any other users. > > If you change local_irq_save(flags) to local_irq_disable() then you can > likely get better performance for 1 page requests via this API. This > limits the API to be used in cases where IRQs are enabled (which is > most cases). (For my use-case I will not do 1 page requests). > I do not want to constrain the API to being IRQ-only prematurely. An obvious alternative use case is the SLUB allocation path when a high-order allocation fails. It's known that if the SLUB order is reduced that it has an impact on hackbench communicating over sockets. It would be interesting to see what happens if order-0 pages are bulk allocated when s->min == 0 and that can be called from a blocking context. Tricky to test but could be fudged by forcing all high-order allocations to fail when s->min == 0 to evaluate the worst case scenario. In addition, it would constrain any potential refactoring if the lower levels have to choose between local_irq_disable() vs local_irq_save() depending on the caller context. -- Mel Gorman SUSE Labs