From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-16.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 623B5C2B9F8 for ; Tue, 25 May 2021 08:02:15 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0AF9D613F9 for ; Tue, 25 May 2021 08:02:15 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0AF9D613F9 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=techsingularity.net Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 1F1436B0073; Tue, 25 May 2021 04:02:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 174A06B0074; Tue, 25 May 2021 04:02:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id BCB256B0075; Tue, 25 May 2021 04:02:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0058.hostedemail.com [216.40.44.58]) by kanga.kvack.org (Postfix) with ESMTP id 6E95D6B0073 for ; Tue, 25 May 2021 04:02:13 -0400 (EDT) Received: from smtpin03.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 0E6BD9895 for ; Tue, 25 May 2021 08:02:13 +0000 (UTC) X-FDA: 78179010546.03.57546FA Received: from outbound-smtp61.blacknight.com (outbound-smtp61.blacknight.com [46.22.136.249]) by imf01.hostedemail.com (Postfix) with ESMTP id 64DEE500165A for ; Tue, 25 May 2021 08:02:06 +0000 (UTC) Received: from mail.blacknight.com (pemlinmail06.blacknight.ie [81.17.255.152]) by outbound-smtp61.blacknight.com (Postfix) with ESMTPS id 46687FAB32 for ; Tue, 25 May 2021 09:02:11 +0100 (IST) Received: (qmail 6155 invoked from network); 25 May 2021 08:02:10 -0000 Received: from unknown (HELO stampy.112glenside.lan) (mgorman@techsingularity.net@[84.203.23.168]) by 81.17.254.9 with ESMTPA; 25 May 2021 08:02:10 -0000 From: Mel Gorman To: Andrew Morton Cc: Hillf Danton , Dave Hansen , Vlastimil Babka , Michal Hocko , LKML , Linux-MM , Mel Gorman Subject: [PATCH 4/6] mm/page_alloc: Scale the number of pages that are batch freed Date: Tue, 25 May 2021 09:01:17 +0100 Message-Id: <20210525080119.5455-5-mgorman@techsingularity.net> X-Mailer: git-send-email 2.26.2 In-Reply-To: <20210525080119.5455-1-mgorman@techsingularity.net> References: <20210525080119.5455-1-mgorman@techsingularity.net> MIME-Version: 1.0 X-Rspamd-Queue-Id: 64DEE500165A Authentication-Results: imf01.hostedemail.com; dkim=none; spf=pass (imf01.hostedemail.com: domain of mgorman@techsingularity.net designates 46.22.136.249 as permitted sender) smtp.mailfrom=mgorman@techsingularity.net; dmarc=none X-Rspamd-Server: rspam03 X-Stat-Signature: my7b9zre35soibir6mmgqrktt9yt8w3n X-HE-Tag: 1621929726-373890 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: When a task is freeing a large number of order-0 pages, it may acquire the zone->lock multiple times freeing pages in batches. This may unnecessarily contend on the zone lock when freeing very large number of pages. This patch adapts the size of the batch based on the recent pattern to scale the batch size for subsequent frees. As the machines I used were not large enough to test this are not large enough to illustrate a problem, a debugging patch shows patterns like the following (slightly editted for clarity) Baseline vanilla kernel time-unmap-14426 [...] free_pcppages_bulk: free 63 count 378 high = 378 time-unmap-14426 [...] free_pcppages_bulk: free 63 count 378 high = 378 time-unmap-14426 [...] free_pcppages_bulk: free 63 count 378 high = 378 time-unmap-14426 [...] free_pcppages_bulk: free 63 count 378 high = 378 time-unmap-14426 [...] free_pcppages_bulk: free 63 count 378 high = 378 With patches time-unmap-7724 [...] free_pcppages_bulk: free 126 count 814 high = 814 time-unmap-7724 [...] free_pcppages_bulk: free 252 count 814 high = 814 time-unmap-7724 [...] free_pcppages_bulk: free 504 count 814 high = 814 time-unmap-7724 [...] free_pcppages_bulk: free 751 count 814 high = 814 time-unmap-7724 [...] free_pcppages_bulk: free 751 count 814 high = 814 Signed-off-by: Mel Gorman Acked-by: Dave Hansen --- include/linux/mmzone.h | 3 ++- mm/page_alloc.c | 41 +++++++++++++++++++++++++++++++++++++++-- 2 files changed, 41 insertions(+), 3 deletions(-) diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h index b449151745d7..92182e0299b2 100644 --- a/include/linux/mmzone.h +++ b/include/linux/mmzone.h @@ -343,8 +343,9 @@ struct per_cpu_pages { int count; /* number of pages in the list */ int high; /* high watermark, emptying needed */ int batch; /* chunk size for buddy add/remove */ + short free_factor; /* batch scaling factor during free */ #ifdef CONFIG_NUMA - int expire; /* When 0, remote pagesets are drained */ + short expire; /* When 0, remote pagesets are drained */ #endif =20 /* Lists of pages, one per migrate type stored on the pcp-lists */ diff --git a/mm/page_alloc.c b/mm/page_alloc.c index dc4ac309bc21..89e60005dd27 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3267,18 +3267,47 @@ static bool free_unref_page_prepare(struct page *= page, unsigned long pfn) return true; } =20 +static int nr_pcp_free(struct per_cpu_pages *pcp, int high, int batch) +{ + int min_nr_free, max_nr_free; + + /* Check for PCP disabled or boot pageset */ + if (unlikely(high < batch)) + return 1; + + /* Leave at least pcp->batch pages on the list */ + min_nr_free =3D batch; + max_nr_free =3D high - batch; + + /* + * Double the number of pages freed each time there is subsequent + * freeing of pages without any allocation. + */ + batch <<=3D pcp->free_factor; + if (batch < max_nr_free) + pcp->free_factor++; + batch =3D clamp(batch, min_nr_free, max_nr_free); + + return batch; +} + static void free_unref_page_commit(struct page *page, unsigned long pfn, int migratetype) { struct zone *zone =3D page_zone(page); struct per_cpu_pages *pcp; + int high; =20 __count_vm_event(PGFREE); pcp =3D this_cpu_ptr(zone->per_cpu_pageset); list_add(&page->lru, &pcp->lists[migratetype]); pcp->count++; - if (pcp->count >=3D READ_ONCE(pcp->high)) - free_pcppages_bulk(zone, READ_ONCE(pcp->batch), pcp); + high =3D READ_ONCE(pcp->high); + if (pcp->count >=3D high) { + int batch =3D READ_ONCE(pcp->batch); + + free_pcppages_bulk(zone, nr_pcp_free(pcp, high, batch), pcp); + } } =20 /* @@ -3530,7 +3559,14 @@ static struct page *rmqueue_pcplist(struct zone *p= referred_zone, unsigned long flags; =20 local_lock_irqsave(&pagesets.lock, flags); + + /* + * On allocation, reduce the number of pages that are batch freed. + * See nr_pcp_free() where free_factor is increased for subsequent + * frees. + */ pcp =3D this_cpu_ptr(zone->per_cpu_pageset); + pcp->free_factor >>=3D 1; list =3D &pcp->lists[migratetype]; page =3D __rmqueue_pcplist(zone, migratetype, alloc_flags, pcp, list); local_unlock_irqrestore(&pagesets.lock, flags); @@ -6698,6 +6734,7 @@ static void per_cpu_pages_init(struct per_cpu_pages= *pcp, struct per_cpu_zonesta */ pcp->high =3D BOOT_PAGESET_HIGH; pcp->batch =3D BOOT_PAGESET_BATCH; + pcp->free_factor =3D 0; } =20 static void __zone_set_pageset_high_and_batch(struct zone *zone, unsigne= d long high, --=20 2.26.2