From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B7A11C5478C for ; Tue, 27 Feb 2024 17:43:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BC3A6B0108; Tue, 27 Feb 2024 12:43:07 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 18C896B0109; Tue, 27 Feb 2024 12:43:07 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id AD86A6B0105; Tue, 27 Feb 2024 12:43:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 39CF994000A for ; Tue, 27 Feb 2024 12:43:06 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id B62A41406C3 for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.05.118D8F7 Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf11.hostedemail.com (Postfix) with ESMTP id 0B31E40009 for ; Tue, 27 Feb 2024 17:42:59 +0000 (UTC) Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FzBqebYw; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GbovS5FPii+pMRhRNiTk/ANTExABfXg9lzmO/c4iads=; b=1A1L8CvNhiWcH3neTojtWRJzK/2tWb2JVNb5Y8Ts7sXxQ0U6tiGiKOB+1drc/3o+AbR04I lYxYm8GQmy9AD071lylte1zKMnyCKfWyUNwMh9g3zho+x858wDXTWwHjewz0OKguVujKaY wgbIomsAFrAin0dOq7gonfVLWa1gvL4= ARC-Authentication-Results: i=1; imf11.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=FzBqebYw; dmarc=none; spf=none (imf11.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=JTw8C8dnLwqcvlempgpPDh8HU7qnpjIHQMen5O5tKiJDpCdpsw2I/zI63aTS3VwEKjGFZt YuCzGbQKnGFD8bjgeTXaBmatGZ9fBkSaqQzxsCJA7438TSUce5hYns7Z9/nrC6WPWAwyzg Tgz61gqnIsROXLBoBNraijGxLaLB0lU= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=GbovS5FPii+pMRhRNiTk/ANTExABfXg9lzmO/c4iads=; b=FzBqebYwvSGCp8nWlvoCbpv+FQ I4WEGsuyw8nlTqlulYAN0xetVPSOJjcd8siGHSrecK2bSGyhXihq7BZc6hFZ8+6CI6Xl5O9tyrlgz ozKA4DeUMCg6XNzxuXVmYUmFs37Hb6HfwxKeW8gprfNT4DSaSK3QNOOUAXKXD+dgiJtN0G/+MI0hS mEM5vKnW1/Mk7CJokCspgQC4WhxADysRKoULkV9dtqdAQr1/fErwciODxYAraztF6LbOYft7j73A9 ZheFvxkdIsZ3FRwwSuZEDneXtZhxkiouzvnp/RZx8nkYgKrRFVDH/NH9IcUQsTWJrlHv/pF6oaL4T x3pBprLA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tm-00000002yrM-314L; Tue, 27 Feb 2024 17:42:58 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 08/18] mm: use __page_cache_release() in folios_put() Date: Tue, 27 Feb 2024 17:42:42 +0000 Message-ID: <20240227174254.710559-9-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 0B31E40009 X-Rspam-User: X-Rspamd-Server: rspam04 X-Stat-Signature: 6izx4yorbj7tuofiqyzrfwb7nmdrgkmb X-HE-Tag: 1709055779-323475 X-HE-Meta: U2FsdGVkX18+lsqDX4On6WnSEBnsjJYs7JN0lrAlRGPQ57OuRch/7PC2ECiLktMqHCgixE2l4pkoRAfKWQNMj7ApEOS/WyNNXVQX7gsG0UbZy0+Y7+CujyHYg7696318/+kaIxyDdfelhgYFwRGBVh7zKuV+MGYwfWlmZkROZF6PYzzFInFk0jmmB7vz7pStbeqF/spddiZgD2WbIxcaS3HombvpUXZ9xk1DlyLQ8bb6qcoNNcNIop6tVn2/FHp4TUQz9zt8LzBLMJAcrdCdU8rsNMxBR1G/K2i2xRzfiGahvVFAo0zRBKEJEhfsJtd1O3llX5PtOKhWWJqw9qCcVVCabaJqYSXPGvaWctaE/tYLOW70jC86t0/HJfH5CWf1evu8Loyk5SCc5mfSFQPzsY5W+VX6Dc/7sO/ludEUSIhckDTe6fSGydQjx7GcvKCM6gvOnovNGEItDzTSWjUwJfVZWihRdEi2T/XEWRb6pvvF8kV6YeIH7cbbuCD4NG3RGPxXDdf9R5oJq1kCbwkWYfSPr+xizO0uAHfRf4zJyl4hgpq+mfjGjByloxE/Co1P36MPG1ejDRqZp0uLvCE2seUWaaskwwqRTuwmX8pOHN+NBBonmhzRPzxXWtCBmiIbxtdA1bStS2URuewiG4HDaFV9zyUF20ncOeOF2JqetTNDxW+xETo6B5LcAWOr1Ei6rjDLJpJVL3GWBSOSBBJqcaBpsnbBkuynZAzK7+nXPdGrvBmwFMrSqg8j93hVusxi94bji/5/g6dWYDphvwwY8fgMDtSZYO1t1Wm9hG2gBy5QTKUpf4wNevk3b8OCo359V/PEjrHOuzdwtfKEu13+E4W0scGQWXcLSvViB6cJiChCeqxtWYIxQx6GAwsV+Ca7IX+URKSVvxaP3nDFOk6HtDY07lDJH1+JRYh7xkhHkwg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Pass a pointer to the lruvec so we can take advantage of the folio_lruvec_relock_irqsave(). Adjust the calling convention of folio_lruvec_relock_irqsave() to suit and add a page_cache_release() wrapper. Signed-off-by: Matthew Wilcox (Oracle) --- include/linux/memcontrol.h | 16 +++++----- mm/swap.c | 62 ++++++++++++++++++-------------------- 2 files changed, 37 insertions(+), 41 deletions(-) diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h index 46d9abb20761..8a0e8972a3d3 100644 --- a/include/linux/memcontrol.h +++ b/include/linux/memcontrol.h @@ -1705,18 +1705,18 @@ static inline struct lruvec *folio_lruvec_relock_irq(struct folio *folio, return folio_lruvec_lock_irq(folio); } -/* Don't lock again iff page's lruvec locked */ -static inline struct lruvec *folio_lruvec_relock_irqsave(struct folio *folio, - struct lruvec *locked_lruvec, unsigned long *flags) +/* Don't lock again iff folio's lruvec locked */ +static inline void folio_lruvec_relock_irqsave(struct folio *folio, + struct lruvec **lruvecp, unsigned long *flags) { - if (locked_lruvec) { - if (folio_matches_lruvec(folio, locked_lruvec)) - return locked_lruvec; + if (*lruvecp) { + if (folio_matches_lruvec(folio, *lruvecp)) + return; - unlock_page_lruvec_irqrestore(locked_lruvec, *flags); + unlock_page_lruvec_irqrestore(*lruvecp, *flags); } - return folio_lruvec_lock_irqsave(folio, flags); + *lruvecp = folio_lruvec_lock_irqsave(folio, flags); } #ifdef CONFIG_CGROUP_WRITEBACK diff --git a/mm/swap.c b/mm/swap.c index ad3f2e9448a4..dce5ea67ae05 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -74,22 +74,21 @@ static DEFINE_PER_CPU(struct cpu_fbatches, cpu_fbatches) = { .lock = INIT_LOCAL_LOCK(lock), }; -/* - * This path almost never happens for VM activity - pages are normally freed - * in batches. But it gets used by networking - and for compound pages. - */ -static void __page_cache_release(struct folio *folio) +static void __page_cache_release(struct folio *folio, struct lruvec **lruvecp, + unsigned long *flagsp) { if (folio_test_lru(folio)) { - struct lruvec *lruvec; - unsigned long flags; - - lruvec = folio_lruvec_lock_irqsave(folio, &flags); - lruvec_del_folio(lruvec, folio); + folio_lruvec_relock_irqsave(folio, lruvecp, flagsp); + lruvec_del_folio(*lruvecp, folio); __folio_clear_lru_flags(folio); - unlock_page_lruvec_irqrestore(lruvec, flags); } - /* See comment on folio_test_mlocked in folios_put() */ + + /* + * In rare cases, when truncation or holepunching raced with + * munlock after VM_LOCKED was cleared, Mlocked may still be + * found set here. This does not indicate a problem, unless + * "unevictable_pgs_cleared" appears worryingly large. + */ if (unlikely(folio_test_mlocked(folio))) { long nr_pages = folio_nr_pages(folio); @@ -99,9 +98,23 @@ static void __page_cache_release(struct folio *folio) } } +/* + * This path almost never happens for VM activity - pages are normally freed + * in batches. But it gets used by networking - and for compound pages. + */ +static void page_cache_release(struct folio *folio) +{ + struct lruvec *lruvec = NULL; + unsigned long flags; + + __page_cache_release(folio, &lruvec, &flags); + if (lruvec) + unlock_page_lruvec_irqrestore(lruvec, flags); +} + static void __folio_put_small(struct folio *folio) { - __page_cache_release(folio); + page_cache_release(folio); mem_cgroup_uncharge(folio); free_unref_page(&folio->page, 0); } @@ -115,7 +128,7 @@ static void __folio_put_large(struct folio *folio) * be called for hugetlb (it has a separate hugetlb_cgroup.) */ if (!folio_test_hugetlb(folio)) - __page_cache_release(folio); + page_cache_release(folio); destroy_large_folio(folio); } @@ -216,7 +229,7 @@ static void folio_batch_move_lru(struct folio_batch *fbatch, move_fn_t move_fn) if (move_fn != lru_add_fn && !folio_test_clear_lru(folio)) continue; - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, &flags); + folio_lruvec_relock_irqsave(folio, &lruvec, &flags); move_fn(lruvec, folio); folio_set_lru(folio); @@ -999,24 +1012,7 @@ void folios_put_refs(struct folio_batch *folios, unsigned int *refs) continue; } - if (folio_test_lru(folio)) { - lruvec = folio_lruvec_relock_irqsave(folio, lruvec, - &flags); - lruvec_del_folio(lruvec, folio); - __folio_clear_lru_flags(folio); - } - - /* - * In rare cases, when truncation or holepunching raced with - * munlock after VM_LOCKED was cleared, Mlocked may still be - * found set here. This does not indicate a problem, unless - * "unevictable_pgs_cleared" appears worryingly large. - */ - if (unlikely(folio_test_mlocked(folio))) { - __folio_clear_mlocked(folio); - zone_stat_sub_folio(folio, NR_MLOCK); - count_vm_event(UNEVICTABLE_PGCLEARED); - } + __page_cache_release(folio, &lruvec, &flags); if (j != i) folios->folios[j] = folio; -- 2.43.0