From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER, INCLUDES_PATCH,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 48D80C4361B for ; Tue, 15 Dec 2020 20:34:25 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D6B9322AAE for ; Tue, 15 Dec 2020 20:34:24 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D6B9322AAE Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7157C8D000F; Tue, 15 Dec 2020 15:34:24 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 6C5578D000E; Tue, 15 Dec 2020 15:34:24 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5B4F68D000F; Tue, 15 Dec 2020 15:34:24 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0170.hostedemail.com [216.40.44.170]) by kanga.kvack.org (Postfix) with ESMTP id 424778D000E for ; Tue, 15 Dec 2020 15:34:24 -0500 (EST) Received: from smtpin10.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 0D72F363C for ; Tue, 15 Dec 2020 20:34:24 +0000 (UTC) X-FDA: 77596669248.10.wrist46_2f163a227426 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin10.hostedemail.com (Postfix) with ESMTP id E196E16A0AB for ; Tue, 15 Dec 2020 20:34:23 +0000 (UTC) X-HE-Tag: wrist46_2f163a227426 X-Filterd-Recvd-Size: 10267 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by imf37.hostedemail.com (Postfix) with ESMTP for ; Tue, 15 Dec 2020 20:34:23 +0000 (UTC) Date: Tue, 15 Dec 2020 12:34:20 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linux-foundation.org; s=korg; t=1608064462; bh=Nw7aksrDz7LrQPm+ighfoLv9YoD24ca39i1zcMsdOuo=; h=From:To:Subject:In-Reply-To:From; b=cQVSDXDDkNcYfbzByCNtJoAOBIdqam4Z4JHP2GajXpYNER3cfQbGNA4aKa20CqKcz cKNkG60ZrEMiRNOKQzOFf4o0j0VosS1Mwl+S3gKwCy2I0566yBpbnahQ31XfKfWt// +NA3JlEoC4WnT862V4sjhFOn3P5wJVAXXwZqwiLM= From: Andrew Morton To: aarcange@redhat.com, akpm@linux-foundation.org, alex.shi@linux.alibaba.com, alexander.duyck@gmail.com, aryabinin@virtuozzo.com, daniel.m.jordan@oracle.com, hannes@cmpxchg.org, hughd@google.com, iamjoonsoo.kim@lge.com, jannh@google.com, khlebnikov@yandex-team.ru, kirill.shutemov@linux.intel.com, kirill@shutemov.name, linux-mm@kvack.org, mgorman@techsingularity.net, mhocko@kernel.org, mhocko@suse.com, mika.penttila@nextfour.com, minchan@kernel.org, mm-commits@vger.kernel.org, richard.weiyang@gmail.com, rong.a.chen@intel.com, shakeelb@google.com, tglx@linutronix.de, tj@kernel.org, torvalds@linux-foundation.org, vbabka@suse.cz, vdavydov.dev@gmail.com, willy@infradead.org, yang.shi@linux.alibaba.com, ying.huang@intel.com Subject: [patch 15/19] mm/compaction: do page isolation first in compaction Message-ID: <20201215203420.HrcA96Ocz%akpm@linux-foundation.org> In-Reply-To: <20201215123253.954eca9a5ef4c0d52fd381fa@linux-foundation.org> User-Agent: s-nail v14.8.16 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: =46rom: Alex Shi Subject: mm/compaction: do page isolation first in compaction Currently, compaction would get the lru_lock and then do page isolation which works fine with pgdat->lru_lock, since any page isoltion would compete for the lru_lock. If we want to change to memcg lru_lock, we have to isolate the page before getting lru_lock, thus isoltion would block page's memcg change which relay on page isoltion too. Then we could safely use per memcg lru_lock later. The new page isolation use previous introduced TestClearPageLRU() + pgdat lru locking which will be changed to memcg lru lock later. Hugh Dickins fixed following bugs in this patch's early version: Fix lots of crashes under compaction load: isolate_migratepages_block() must clean up appropriately when rejecting a page, setting PageLRU again if it had been cleared; and a put_page() after get_page_unless_zero() cannot safely be done while holding locked_lruvec - it may turn out to be the final put_page(), which will take an lruvec lock when PageLRU. And move __isolate_lru_page_prepare back after get_page_unless_zero to make trylock_page() safe: trylock_page() is not safe to use at this time: its setting PG_locked can race with the page being freed or allocated ("Bad page"), and can also erase flags being set by one of those "sole owners" of a freshly allocated page who use non-atomic __SetPageFlag(). Link: https://lkml.kernel.org/r/1604566549-62481-16-git-send-email-alex.shi= @linux.alibaba.com Suggested-by: Johannes Weiner Signed-off-by: Alex Shi Acked-by: Hugh Dickins Acked-by: Johannes Weiner Acked-by: Vlastimil Babka Cc: Matthew Wilcox Cc: Alexander Duyck Cc: Andrea Arcangeli Cc: Andrey Ryabinin Cc: "Chen, Rong A" Cc: Daniel Jordan Cc: "Huang, Ying" Cc: Jann Horn Cc: Joonsoo Kim Cc: Kirill A. Shutemov Cc: Kirill A. Shutemov Cc: Konstantin Khlebnikov Cc: Mel Gorman Cc: Michal Hocko Cc: Michal Hocko Cc: Mika Penttil=C3=A4 Cc: Minchan Kim Cc: Shakeel Butt Cc: Tejun Heo Cc: Thomas Gleixner Cc: Vladimir Davydov Cc: Wei Yang Cc: Yang Shi Signed-off-by: Andrew Morton --- include/linux/swap.h | 2 - mm/compaction.c | 42 +++++++++++++++++++++++++++++++--------- mm/vmscan.c | 43 ++++++++++++++++++++--------------------- 3 files changed, 56 insertions(+), 31 deletions(-) --- a/include/linux/swap.h~mm-compaction-do-page-isolation-first-in-compact= ion +++ a/include/linux/swap.h @@ -356,7 +356,7 @@ extern void lru_cache_add_inactive_or_un extern unsigned long zone_reclaimable_pages(struct zone *zone); extern unsigned long try_to_free_pages(struct zonelist *zonelist, int orde= r, gfp_t gfp_mask, nodemask_t *mask); -extern int __isolate_lru_page(struct page *page, isolate_mode_t mode); +extern int __isolate_lru_page_prepare(struct page *page, isolate_mode_t mo= de); extern unsigned long try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, unsigned long nr_pages, gfp_t gfp_mask, --- a/mm/compaction.c~mm-compaction-do-page-isolation-first-in-compaction +++ a/mm/compaction.c @@ -890,6 +890,7 @@ isolate_migratepages_block(struct compac if (!valid_page && IS_ALIGNED(low_pfn, pageblock_nr_pages)) { if (!cc->ignore_skip_hint && get_pageblock_skip(page)) { low_pfn =3D end_pfn; + page =3D NULL; goto isolate_abort; } valid_page =3D page; @@ -971,6 +972,21 @@ isolate_migratepages_block(struct compac if (!(cc->gfp_mask & __GFP_FS) && page_mapping(page)) goto isolate_fail; =20 + /* + * Be careful not to clear PageLRU until after we're + * sure the page is not being freed elsewhere -- the + * page release code relies on it. + */ + if (unlikely(!get_page_unless_zero(page))) + goto isolate_fail; + + if (__isolate_lru_page_prepare(page, isolate_mode) !=3D 0) + goto isolate_fail_put; + + /* Try isolate the page */ + if (!TestClearPageLRU(page)) + goto isolate_fail_put; + /* If we already hold the lock, we can skip some rechecking */ if (!locked) { locked =3D compact_lock_irqsave(&pgdat->lru_lock, @@ -983,10 +999,6 @@ isolate_migratepages_block(struct compac goto isolate_abort; } =20 - /* Recheck PageLRU and PageCompound under lock */ - if (!PageLRU(page)) - goto isolate_fail; - /* * Page become compound since the non-locked check, * and it's on LRU. It can only be a THP so the order @@ -994,16 +1006,13 @@ isolate_migratepages_block(struct compac */ if (unlikely(PageCompound(page) && !cc->alloc_contig)) { low_pfn +=3D compound_nr(page) - 1; - goto isolate_fail; + SetPageLRU(page); + goto isolate_fail_put; } } =20 lruvec =3D mem_cgroup_page_lruvec(page, pgdat); =20 - /* Try isolate the page */ - if (__isolate_lru_page(page, isolate_mode) !=3D 0) - goto isolate_fail; - /* The whole page is taken off the LRU; skip the tail pages. */ if (PageCompound(page)) low_pfn +=3D compound_nr(page) - 1; @@ -1032,6 +1041,15 @@ isolate_success: } =20 continue; + +isolate_fail_put: + /* Avoid potential deadlock in freeing page under lru_lock */ + if (locked) { + spin_unlock_irqrestore(&pgdat->lru_lock, flags); + locked =3D false; + } + put_page(page); + isolate_fail: if (!skip_on_failure) continue; @@ -1068,9 +1086,15 @@ isolate_fail: if (unlikely(low_pfn > end_pfn)) low_pfn =3D end_pfn; =20 + page =3D NULL; + isolate_abort: if (locked) spin_unlock_irqrestore(&pgdat->lru_lock, flags); + if (page) { + SetPageLRU(page); + put_page(page); + } =20 /* * Updated the cached scanner pfn once the pageblock has been scanned --- a/mm/vmscan.c~mm-compaction-do-page-isolation-first-in-compaction +++ a/mm/vmscan.c @@ -1539,7 +1539,7 @@ unsigned int reclaim_clean_pages_from_li * * returns 0 on success, -ve errno on failure. */ -int __isolate_lru_page(struct page *page, isolate_mode_t mode) +int __isolate_lru_page_prepare(struct page *page, isolate_mode_t mode) { int ret =3D -EBUSY; =20 @@ -1591,22 +1591,9 @@ int __isolate_lru_page(struct page *page if ((mode & ISOLATE_UNMAPPED) && page_mapped(page)) return ret; =20 - if (likely(get_page_unless_zero(page))) { - /* - * Be careful not to clear PageLRU until after we're - * sure the page is not being freed elsewhere -- the - * page release code relies on it. - */ - if (TestClearPageLRU(page)) - ret =3D 0; - else - put_page(page); - } - - return ret; + return 0; } =20 - /* * Update LRU sizes after isolating pages. The LRU size updates must * be complete before mem_cgroup_update_lru_size due to a sanity check. @@ -1686,20 +1673,34 @@ static unsigned long isolate_lru_pages(u * only when the page is being freed somewhere else. */ scan +=3D nr_pages; - switch (__isolate_lru_page(page, mode)) { + switch (__isolate_lru_page_prepare(page, mode)) { case 0: + /* + * Be careful not to clear PageLRU until after we're + * sure the page is not being freed elsewhere -- the + * page release code relies on it. + */ + if (unlikely(!get_page_unless_zero(page))) + goto busy; + + if (!TestClearPageLRU(page)) { + /* + * This page may in other isolation path, + * but we still hold lru_lock. + */ + put_page(page); + goto busy; + } + nr_taken +=3D nr_pages; nr_zone_taken[page_zonenum(page)] +=3D nr_pages; list_move(&page->lru, dst); break; =20 - case -EBUSY: + default: +busy: /* else it is being freed elsewhere */ list_move(&page->lru, src); - continue; - - default: - BUG(); } } =20 _