From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,UNPARSEABLE_RELAY,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 69923C433DF for ; Mon, 24 Aug 2020 12:56:16 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3103D20706 for ; Mon, 24 Aug 2020 12:56:16 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3103D20706 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=linux.alibaba.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id B4D4B8D0013; Mon, 24 Aug 2020 08:55:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AD8708D0014; Mon, 24 Aug 2020 08:55:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 929158D0013; Mon, 24 Aug 2020 08:55:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0026.hostedemail.com [216.40.44.26]) by kanga.kvack.org (Postfix) with ESMTP id 785C68D0014 for ; Mon, 24 Aug 2020 08:55:35 -0400 (EDT) Received: from smtpin12.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 490631EF1 for ; Mon, 24 Aug 2020 12:55:35 +0000 (UTC) X-FDA: 77185458630.12.shoe78_610108027053 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin12.hostedemail.com (Postfix) with ESMTP id 1905F1801273A for ; Mon, 24 Aug 2020 12:55:35 +0000 (UTC) X-HE-Tag: shoe78_610108027053 X-Filterd-Recvd-Size: 4595 Received: from out30-44.freemail.mail.aliyun.com (out30-44.freemail.mail.aliyun.com [115.124.30.44]) by imf09.hostedemail.com (Postfix) with ESMTP for ; Mon, 24 Aug 2020 12:55:33 +0000 (UTC) X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R691e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=e01f04397;MF=alex.shi@linux.alibaba.com;NM=1;PH=DS;RN=23;SR=0;TI=SMTPD_---0U6k9-bl_1598273712; Received: from aliy80.localdomain(mailfrom:alex.shi@linux.alibaba.com fp:SMTPD_---0U6k9-bl_1598273712) by smtp.aliyun-inc.com(127.0.0.1); Mon, 24 Aug 2020 20:55:23 +0800 From: Alex Shi To: akpm@linux-foundation.org, mgorman@techsingularity.net, tj@kernel.org, hughd@google.com, khlebnikov@yandex-team.ru, daniel.m.jordan@oracle.com, willy@infradead.org, hannes@cmpxchg.org, lkp@intel.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, cgroups@vger.kernel.org, shakeelb@google.com, iamjoonsoo.kim@lge.com, richard.weiyang@gmail.com, kirill@shutemov.name, alexander.duyck@gmail.com, rong.a.chen@intel.com, mhocko@suse.com, vdavydov.dev@gmail.com, shy828301@gmail.com Cc: Alexander Duyck , Stephen Rothwell Subject: [PATCH v18 29/32] mm: Identify compound pages sooner in isolate_migratepages_block Date: Mon, 24 Aug 2020 20:55:02 +0800 Message-Id: <1598273705-69124-30-git-send-email-alex.shi@linux.alibaba.com> X-Mailer: git-send-email 1.8.3.1 In-Reply-To: <1598273705-69124-1-git-send-email-alex.shi@linux.alibaba.com> References: <1598273705-69124-1-git-send-email-alex.shi@linux.alibaba.com> X-Rspamd-Queue-Id: 1905F1801273A X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck Since we are holding a reference to the page much sooner in isolate_migratepages_block we can move the PageCompound check out of the LRU locked section and instead just place it after get_page_unless_zero. By doing this we can allow any of the items that might trigger a failure to trigger a failure for the compound page rather than the order 0 page and as a result we should be able to process the pageblock faster. In addition by testing for PageCompound sooner we can avoid having the LRU flag cleared and then reset in the exception case. As a result this should prevent possible races where another thread might be attempting to pull the LRU pages from the list. Signed-off-by: Alexander Duyck Signed-off-by: Alex Shi Cc: Andrew Morton Cc: Stephen Rothwell Cc: linux-kernel@vger.kernel.org Cc: linux-mm@kvack.org --- mm/compaction.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index 6bf5ccd8fcf6..a0e48d079124 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -984,6 +984,24 @@ static bool too_many_isolated(pg_data_t *pgdat) if (unlikely(!get_page_unless_zero(page))) goto isolate_fail; + /* + * Page is compound. We know the order before we know if it is + * on the LRU so we cannot assume it is THP. However since the + * page will have the LRU validated shortly we can use the value + * to skip over this page for now or validate the LRU is set and + * then isolate the entire compound page if we are isolating to + * generate a CMA page. + */ + if (PageCompound(page)) { + const unsigned int order = compound_order(page); + + if (likely(order < MAX_ORDER)) + low_pfn += (1UL << order) - 1; + + if (!cc->alloc_contig) + goto isolate_fail_put; + } + if (__isolate_lru_page_prepare(page, isolate_mode) != 0) goto isolate_fail_put; @@ -1009,23 +1027,8 @@ static bool too_many_isolated(pg_data_t *pgdat) if (test_and_set_skip(cc, page, low_pfn)) goto isolate_abort; } - - /* - * Page become compound since the non-locked check, - * and it's on LRU. It can only be a THP so the order - * is safe to read and it's 0 for tail pages. - */ - if (unlikely(PageCompound(page) && !cc->alloc_contig)) { - low_pfn += compound_nr(page) - 1; - SetPageLRU(page); - goto isolate_fail_put; - } } - /* The whole page is taken off the LRU; skip the tail pages. */ - if (PageCompound(page)) - low_pfn += compound_nr(page) - 1; - /* Successfully isolated */ del_page_from_lru_list(page, lruvec, page_lru(page)); mod_node_page_state(page_pgdat(page), -- 1.8.3.1