From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 46F9CC433E1 for ; Wed, 19 Aug 2020 04:27:11 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 0476220772 for ; Wed, 19 Aug 2020 04:27:11 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="pO3zqnqL" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 0476220772 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9CC9B8D0002; Wed, 19 Aug 2020 00:27:10 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A2BD6B0036; Wed, 19 Aug 2020 00:27:10 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8B8A98D0002; Wed, 19 Aug 2020 00:27:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0237.hostedemail.com [216.40.44.237]) by kanga.kvack.org (Postfix) with ESMTP id 7661A6B0033 for ; Wed, 19 Aug 2020 00:27:10 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 15BE2180AD817 for ; Wed, 19 Aug 2020 04:27:10 +0000 (UTC) X-FDA: 77166033420.05.bear22_391568727025 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id CEA3618016858 for ; Wed, 19 Aug 2020 04:27:09 +0000 (UTC) X-HE-Tag: bear22_391568727025 X-Filterd-Recvd-Size: 6126 Received: from mail-qv1-f67.google.com (mail-qv1-f67.google.com [209.85.219.67]) by imf02.hostedemail.com (Postfix) with ESMTP for ; Wed, 19 Aug 2020 04:27:09 +0000 (UTC) Received: by mail-qv1-f67.google.com with SMTP id dd12so10712159qvb.0 for ; Tue, 18 Aug 2020 21:27:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:from:to:cc:date:message-id:in-reply-to:references :user-agent:mime-version:content-transfer-encoding; bh=O0GPbeNx4pE5Cj/la6lYT0zllz6aGU2XNC8iLf9uFA4=; b=pO3zqnqL4c/mDAjAfh4ZWDwEb7d57VhBDoidBgKwr3U5coFnxDOXwnl2WrCcEy2ytc vPoW0tq9uKas+aNj5L2ZSate75S8hark40KC/B1fHTbUYPsjwNaa6uOzkoMyUQDpmlBL dOt7/SWEQPgLq23TAemQobbfIE2Jpi9Y99z4Ne9S+gMTKMvQKs3YapyvSlEmI2di09pJ dHjRXJ2gP38hCNvvOZk4UqKmcWaQRFnTLqeFlhyGet0x8XXRiPSc12oSao2IOrQonhDN 0qNWU11jMey8c9BEr8FQP9SCHMrSyMK8EtIwBEOpavLMBYW6DnNK/t44D+cr7nXdgrJv 7tUg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:from:to:cc:date:message-id:in-reply-to :references:user-agent:mime-version:content-transfer-encoding; bh=O0GPbeNx4pE5Cj/la6lYT0zllz6aGU2XNC8iLf9uFA4=; b=uMGp8gkk7oUHStXvOV5Ak6PdUi0ux6w04SjUAZEfhdKlogbArWLrwEL6KlSQbEX9S7 3DjQXMMZ6tpFkgdBaCC1eCc38SASKo2qQE3UHNxb7kYc2Xu9AVyjpcgucNMs77yX0T68 /hJnYeM2xk+vU0iADI+PJAoWcTitZ0dn9A83yTlLeWWLiQv/Mc8XRui8hDjpTMlRHRyR WKJgk/qOrtcVAhC7rPls4szFk/R9DDRieEgBg3lF78Bni0LbsS5E3Ei/SQsWQinW/FOg AmvTQRYebgAJpv3BiSAQLbazTqlqtqMJAwgK6lp+Jys1pwgUyk8/1eTh+5MXR50rd0Lg a0Iw== X-Gm-Message-State: AOAM531BxqRkT8crk5cmDa2QFjxuo0SL22IPIQYfTdfZ92dj35o/DG4A TpQQI3+Qps6Ub3vEWorUH64= X-Google-Smtp-Source: ABdhPJxW+1uPbr69HlT8zdk5DCZHvushfLGH5Z8cxaaxbXoq04RhdHu82fDyOZeUTBoieLU6jFhc1g== X-Received: by 2002:ad4:4b0b:: with SMTP id r11mr22115044qvw.94.1597811228761; Tue, 18 Aug 2020 21:27:08 -0700 (PDT) Received: from localhost.localdomain ([2001:470:b:9c3:9e5c:8eff:fe4f:f2d0]) by smtp.gmail.com with ESMTPSA id p17sm21999894qkj.69.2020.08.18.21.27.06 (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 18 Aug 2020 21:27:08 -0700 (PDT) Subject: [RFC PATCH v2 1/5] mm: Identify compound pages sooner in isolate_migratepages_block From: Alexander Duyck To: alex.shi@linux.alibaba.com Cc: yang.shi@linux.alibaba.com, lkp@intel.com, rong.a.chen@intel.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, hughd@google.com, linux-kernel@vger.kernel.org, alexander.duyck@gmail.com, daniel.m.jordan@oracle.com, linux-mm@kvack.org, shakeelb@google.com, willy@infradead.org, hannes@cmpxchg.org, tj@kernel.org, cgroups@vger.kernel.org, akpm@linux-foundation.org, richard.weiyang@gmail.com, mgorman@techsingularity.net, iamjoonsoo.kim@lge.com Date: Tue, 18 Aug 2020 21:27:05 -0700 Message-ID: <20200819042705.23414.84098.stgit@localhost.localdomain> In-Reply-To: <20200819041852.23414.95939.stgit@localhost.localdomain> References: <20200819041852.23414.95939.stgit@localhost.localdomain> User-Agent: StGit/0.17.1-dirty MIME-Version: 1.0 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: CEA3618016858 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam05 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: Alexander Duyck Since we are holding a reference to the page much sooner in isolate_migratepages_block we can move the PageCompound check out of the LRU locked section and instead just place it after get_page_unless_zero. By doing this we can allow any of the items that might trigger a failure to trigger a failure for the compound page rather than the order 0 page and as a result we should be able to process the pageblock faster. In addition by testing for PageCompound sooner we can avoid having the LRU flag cleared and then reset in the exception case. As a result this should prevent possible races where another thread might be attempting to pull the LRU pages from the list. Signed-off-by: Alexander Duyck --- mm/compaction.c | 33 ++++++++++++++++++--------------- 1 file changed, 18 insertions(+), 15 deletions(-) diff --git a/mm/compaction.c b/mm/compaction.c index d3f87f759773..88c7b950f676 100644 --- a/mm/compaction.c +++ b/mm/compaction.c @@ -984,6 +984,24 @@ static bool too_many_isolated(pg_data_t *pgdat) if (unlikely(!get_page_unless_zero(page))) goto isolate_fail; + /* + * Page is compound. We know the order before we know if it is + * on the LRU so we cannot assume it is THP. However since the + * page will have the LRU validated shortly we can use the value + * to skip over this page for now or validate the LRU is set and + * then isolate the entire compound page if we are isolating to + * generate a CMA page. + */ + if (PageCompound(page)) { + const unsigned int order = compound_order(page); + + if (likely(order < MAX_ORDER)) + low_pfn += (1UL << order) - 1; + + if (!cc->alloc_contig) + goto isolate_fail_put; + } + if (__isolate_lru_page_prepare(page, isolate_mode) != 0) goto isolate_fail_put; @@ -1009,23 +1027,8 @@ static bool too_many_isolated(pg_data_t *pgdat) if (test_and_set_skip(cc, page, low_pfn)) goto isolate_abort; } - - /* - * Page become compound since the non-locked check, - * and it's on LRU. It can only be a THP so the order - * is safe to read and it's 0 for tail pages. - */ - if (unlikely(PageCompound(page) && !cc->alloc_contig)) { - low_pfn += compound_nr(page) - 1; - SetPageLRU(page); - goto isolate_fail_put; - } } - /* The whole page is taken off the LRU; skip the tail pages. */ - if (PageCompound(page)) - low_pfn += compound_nr(page) - 1; - /* Successfully isolated */ del_page_from_lru_list(page, lruvec, page_lru(page)); mod_node_page_state(page_pgdat(page),