From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 89F53C3F2CD for ; Mon, 23 Mar 2020 03:50:51 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 27C6B2070A for ; Mon, 23 Mar 2020 03:50:51 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="PIg5w6jz" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 27C6B2070A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C76A96B0005; Sun, 22 Mar 2020 23:50:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id C4DBC6B0006; Sun, 22 Mar 2020 23:50:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B66238E0003; Sun, 22 Mar 2020 23:50:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0201.hostedemail.com [216.40.44.201]) by kanga.kvack.org (Postfix) with ESMTP id 9E16D6B0005 for ; Sun, 22 Mar 2020 23:50:50 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id BD4D5180AD811 for ; Mon, 23 Mar 2020 03:50:50 +0000 (UTC) X-FDA: 76625250660.22.water08_418a544dd422e X-HE-Tag: water08_418a544dd422e X-Filterd-Recvd-Size: 11853 Received: from mail-qk1-f195.google.com (mail-qk1-f195.google.com [209.85.222.195]) by imf03.hostedemail.com (Postfix) with ESMTP for ; Mon, 23 Mar 2020 03:50:50 +0000 (UTC) Received: by mail-qk1-f195.google.com with SMTP id j4so13890777qkc.11 for ; Sun, 22 Mar 2020 20:50:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=6INYt4gGhpcoFn705pKJj9o8HPNDr8E2DVtk1eeKAJQ=; b=PIg5w6jzjqlwC2zGBgV01xRGV1zVFSK8wau6VhPDQT6HSNq653JjVBe9KQ/1QBIGus n+mN86UBBgu5RbxQ7DEI+3z9l3W8auqXtQct8bXnOrPZkXjdpqOlWePJPH69v65KVgod EuUX99twNQpzFignvuSb6NMjRxULtYAQghdmXRZXXh+eL5DP0/F+qIjEXKuy+0eBgWzD d3tRWwEzCIa4OHQVBpXsZLv8TXdq2Gr5ayy0YOo6TGhas0sFpNwymcK7nXN9mDuJz897 lcyYFoUO7DYMzPqbWT4kAY92SEnOtQ4sLPnjwG3uhvGqJ4wbKjbubGh0owbQLR1GlxNH g3Mg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=6INYt4gGhpcoFn705pKJj9o8HPNDr8E2DVtk1eeKAJQ=; b=H46X6AJwFB8yv1m/f5eaYK/6vDe9hMB/D55yu8GkWfVAVlbSb4MNcSUYiUlms0u2/Z +yhX/1c6lZxnUFWDlasKE6vmI4Z78clnRVY39Hm5HVNi8o9dlP5mMm9mAynBe6zMp7Li AjVAJ5I86qQk8Lrm8jqc0m6cj8HdHunuAIbDKi6YILPUV1x1GdV5Yp5AMzMnQFDPtLme KJ7g/e/3vSlUare7+LnR3KSe6k50JSBJKWQSj7NlOu4FVGuJ0FbNhNHCjt0m033vwiNX 14uR4Rx7cccGyWVHl+6IqQL0+4OPwnUfSbLWi8LIfYqIcYdkjoDY38MTAjrrUaeTlfyO UIyw== X-Gm-Message-State: ANhLgQ39wri6ZJbvbnvpzMTXPNTsEvlTEiVS//Rwp8uPB3MK53vJWN7V 6uH43/Mlq8JubbY7Vx4lIFu6H3zYgjbfoaNq6pA= X-Google-Smtp-Source: ADFU+vuBpFkesQKVsaTAI45IdP+IwHPp5MHAXNfmfIg9nsLYKcA+XHq6wuHJbaoUG/9iKyEp/Wy6b0ZFkovdX9sqNiQ= X-Received: by 2002:a37:b4c1:: with SMTP id d184mr19779707qkf.452.1584935449102; Sun, 22 Mar 2020 20:50:49 -0700 (PDT) MIME-Version: 1.0 References: <1584693135-4396-1-git-send-email-iamjoonsoo.kim@lge.com> <1584693135-4396-2-git-send-email-iamjoonsoo.kim@lge.com> <20200320103000.GB3039@MiWiFi-R3L-srv> In-Reply-To: <20200320103000.GB3039@MiWiFi-R3L-srv> From: Joonsoo Kim Date: Mon, 23 Mar 2020 12:50:38 +0900 Message-ID: Subject: Re: [PATCH v3 1/2] mm/page_alloc: use ac->high_zoneidx for classzone_idx To: Baoquan He Cc: Andrew Morton , Linux Memory Management List , LKML , Johannes Weiner , Michal Hocko , Minchan Kim , Vlastimil Babka , Mel Gorman , kernel-team@lge.com, Ye Xiaolong , David Rientjes , Joonsoo Kim Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, Baoquan. 2020=EB=85=84 3=EC=9B=94 20=EC=9D=BC (=EA=B8=88) =EC=98=A4=ED=9B=84 7:30, B= aoquan He =EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84=B1: > > > On 03/20/20 at 05:32pm, js1304@gmail.com wrote: > > From: Joonsoo Kim > > > > Currently, we use the zone index of preferred_zone which represents > > the best matching zone for allocation, as classzone_idx. It has > > a problem on NUMA systems when the lowmem reserve protection exists > > for some zones on a node that do not exist on other nodes. > > > > In NUMA system, it can be possible that each node has different populat= ed > > zones. For example, node 0 could have DMA/DMA32/NORMAL/MOVABLE zone and > > node 1 could have only NORMAL zone. In this setup, allocation request > > initiated on node 0 and the one on node 1 would have different > > classzone_idx, 3 and 2, respectively, since their preferred_zones are > > different. If the allocation is local, there is no problem. However, > > if it is handled by the remote node due to memory shortage, the problem > > would happen. > > Hi Joonsoo, > > Not sure if adding one sentence into above paragraph would be make it > easier to understand. Assume you are only talking about the high_zoneidx > is MOVABLE_ZONE with calculation of gfp_zone(gfp_mask), since any other > case doesn't have this problem. Please correct me if I am wrong. You're right. This example is for the allocation request with gfp_zone(gfp_mask), MOVABLE_ZONE. > In NUMA system, it can be possible that each node has different populated > zones. For example, node 0 could have DMA/DMA32/NORMAL/MOVABLE zone and > node 1 could have only NORMAL zone. In this setup, if we get high_zoneidx > as 3 (namely MOVABLE zone), with gfp_zone(gfp_mask), allocation request > initiated on node 0 and the one on node 1 would have different > classzone_idx, 3 and 2, respectively, since their preferred_zones are > different. If the allocation is local, there is no problem. However, > if it is handled by the remote node due to memory shortage, the problem > would happen. I'm okay with your change but I try again to be better. Please check the following rewritten commit message and please let me know if it is better than before. Thanks. ------------------------>8------------------------------- >From 3842d99d54c02c25cc6f6077fd0e97acabc10917 Mon Sep 17 00:00:00 2001 From: Joonsoo Kim Date: Fri, 4 May 2018 09:55:21 +0900 Subject: [PATCH] mm/page_alloc: use ac->high_zoneidx for classzone_idx Currently, we use classzone_idx to calculate lowmem reserve proetection for an allocation request. This classzone_idx causes a problem on NUMA systems when the lowmem reserve protection exists for some zones on a node that do not exist on other nodes. Before further explanation, I should first clarify how to compute the classzone_idx and the high_zoneidx. - ac->high_zoneidx is computed via the arcane gfp_zone(gfp_mask) and represents the index of the highest zone the allocation can use - classzone_idx was supposed to be the index of the highest zone on the local node that the allocation can use, that is actually available in the system Think about following example. Node 0 has 4 populated zone, DMA/DMA32/NORMAL/MOVABLE. Node 1 has 1 populated zone, NORMAL. Some zones, such as MOVABLE, doesn't exist on node 1 and this makes following difference. Assume that there is an allocation request whose gfp_zone(gfp_mask) is the zone, MOVABLE. Then, it's high_zoneidx is 3. If this allocation is initiated on node 0, it's classzone_idx is 3 since actually available/usable zone on local (node 0) is MOVABLE. If this allocation is initiated on node 1, it's classzone_idx is 2 since actually available/usable zone on local (node 1) is NORMAL. You can see that classzone_idx of the allocation request are different according to their starting node, even if their high_zoneidx is the same. Think more about these two allocation requests. If they are processed on local, there is no problem. However, if allocation is initiated on node 1 are processed on remote, in this example, at the NORMAL zone on node 0, due to memory shortage, problem occurs. Their different classzone_idx leads to different lowmem reserve and then different min watermark. See the following example. root@ubuntu:/sys/devices/system/memory# cat /proc/zoneinfo Node 0, zone DMA per-node stats ... pages free 3965 min 5 low 8 high 11 spanned 4095 present 3998 managed 3977 protection: (0, 2961, 4928, 5440) ... Node 0, zone DMA32 pages free 757955 min 1129 low 1887 high 2645 spanned 1044480 present 782303 managed 758116 protection: (0, 0, 1967, 2479) ... Node 0, zone Normal pages free 459806 min 750 low 1253 high 1756 spanned 524288 present 524288 managed 503620 protection: (0, 0, 0, 4096) ... Node 0, zone Movable pages free 130759 min 195 low 326 high 457 spanned 1966079 present 131072 managed 131072 protection: (0, 0, 0, 0) ... Node 1, zone DMA pages free 0 min 0 low 0 high 0 spanned 0 present 0 managed 0 protection: (0, 0, 1006, 1006) Node 1, zone DMA32 pages free 0 min 0 low 0 high 0 spanned 0 present 0 managed 0 protection: (0, 0, 1006, 1006) Node 1, zone Normal per-node stats ... pages free 233277 min 383 low 640 high 897 spanned 262144 present 262144 managed 257744 protection: (0, 0, 0, 0) ... Node 1, zone Movable pages free 0 min 0 low 0 high 0 spanned 262144 present 0 managed 0 protection: (0, 0, 0, 0) - static min watermark for the NORMAL zone on node 0 is 750. - lowmem reserve for the request with classzone idx 3 at the NORMAL on node 0 is 4096. - lowmem reserve for the request with classzone idx 2 at the NORMAL on node 0 is 0. So, overall min watermark is: allocation initiated on node 0 (classzone_idx 3): 750 + 4096 =3D 4846 allocation initiated on node 1 (classzone_idx 2): 750 + 0 =3D 750 allocation initiated on node 1 will have some precedence than allocation initiated on node 0 because min watermark of the former allocation is lower than the other. So, allocation initiated on node 1 could succeed on node 0 when allocation initiated on node 0 could not, and, this could cause too many numa_miss allocation. Then, performance could be downgraded. Recently, there was a regression report about this problem on CMA patches since CMA memory are placed in ZONE_MOVABLE by those patches. I checked that problem is disappeared with this fix that uses high_zoneidx for classzone_idx. http://lkml.kernel.org/r/20180102063528.GG30397@yexl-desktop Using high_zoneidx for classzone_idx is more consistent way than previous approach because system's memory layout doesn't affect anything to it. With this patch, both classzone_idx on above example will be 3 so will have the same min watermark. allocation initiated on node 0: 750 + 4096 =3D 4846 allocation initiated on node 1: 750 + 4096 =3D 4846 One could wonder if there is a side effect that allocation initiated on node 1 will use higher bar when allocation is handled on local since classzone_idx could be higher than before. It will not happen because the zone without managed page doesn't contributes lowmem_reserve at all. Reported-by: Ye Xiaolong Tested-by: Ye Xiaolong Acked-by: Vlastimil Babka Signed-off-by: Joonsoo Kim --- mm/internal.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/mm/internal.h b/mm/internal.h index c39c895..aebaa33 100644 --- a/mm/internal.h +++ b/mm/internal.h @@ -119,7 +119,7 @@ struct alloc_context { bool spread_dirty_pages; }; -#define ac_classzone_idx(ac) zonelist_zone_idx(ac->preferred_zoneref) +#define ac_classzone_idx(ac) (ac->high_zoneidx) /* * Locate the struct page for both the matching buddy in our --=20 2.7.4