From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-15.1 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,NICE_REPLY_A,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED, USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id E3688C433E0 for ; Tue, 16 Mar 2021 01:57:36 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 23B176500A for ; Tue, 16 Mar 2021 01:57:36 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 23B176500A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=tom.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 7D4716B006C; Mon, 15 Mar 2021 21:57:35 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 784D66B006E; Mon, 15 Mar 2021 21:57:35 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 5D83F6B0070; Mon, 15 Mar 2021 21:57:35 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id 366376B006C for ; Mon, 15 Mar 2021 21:57:35 -0400 (EDT) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id DD36A180397CC for ; Tue, 16 Mar 2021 01:57:34 +0000 (UTC) X-FDA: 77924075628.01.62DA9AA Received: from smtp.tom.com (smtprz15.163.net [106.3.154.248]) by imf27.hostedemail.com (Postfix) with SMTP id 0348D8019144 for ; Tue, 16 Mar 2021 01:57:32 +0000 (UTC) Received: from my-app02.tom.com (my-app02.tom.com [127.0.0.1]) by freemail02.tom.com (Postfix) with ESMTP id A8430B00D6A for ; Tue, 16 Mar 2021 09:58:38 +0800 (CST) Received: from my-app02.tom.com (HELO smtp.tom.com) ([127.0.0.1]) by my-app02 (TOM SMTP Server) with SMTP ID -598176206 for ; Tue, 16 Mar 2021 09:58:38 +0800 (CST) Received: from antispam2.tom.com (unknown [172.25.16.56]) by freemail02.tom.com (Postfix) with ESMTP id 96C69B00D55 for ; Tue, 16 Mar 2021 09:58:33 +0800 (CST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=tom.com; s=201807; t=1615859918; bh=Yj0AE6EFJl1fH1sa6IQH9sHmMadSgtaYZm1ivQeBMw4=; h=Subject:To:Cc:References:From:Date:In-Reply-To:From; b=sIm8rxWnE7nRWnFFFaYIq+2+dS9DqV0M2W203WHxtsz/kukyIayVNuCoNuMVyFp8K 6kX9P62weRnI5MKnUNcUCUxzGzAkVjopTH9Ls2YbGfzv69mRRN+tNTqWgDf5ooKwv2 AQsBUIHkeWY0hl6n9nrXa/rgZiu1W7l83pkv6xrI= Received: from antispam2.tom.com (antispam2.tom.com [127.0.0.1]) by antispam2.tom.com (Postfix) with ESMTP id D6A3981A8D for ; Tue, 16 Mar 2021 09:55:08 +0800 (CST) X-Virus-Scanned: Debian amavisd-new at antispam2.tom.com Received: from antispam2.tom.com ([127.0.0.1]) by antispam2.tom.com (antispam2.tom.com [127.0.0.1]) (amavisd-new, port 10024) with ESMTP id zD8uhqEqznVu for ; Tue, 16 Mar 2021 09:55:04 +0800 (CST) Received: from [127.0.0.1] (unknown [58.34.42.107]) by antispam2.tom.com (Postfix) with ESMTPA id B1267815E3; Tue, 16 Mar 2021 09:55:03 +0800 (CST) Subject: Re: [PATCH] kswapd: no need reclaim cma pages triggered by unmovable allocation To: David Hildenbrand , zhou , linux-mm@kvack.org Cc: linux-kernel@vger.kernel.org, akpm@linux-foundation.org, mhocko@kernel.org, mgorman@suse.de, willy@linux.intel.com, rostedt@goodmis.org, mingo@redhat.com, vbabka@suse.cz, rientjes@google.com, pankaj.gupta.linux@gmail.com, bhe@redhat.com, ying.huang@intel.com, iamjoonsoo.kim@lge.com, minchan@kernel.org, ruxian.feng@transsion.com, kai.cheng@transsion.com, zhao.xu@transsion.com, zhou xianrong References: <20210313083109.5410-1-xianrong_zhou@163.com> <64f8c03f-7fd9-2e03-6b90-67e2a5a45b9d@redhat.com> From: zhou xianrong Message-ID: <944637a9-f767-70fb-1f4e-7ae3fd011fd3@tom.com> Date: Tue, 16 Mar 2021 09:57:15 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <64f8c03f-7fd9-2e03-6b90-67e2a5a45b9d@redhat.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: quoted-printable X-Stat-Signature: giouw6kkbt6k8ew63j9mefrq5sb1g3dd X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 0348D8019144 Received-SPF: none (tom.com>: No applicable sender policy available) receiver=imf27; identity=mailfrom; envelope-from=""; helo=smtp.tom.com; client-ip=106.3.154.248 X-HE-DKIM-Result: fail/fail (body has been altered) X-HE-Tag: 1615859852-963186 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: =E5=9C=A8 2021/3/15 23:46, David Hildenbrand =E5=86=99=E9=81=93: > On 13.03.21 09:31, zhou wrote: >> From: zhou xianrong >> >> For purpose of better migration cma pages are allocated after >> failure movalbe allocations and are used normally for file pages >> or anonymous pages. > > I failed to parse that senctence. > > "For better migration, CMA pages are allocated after failing=20 > allocation of movable allocations and are used for backing files or=20 > anonymous memory." > > Still doesn't make any sense to me. Can you clarify? > I mean alloc_page() allocates cma pages using fallback style so that=20 alloc_contig_range() can allocate it easily. > >> >> In reclaim path many cma pages if configurated are reclaimed > > s/configurated/configured/ > >> from lru lists in kswapd mainly or direct reclaim triggered by >> unmovable or reclaimable allocations. But these reclaimed cma >> pages can not be used by original unmovable or reclaimable >> allocations. So the reclaim are unnecessary. > > Might be a dump question, but why can't reclaimable allocations end up=20 > on CMA? (for unmovable allocations, this is clear) Or did I=20 > misunderstand what that paragraph was trying to tell me? > I remember only movable migratetype can fallback into cma. >> >> So the unmovable or reclaimable allocations should not trigger >> reclaiming cma pages. The patch adds third factor of migratetype >> which is just like factors of zone index or order kswapd need >> consider. The modification follows codes of zone index >> consideration. And it is straightforward that skips reclaiming >> cma pages in reclaim procedure which is triggered only by >> unmovable or reclaimable allocations. >> >> This optimization can avoid ~3% unnecessary isolations from cma >> (cma isolated / total isolated) with configuration of total 100Mb >> cma pages. > > Can you say a few words about interaction with ZONE_MOVABLE, which=20 > behaves similar to CMA? I.e., does the same apply to ZONE_MOVABLE? Is=20 > it already handled? > >> >> Signed-off-by: zhou xianrong >> Signed-off-by: feng ruxian >> --- >> =C2=A0 include/linux/mmzone.h=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= |=C2=A0 6 ++-- >> =C2=A0 include/trace/events/vmscan.h | 20 +++++++---- >> =C2=A0 mm/page_alloc.c=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 |=C2=A0 5 +-- >> =C2=A0 mm/vmscan.c=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 | 63 +++++++++++++= ++++++++++++++++------ >> =C2=A0 4 files changed, 73 insertions(+), 21 deletions(-) >> >> diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h >> index b593316bff3d..7dd38d7372b9 100644 >> --- a/include/linux/mmzone.h >> +++ b/include/linux/mmzone.h >> @@ -301,6 +301,8 @@ struct lruvec { >> =C2=A0 #define ISOLATE_ASYNC_MIGRATE=C2=A0=C2=A0=C2=A0 ((__force isola= te_mode_t)0x4) >> =C2=A0 /* Isolate unevictable pages */ >> =C2=A0 #define ISOLATE_UNEVICTABLE=C2=A0=C2=A0=C2=A0 ((__force isolate= _mode_t)0x8) >> +/* Isolate none cma pages */ >> +#define ISOLATE_NONCMA=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ((__= force isolate_mode_t)0x10) >> =C2=A0 =C2=A0 /* LRU Isolation modes. */ >> =C2=A0 typedef unsigned __bitwise isolate_mode_t; >> @@ -756,7 +758,7 @@ typedef struct pglist_data { >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 wait_queue_head_t pfmemalloc_wait; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct task_struct *kswapd;=C2=A0=C2=A0= =C2=A0 /* Protected by >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 = mem_hotplug_begin/end() */ >> -=C2=A0=C2=A0=C2=A0 int kswapd_order; >> +=C2=A0=C2=A0=C2=A0 int kswapd_order, kswapd_migratetype; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 enum zone_type kswapd_highest_zoneidx; >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 int kswapd_failures;=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* Number of 'reclaimed =3D=3D 0'=20 >> runs */ >> @@ -840,7 +842,7 @@ static inline bool pgdat_is_empty(pg_data_t *pgdat= ) >> =C2=A0 =C2=A0 void build_all_zonelists(pg_data_t *pgdat); >> =C2=A0 void wakeup_kswapd(struct zone *zone, gfp_t gfp_mask, int order= , >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 enum zon= e_type highest_zoneidx); >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 int migr= atetype, enum zone_type highest_zoneidx); >> =C2=A0 bool __zone_watermark_ok(struct zone *z, unsigned int order,=20 >> unsigned long mark, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 int highest_zoneidx, unsigned int alloc_flags, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 long free_pages); >> diff --git a/include/trace/events/vmscan.h=20 >> b/include/trace/events/vmscan.h >> index 2070df64958e..41bbafdfde84 100644 >> --- a/include/trace/events/vmscan.h >> +++ b/include/trace/events/vmscan.h >> @@ -51,37 +51,41 @@ TRACE_EVENT(mm_vmscan_kswapd_sleep, >> =C2=A0 =C2=A0 TRACE_EVENT(mm_vmscan_kswapd_wake, >> =C2=A0 -=C2=A0=C2=A0=C2=A0 TP_PROTO(int nid, int zid, int order), >> +=C2=A0=C2=A0=C2=A0 TP_PROTO(int nid, int zid, int order, int mt), >> =C2=A0 -=C2=A0=C2=A0=C2=A0 TP_ARGS(nid, zid, order), >> +=C2=A0=C2=A0=C2=A0 TP_ARGS(nid, zid, order, mt), >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 TP_STRUCT__entry( >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __field(=C2=A0=C2= =A0=C2=A0 int,=C2=A0=C2=A0=C2=A0 nid=C2=A0=C2=A0=C2=A0 ) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __field(=C2=A0=C2= =A0=C2=A0 int,=C2=A0=C2=A0=C2=A0 zid=C2=A0=C2=A0=C2=A0 ) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __field(=C2=A0=C2= =A0=C2=A0 int,=C2=A0=C2=A0=C2=A0 order=C2=A0=C2=A0=C2=A0 ) >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __field(=C2=A0=C2=A0=C2=A0= int,=C2=A0=C2=A0=C2=A0 mt=C2=A0=C2=A0=C2=A0 ) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ), >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 TP_fast_assign( >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->nid=C2= =A0=C2=A0=C2=A0 =3D nid; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->zid=C2= =A0=C2=A0=C2=A0 =3D zid; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->order=C2= =A0=C2=A0=C2=A0 =3D order; >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->mt=C2=A0=C2=A0=C2= =A0 =3D mt; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ), >> =C2=A0 -=C2=A0=C2=A0=C2=A0 TP_printk("nid=3D%d order=3D%d", >> +=C2=A0=C2=A0=C2=A0 TP_printk("nid=3D%d order=3D%d migratetype=3D%d", >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->nid, >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->order) >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->order, >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->mt) >> =C2=A0 ); >> =C2=A0 =C2=A0 TRACE_EVENT(mm_vmscan_wakeup_kswapd, >> =C2=A0 -=C2=A0=C2=A0=C2=A0 TP_PROTO(int nid, int zid, int order, gfp_t= gfp_flags), >> +=C2=A0=C2=A0=C2=A0 TP_PROTO(int nid, int zid, int order, int mt, gfp_= t gfp_flags), >> =C2=A0 -=C2=A0=C2=A0=C2=A0 TP_ARGS(nid, zid, order, gfp_flags), >> +=C2=A0=C2=A0=C2=A0 TP_ARGS(nid, zid, order, mt, gfp_flags), >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 TP_STRUCT__entry( >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __field(=C2=A0=C2= =A0=C2=A0 int,=C2=A0=C2=A0=C2=A0 nid=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 ) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __field(=C2=A0=C2= =A0=C2=A0 int,=C2=A0=C2=A0=C2=A0 zid=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 ) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __field(=C2=A0=C2= =A0=C2=A0 int,=C2=A0=C2=A0=C2=A0 order=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 ) >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __field(=C2=A0=C2=A0=C2=A0= int,=C2=A0=C2=A0=C2=A0 mt=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __field(=C2=A0=C2= =A0=C2=A0 gfp_t,=C2=A0=C2=A0=C2=A0 gfp_flags=C2=A0=C2=A0=C2=A0 ) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ), >> =C2=A0 @@ -89,12 +93,14 @@ TRACE_EVENT(mm_vmscan_wakeup_kswapd, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->nid=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D nid; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->zid=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D zid; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->order=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D order; >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->mt=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 =3D mt; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->gfp_fl= ags=C2=A0=C2=A0=C2=A0 =3D gfp_flags; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ), >> =C2=A0 -=C2=A0=C2=A0=C2=A0 TP_printk("nid=3D%d order=3D%d gfp_flags=3D= %s", >> +=C2=A0=C2=A0=C2=A0 TP_printk("nid=3D%d order=3D%d migratetype=3D%d gf= p_flags=3D%s", >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->nid, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->order, >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 __entry->mt, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 show_gfp_flags(= __entry->gfp_flags)) >> =C2=A0 ); >> =C2=A0 diff --git a/mm/page_alloc.c b/mm/page_alloc.c >> index 519a60d5b6f7..45ceb15721b8 100644 >> --- a/mm/page_alloc.c >> +++ b/mm/page_alloc.c >> @@ -3517,7 +3517,7 @@ struct page *rmqueue(struct zone *preferred_zone= , >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* Separate test+clear to avoid unneces= sary atomics */ >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (test_bit(ZONE_BOOSTED_WATERMARK, &z= one->flags)) { >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 clear_bit(ZONE_= BOOSTED_WATERMARK, &zone->flags); >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 wakeup_kswapd(zone, 0, 0, = zone_idx(zone)); >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 wakeup_kswapd(zone, 0, 0, = migratetype, zone_idx(zone)); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 VM_BUG_ON_PAGE(page && bad_range= (zone, page), page); >> @@ -4426,11 +4426,12 @@ static void wake_all_kswapds(unsigned int=20 >> order, gfp_t gfp_mask, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct zone *zone; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pg_data_t *last_pgdat =3D NULL; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 enum zone_type highest_zoneidx =3D ac->= highest_zoneidx; >> +=C2=A0=C2=A0=C2=A0 int migratetype =3D ac->migratetype; >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for_each_zone_zonelist_nodemask(= zone, z, ac->zonelist,=20 >> highest_zoneidx, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ac->nodemask) { >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (last_pgdat = !=3D zone->zone_pgdat) >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 wa= keup_kswapd(zone, gfp_mask, order, highest_zoneidx); >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 wa= keup_kswapd(zone, gfp_mask, order, migratetype,=20 >> highest_zoneidx); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 last_pgdat =3D = zone->zone_pgdat; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } >> =C2=A0 } >> diff --git a/mm/vmscan.c b/mm/vmscan.c >> index b1b574ad199d..184f0c4c7151 100644 >> --- a/mm/vmscan.c >> +++ b/mm/vmscan.c >> @@ -99,6 +99,9 @@ struct scan_control { >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* Can pages be swapped as part of recl= aim? */ >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 unsigned int may_swap:1; >> =C2=A0 +=C2=A0=C2=A0=C2=A0 /* Can cma pages be reclaimed? */ >> +=C2=A0=C2=A0=C2=A0 unsigned int may_cma:1; >> + >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Cgroups are not reclaimed below= their configured memory.low, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * unless we threaten to OOM. If a= ny cgroups are skipped due to >> @@ -286,6 +289,11 @@ static bool writeback_throttling_sane(struct=20 >> scan_control *sc) >> =C2=A0 } >> =C2=A0 #endif >> =C2=A0 +static bool movable_reclaim(gfp_t gfp_mask) >> +{ >> +=C2=A0=C2=A0=C2=A0 return is_migrate_movable(gfp_migratetype(gfp_mask= )); >> +} >> + >> =C2=A0 /* >> =C2=A0=C2=A0 * This misses isolated pages which are not accounted for = to save=20 >> counters. >> =C2=A0=C2=A0 * As the data only determines if reclaim or compaction co= ntinues,=20 >> it is >> @@ -1499,6 +1507,7 @@ unsigned int=20 >> reclaim_clean_pages_from_list(struct zone *zone, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .gfp_mask =3D G= FP_KERNEL, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .priority =3D D= EF_PRIORITY, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_unmap =3D = 1, >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_cma =3D 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct reclaim_stat stat; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 unsigned int nr_reclaimed; >> @@ -1593,6 +1602,9 @@ int __isolate_lru_page_prepare(struct page=20 >> *page, isolate_mode_t mode) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if ((mode & ISOLATE_UNMAPPED) && page_m= apped(page)) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return ret; >> =C2=A0 +=C2=A0=C2=A0=C2=A0 if ((mode & ISOLATE_NONCMA) &&=20 >> is_migrate_cma(get_pageblock_migratetype(page))) >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return ret; >> + >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return 0; >> =C2=A0 } >> =C2=A0 @@ -1647,7 +1659,10 @@ static unsigned long=20 >> isolate_lru_pages(unsigned long nr_to_scan, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 unsigned long skipped =3D 0; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 unsigned long scan, total_scan, nr_page= s; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 LIST_HEAD(pages_skipped); >> -=C2=A0=C2=A0=C2=A0 isolate_mode_t mode =3D (sc->may_unmap ? 0 : ISOLA= TE_UNMAPPED); >> +=C2=A0=C2=A0=C2=A0 isolate_mode_t mode; >> + >> +=C2=A0=C2=A0=C2=A0 mode =3D (sc->may_unmap ? 0 : ISOLATE_UNMAPPED); >> +=C2=A0=C2=A0=C2=A0 mode |=3D (sc->may_cma ? 0 : ISOLATE_NONCMA); >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 total_scan =3D 0; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 scan =3D 0; >> @@ -2125,6 +2140,7 @@ unsigned long reclaim_pages(struct list_head=20 >> *page_list) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_writepage = =3D 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_unmap =3D = 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_swap =3D 1= , >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_cma =3D 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }; >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 while (!list_empty(page_list)) { >> @@ -3253,6 +3269,7 @@ unsigned long try_to_free_pages(struct zonelist=20 >> *zonelist, int order, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_writepage = =3D !laptop_mode, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_unmap =3D = 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_swap =3D 1= , >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_cma =3D movable_recla= im(gfp_mask), >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }; >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* >> @@ -3298,6 +3315,7 @@ unsigned long mem_cgroup_shrink_node(struct=20 >> mem_cgroup *memcg, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_unmap =3D = 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .reclaim_idx =3D= MAX_NR_ZONES - 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_swap =3D != noswap, >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_cma =3D 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }; >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 WARN_ON_ONCE(!current->reclaim_s= tate); >> @@ -3341,6 +3359,7 @@ unsigned long=20 >> try_to_free_mem_cgroup_pages(struct mem_cgroup *memcg, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_writepage = =3D !laptop_mode, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_unmap =3D = 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_swap =3D m= ay_swap, >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_cma =3D 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Traverse the ZONELIST_FALLBACK = zonelist of the current node=20 >> to put >> @@ -3548,7 +3567,7 @@ static bool kswapd_shrink_node(pg_data_t *pgdat, >> =C2=A0=C2=A0 * or lower is eligible for reclaim until at least one usa= ble zone is >> =C2=A0=C2=A0 * balanced. >> =C2=A0=C2=A0 */ >> -static int balance_pgdat(pg_data_t *pgdat, int order, int=20 >> highest_zoneidx) >> +static int balance_pgdat(pg_data_t *pgdat, int order, int=20 >> migratetype, int highest_zoneidx) >> =C2=A0 { >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 int i; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 unsigned long nr_soft_reclaimed; >> @@ -3650,6 +3669,7 @@ static int balance_pgdat(pg_data_t *pgdat, int=20 >> order, int highest_zoneidx) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 */ >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sc.may_writepag= e =3D !laptop_mode && !nr_boost_reclaim; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sc.may_swap =3D= !nr_boost_reclaim; >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 sc.may_cma =3D is_migrate_= movable(migratetype); >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * Do some= background aging of the anon list, to give >> @@ -3771,8 +3791,15 @@ static enum zone_type=20 >> kswapd_highest_zoneidx(pg_data_t *pgdat, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return curr_idx =3D=3D MAX_NR_ZONES ? p= rev_highest_zoneidx : curr_idx; >> =C2=A0 } >> =C2=A0 +static int kswapd_migratetype(pg_data_t *pgdat, int prev_migra= tetype) >> +{ >> +=C2=A0=C2=A0=C2=A0 int curr_migratetype =3D READ_ONCE(pgdat->kswapd_m= igratetype); >> + >> +=C2=A0=C2=A0=C2=A0 return curr_migratetype =3D=3D MIGRATE_TYPES ? pre= v_migratetype :=20 >> curr_migratetype; >> +} >> + >> =C2=A0 static void kswapd_try_to_sleep(pg_data_t *pgdat, int alloc_ord= er,=20 >> int reclaim_order, >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 unsigned int highest_zoneidx) >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 int migratetype, unsigned int highest_zoneidx) >> =C2=A0 { >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 long remaining =3D 0; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 DEFINE_WAIT(wait); >> @@ -3807,8 +3834,8 @@ static void kswapd_try_to_sleep(pg_data_t=20 >> *pgdat, int alloc_order, int reclaim_o >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 remaining =3D s= chedule_timeout(HZ/10); >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * If woken premature= ly then reset kswapd_highest_zoneidx and >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * order. The values = will either be from a wakeup request or >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * If woken premature= ly then reset kswapd_highest_zoneidx,=20 >> order >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * and migratetype. T= he values will either be from a wakeup=20 >> request or >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * the pre= vious request that slept prematurely. >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 */ >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (remaining) = { >> @@ -3818,6 +3845,10 @@ static void kswapd_try_to_sleep(pg_data_t=20 >> *pgdat, int alloc_order, int reclaim_o >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0 if (READ_ONCE(pgdat->kswapd_order) < reclaim_order) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgdat->kswapd_order, reclaim_or= der); >> + >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if= =20 >> (!is_migrate_movable(READ_ONCE(pgdat->kswapd_migratetype))) >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgdat->kswapd_migratetype, >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ksw= apd_migratetype(pgdat, migratetype)); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 finish_w= ait(&pgdat->kswapd_wait, &wait); >> @@ -3870,6 +3901,7 @@ static void kswapd_try_to_sleep(pg_data_t=20 >> *pgdat, int alloc_order, int reclaim_o >> =C2=A0=C2=A0 */ >> =C2=A0 static int kswapd(void *p) >> =C2=A0 { >> +=C2=A0=C2=A0=C2=A0 int migratetype =3D 0; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 unsigned int alloc_order, reclaim_order= ; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 unsigned int highest_zoneidx =3D MAX_NR= _ZONES - 1; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pg_data_t *pgdat =3D (pg_data_t*)p; >> @@ -3895,23 +3927,27 @@ static int kswapd(void *p) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 set_freezable(); >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgdat->kswapd_order, = 0); >> +=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgdat->kswapd_migratetype, MIGRATE_TYPE= S); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgdat->kswapd_highest_zoneid= x, MAX_NR_ZONES); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 for ( ; ; ) { >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 bool ret; >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 alloc_or= der =3D reclaim_order =3D=20 >> READ_ONCE(pgdat->kswapd_order); >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 migratetype =3D kswapd_mig= ratetype(pgdat, migratetype); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 highest_zoneidx= =3D kswapd_highest_zoneidx(pgdat, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 highest_zoneidx); >> =C2=A0 =C2=A0 kswapd_try_sleep: >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 kswapd_try_to_s= leep(pgdat, alloc_order, reclaim_order, >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 highest_zoneidx); >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 migratetype, highest_zoneid= x); >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 /* Read = the new order and highest_zoneidx */ >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 alloc_order =3D= READ_ONCE(pgdat->kswapd_order); >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 migratetype =3D kswapd_mig= ratetype(pgdat, migratetype); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 highest_zoneidx= =3D kswapd_highest_zoneidx(pgdat, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0 highest_zoneidx); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgda= t->kswapd_order, 0); >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgdat->kswapd_m= igratetype, MIGRATE_TYPES); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgda= t->kswapd_highest_zoneidx, MAX_NR_ZONES); >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 ret =3D = try_to_freeze(); >> @@ -3934,8 +3970,8 @@ static int kswapd(void *p) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 * request= (alloc_order). >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 */ >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 trace_mm_vmscan= _kswapd_wake(pgdat->node_id, highest_zoneidx, >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 all= oc_order); >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 reclaim_order =3D balance_= pgdat(pgdat, alloc_order, >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 all= oc_order, migratetype); >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 reclaim_order =3D balance_= pgdat(pgdat, alloc_order, migratetype, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0 highest_zoneidx); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (reclaim_ord= er < alloc_order) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0 goto kswapd_try_sleep; >> @@ -3953,11 +3989,12 @@ static int kswapd(void *p) >> =C2=A0=C2=A0 * has failed or is not needed, still wake up kcompactd if= only=20 >> compaction is >> =C2=A0=C2=A0 * needed. >> =C2=A0=C2=A0 */ >> -void wakeup_kswapd(struct zone *zone, gfp_t gfp_flags, int order, >> +void wakeup_kswapd(struct zone *zone, gfp_t gfp_flags, int order,=20 >> int migratetype >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0= enum zone_type highest_zoneidx) >> =C2=A0 { >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pg_data_t *pgdat; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 enum zone_type curr_idx; >> +=C2=A0=C2=A0=C2=A0 int curr_migratetype; >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (!managed_zone(zone)) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return; >> @@ -3967,6 +4004,7 @@ void wakeup_kswapd(struct zone *zone, gfp_t=20 >> gfp_flags, int order, >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 pgdat =3D zone->zone_pgdat; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 curr_idx =3D READ_ONCE(pgdat->kswapd_hi= ghest_zoneidx); >> +=C2=A0=C2=A0=C2=A0 curr_migratetype =3D READ_ONCE(pgdat->kswapd_migra= tetype); >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (curr_idx =3D=3D MAX_NR_ZONES= || curr_idx < highest_zoneidx) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgda= t->kswapd_highest_zoneidx, highest_zoneidx); >> @@ -3974,6 +4012,9 @@ void wakeup_kswapd(struct zone *zone, gfp_t=20 >> gfp_flags, int order, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (READ_ONCE(pgdat->kswapd_order) < or= der) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgda= t->kswapd_order, order); >> =C2=A0 +=C2=A0=C2=A0=C2=A0 if (curr_migratetype =3D=3D MIGRATE_TYPES |= |=20 >> is_migrate_movable(migratetype)) >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 WRITE_ONCE(pgdat->kswapd_m= igratetype, migratetype); >> + >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 if (!waitqueue_active(&pgdat->kswapd_wa= it)) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 return; >> =C2=A0 @@ -3994,7 +4035,7 @@ void wakeup_kswapd(struct zone *zone, gfp= _t=20 >> gfp_flags, int order, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 } >> =C2=A0 =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 trace_mm_vmscan_wakeup_kswapd(pg= dat->node_id,=20 >> highest_zoneidx, order, >> -=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 gfp_flags); >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 migratetype, gf= p_flags); >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 wake_up_interruptible(&pgdat->kswapd_wa= it); >> =C2=A0 } >> =C2=A0 @@ -4017,6 +4058,7 @@ unsigned long shrink_all_memory(unsigned = long=20 >> nr_to_reclaim) >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_writepage = =3D 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_unmap =3D = 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_swap =3D 1= , >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_cma =3D 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .hibernation_mo= de =3D 1, >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }; >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 struct zonelist *zonelist =3D node_zone= list(numa_node_id(),=20 >> sc.gfp_mask); >> @@ -4176,6 +4218,7 @@ static int __node_reclaim(struct pglist_data=20 >> *pgdat, gfp_t gfp_mask, unsigned in >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_writepage = =3D !!(node_reclaim_mode & RECLAIM_WRITE), >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_unmap =3D = !!(node_reclaim_mode & RECLAIM_UNMAP), >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_swap =3D 1= , >> +=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .may_cma =3D movable_recla= im(gfp_mask), >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 .reclaim_idx =3D= gfp_zone(gfp_mask), >> =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 }; >> > >