From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f69.google.com (mail-wm0-f69.google.com [74.125.82.69]) by kanga.kvack.org (Postfix) with ESMTP id CEAB96B0253 for ; Mon, 18 Sep 2017 03:10:13 -0400 (EDT) Received: by mail-wm0-f69.google.com with SMTP id r136so8568667wmf.4 for ; Mon, 18 Sep 2017 00:10:13 -0700 (PDT) Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id c10sor2355904wrc.60.2017.09.18.00.10.12 for (Google Transport Security); Mon, 18 Sep 2017 00:10:12 -0700 (PDT) From: Michal Hocko Subject: [PATCH v2 0/2] mm, memory_hotplug: redefine memory offline retry logic Date: Mon, 18 Sep 2017 09:08:32 +0200 Message-Id: <20170918070834.13083-1-mhocko@kernel.org> Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML , Michal Hocko , Vlastimil Babka Hi, this has been previously sent http://lkml.kernel.org/r/20170904082148.23131-1-mhocko@kernel.org No fundamental objections have been raised. There were some questions about potential permanent migration failures but those are deemed unlikely and not really problematic because the context is interruptible. I have tried to clarify the wording to be more clear. original changelog: While testing memory hotplug on a large 4TB machine we have noticed that memory offlining is just too eager to fail. The primary reason is that the retry logic is just too easy to give up. We have 4 ways out of the offline - we have a permanent failure (isolation or memory notifiers fail, or hugetlb pages cannot be dropped) - userspace sends a signal - a hardcoded 120s timeout expires - page migration fails 5 times This is way too convoluted and it doesn't scale very well. We have seen both temporary migration failures as well as 120s being triggered. After removing those restrictions we were able to pass stress testing during memory hot remove without any other negative side effects observed. Therefore I suggest dropping both hard coded policies. I couldn't have found any specific reason for them in the changelog. I neither didn't get any response [1] from Kamezawa. If we need some upper bound - e.g. timeout based - then we should have a proper and user defined policy for that. In any case there should be a clear use case when introducing it. Any comments, objections? Shortlog Michal Hocko (2): mm, memory_hotplug: do not fail offlining too early mm, memory_hotplug: remove timeout from __offline_memory Diffstat mm/memory_hotplug.c | 48 ++++++++++++------------------------------------ 1 file changed, 12 insertions(+), 36 deletions(-) [1] http://lkml.kernel.org/r/20170828094316.GF17097@dhcp22.suse.cz -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f200.google.com (mail-wr0-f200.google.com [209.85.128.200]) by kanga.kvack.org (Postfix) with ESMTP id 60B166B025E for ; Mon, 18 Sep 2017 03:10:15 -0400 (EDT) Received: by mail-wr0-f200.google.com with SMTP id v109so8393345wrc.5 for ; Mon, 18 Sep 2017 00:10:15 -0700 (PDT) Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id d83sor1571392wmc.11.2017.09.18.00.10.14 for (Google Transport Security); Mon, 18 Sep 2017 00:10:14 -0700 (PDT) From: Michal Hocko Subject: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early Date: Mon, 18 Sep 2017 09:08:33 +0200 Message-Id: <20170918070834.13083-2-mhocko@kernel.org> In-Reply-To: <20170918070834.13083-1-mhocko@kernel.org> References: <20170918070834.13083-1-mhocko@kernel.org> Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML , Michal Hocko , Vlastimil Babka From: Michal Hocko Memory offlining can fail just too eagerly under a heavy memory pressure. [ 5410.336792] page:ffffea22a646bd00 count:255 mapcount:252 mapping:ffff88ff926c9f38 index:0x3 [ 5410.336809] flags: 0x9855fe40010048(uptodate|active|mappedtodisk) [ 5410.336811] page dumped because: isolation failed [ 5410.336813] page->mem_cgroup:ffff8801cd662000 [ 5420.655030] memory offlining [mem 0x18b580000000-0x18b5ffffffff] failed Isolation has failed here because the page is not on LRU. Most probably because it was on the pcp LRU cache or it has been removed from the LRU already but it hasn't been freed yet. In both cases the page doesn't look non-migrable so retrying more makes sense. __offline_pages seems rather cluttered when it comes to the retry logic. We have 5 retries at maximum and a timeout. We could argue whether the timeout makes sense but failing just because of a race when somebody isoltes a page from LRU or puts it on a pcp LRU lists is just wrong. It only takes it to race with a process which unmaps some pages and remove them from the LRU list and we can fail the whole offline because of something that is a temporary condition and actually not harmful for the offline. Please note that unmovable pages should be already excluded during start_isolate_page_range. We could argue that has_unmovable_pages is racy and MIGRATE_MOVABLE check doesn't provide any hard guarantee either but kernel zones (aka < ZONE_MOVABLE) will very likely detect unmovable pages in most cases and movable zone shouldn't contain unmovable pages at all. Some of those pages might be pinned but not for ever because that would be a bug on its own. In any case the context is still interruptible and so the userspace can easily bail out when the operation takes too long. This is certainly better behavior than a hardcoded retry loop which is racy. Fix this by removing the max retry count and only rely on the timeout resp. interruption by a signal from the userspace. Also retry rather than fail when check_pages_isolated sees some !free pages because those could be a result of the race as well. Acked-by: Vlastimil Babka Signed-off-by: Michal Hocko --- mm/memory_hotplug.c | 40 ++++++++++------------------------------ 1 file changed, 10 insertions(+), 30 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index 459bbc182d10..c9dcbe6d2ac6 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1597,7 +1597,7 @@ static int __ref __offline_pages(unsigned long start_pfn, { unsigned long pfn, nr_pages, expire; long offlined_pages; - int ret, drain, retry_max, node; + int ret, node; unsigned long flags; unsigned long valid_start, valid_end; struct zone *zone; @@ -1634,43 +1634,25 @@ static int __ref __offline_pages(unsigned long start_pfn, pfn = start_pfn; expire = jiffies + timeout; - drain = 0; - retry_max = 5; repeat: /* start memory hot removal */ - ret = -EAGAIN; + ret = -EBUSY; if (time_after(jiffies, expire)) goto failed_removal; ret = -EINTR; if (signal_pending(current)) goto failed_removal; - ret = 0; - if (drain) { - lru_add_drain_all_cpuslocked(); - cond_resched(); - drain_all_pages(zone); - } + + cond_resched(); + lru_add_drain_all_cpuslocked(); + drain_all_pages(zone); pfn = scan_movable_pages(start_pfn, end_pfn); if (pfn) { /* We have movable pages */ ret = do_migrate_range(pfn, end_pfn); - if (!ret) { - drain = 1; - goto repeat; - } else { - if (ret < 0) - if (--retry_max == 0) - goto failed_removal; - yield(); - drain = 1; - goto repeat; - } + goto repeat; } - /* drain all zone's lru pagevec, this is asynchronous... */ - lru_add_drain_all_cpuslocked(); - yield(); - /* drain pcp pages, this is synchronous. */ - drain_all_pages(zone); + /* * dissolve free hugepages in the memory block before doing offlining * actually in order to make hugetlbfs's object counting consistent. @@ -1680,10 +1662,8 @@ static int __ref __offline_pages(unsigned long start_pfn, goto failed_removal; /* check again */ offlined_pages = check_pages_isolated(start_pfn, end_pfn); - if (offlined_pages < 0) { - ret = -EBUSY; - goto failed_removal; - } + if (offlined_pages < 0) + goto repeat; pr_info("Offlined Pages %ld\n", offlined_pages); /* Ok, all of our target is isolated. We cannot do rollback at this point. */ -- 2.14.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f70.google.com (mail-wm0-f70.google.com [74.125.82.70]) by kanga.kvack.org (Postfix) with ESMTP id CAF016B025F for ; Mon, 18 Sep 2017 03:10:16 -0400 (EDT) Received: by mail-wm0-f70.google.com with SMTP id r74so8565273wme.5 for ; Mon, 18 Sep 2017 00:10:16 -0700 (PDT) Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id v29sor2347447wra.58.2017.09.18.00.10.15 for (Google Transport Security); Mon, 18 Sep 2017 00:10:15 -0700 (PDT) From: Michal Hocko Subject: [PATCH 2/2] mm, memory_hotplug: remove timeout from __offline_memory Date: Mon, 18 Sep 2017 09:08:34 +0200 Message-Id: <20170918070834.13083-3-mhocko@kernel.org> In-Reply-To: <20170918070834.13083-1-mhocko@kernel.org> References: <20170918070834.13083-1-mhocko@kernel.org> Sender: owner-linux-mm@kvack.org List-ID: To: Andrew Morton Cc: KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML , Michal Hocko , Vlastimil Babka From: Michal Hocko We have a hardcoded 120s timeout after which the memory offline fails basically since the hot remove has been introduced. This is essentially a policy implemented in the kernel. Moreover there is no way to adjust the timeout and so we are sometimes facing memory offline failures if the system is under a heavy memory pressure or very intensive CPU workload on large machines. It is not very clear what purpose the timeout actually serves. The offline operation is interruptible by a signal so if userspace wants some timeout based termination this can be done trivially by sending a signal. If there is a strong usecase to do this from the kernel then we should do it properly and have a it tunable from the userspace with the timeout disabled by default along with the explanation who uses it and for what purporse. Acked-by: Vlastimil Babka Signed-off-by: Michal Hocko --- mm/memory_hotplug.c | 10 +++------- 1 file changed, 3 insertions(+), 7 deletions(-) diff --git a/mm/memory_hotplug.c b/mm/memory_hotplug.c index c9dcbe6d2ac6..b8a85c11360e 100644 --- a/mm/memory_hotplug.c +++ b/mm/memory_hotplug.c @@ -1593,9 +1593,9 @@ static void node_states_clear_node(int node, struct memory_notify *arg) } static int __ref __offline_pages(unsigned long start_pfn, - unsigned long end_pfn, unsigned long timeout) + unsigned long end_pfn) { - unsigned long pfn, nr_pages, expire; + unsigned long pfn, nr_pages; long offlined_pages; int ret, node; unsigned long flags; @@ -1633,12 +1633,8 @@ static int __ref __offline_pages(unsigned long start_pfn, goto failed_removal; pfn = start_pfn; - expire = jiffies + timeout; repeat: /* start memory hot removal */ - ret = -EBUSY; - if (time_after(jiffies, expire)) - goto failed_removal; ret = -EINTR; if (signal_pending(current)) goto failed_removal; @@ -1711,7 +1707,7 @@ static int __ref __offline_pages(unsigned long start_pfn, /* Must be protected by mem_hotplug_begin() or a device_lock */ int offline_pages(unsigned long start_pfn, unsigned long nr_pages) { - return __offline_pages(start_pfn, start_pfn + nr_pages, 120 * HZ); + return __offline_pages(start_pfn, start_pfn + nr_pages); } #endif /* CONFIG_MEMORY_HOTREMOVE */ -- 2.14.1 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f198.google.com (mail-pf0-f198.google.com [209.85.192.198]) by kanga.kvack.org (Postfix) with ESMTP id 0516C6B025E for ; Tue, 10 Oct 2017 08:05:16 -0400 (EDT) Received: by mail-pf0-f198.google.com with SMTP id d28so18328446pfe.2 for ; Tue, 10 Oct 2017 05:05:15 -0700 (PDT) Received: from ozlabs.org (ozlabs.org. [103.22.144.67]) by mx.google.com with ESMTPS id z5si8223770pgr.165.2017.10.10.05.05.13 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Oct 2017 05:05:13 -0700 (PDT) From: Michael Ellerman Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early In-Reply-To: <20170918070834.13083-2-mhocko@kernel.org> References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> Date: Tue, 10 Oct 2017 23:05:08 +1100 Message-ID: <87bmlfw6mj.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko , Andrew Morton Cc: KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML , Michal Hocko , Vlastimil Babka Michal Hocko writes: > From: Michal Hocko > > Memory offlining can fail just too eagerly under a heavy memory pressure. > > [ 5410.336792] page:ffffea22a646bd00 count:255 mapcount:252 mapping:ffff88ff926c9f38 index:0x3 > [ 5410.336809] flags: 0x9855fe40010048(uptodate|active|mappedtodisk) > [ 5410.336811] page dumped because: isolation failed > [ 5410.336813] page->mem_cgroup:ffff8801cd662000 > [ 5420.655030] memory offlining [mem 0x18b580000000-0x18b5ffffffff] failed > > Isolation has failed here because the page is not on LRU. Most probably > because it was on the pcp LRU cache or it has been removed from the LRU > already but it hasn't been freed yet. In both cases the page doesn't look > non-migrable so retrying more makes sense. This breaks offline for me. Prior to this commit: /sys/devices/system/memory/memory0# time echo 0 > online -bash: echo: write error: Device or resource busy real 0m0.001s user 0m0.000s sys 0m0.001s After: /sys/devices/system/memory/memory0# time echo 0 > online -bash: echo: write error: Device or resource busy real 2m0.009s user 0m0.000s sys 1m25.035s There's no way that block can be removed, it contains the kernel text, so it should instantly fail - which it used to. With commit 3aa2823fdf66 ("mm, memory_hotplug: remove timeout from __offline_memory") also applied, it appears to just get stuck forever, and I get lots of: [ 1232.112953] INFO: task kworker/3:0:4609 blocked for more than 120 seconds. [ 1232.113067] Not tainted 4.14.0-rc4-gcc6-next-20171009-g49827b9 #1 [ 1232.113183] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. [ 1232.113319] kworker/3:0 D11984 4609 2 0x00000800 [ 1232.113416] Workqueue: memcg_kmem_cache memcg_kmem_cache_create_func [ 1232.113531] Call Trace: [ 1232.113579] [c0000000fb2db7a0] [c0000000fb2db900] 0xc0000000fb2db900 (unreliable) [ 1232.113717] [c0000000fb2db970] [c00000000001c964] __switch_to+0x304/0x6e0 [ 1232.113840] [c0000000fb2dba10] [c000000000a408c0] __schedule+0x2e0/0xa80 [ 1232.113978] [c0000000fb2dbae0] [c000000000a410a8] schedule+0x48/0xc0 [ 1232.114113] [c0000000fb2dbb10] [c000000000a44d88] rwsem_down_read_failed+0x128/0x1b0 [ 1232.114269] [c0000000fb2dbb70] [c0000000001696a8] __percpu_down_read+0x108/0x110 [ 1232.114426] [c0000000fb2dbba0] [c00000000032e498] get_online_mems+0x68/0x80 [ 1232.115487] [c0000000fb2dbbc0] [c0000000002c82ec] memcg_create_kmem_cache+0x4c/0x190 [ 1232.115651] [c0000000fb2dbc60] [c0000000003483b8] memcg_kmem_cache_create_func+0x38/0xf0 [ 1232.115809] [c0000000fb2dbc90] [c000000000121594] process_one_work+0x2b4/0x590 [ 1232.115964] [c0000000fb2dbd20] [c000000000121908] worker_thread+0x98/0x5d0 [ 1232.116095] [c0000000fb2dbdc0] [c00000000012a134] kthread+0x164/0x1b0 [ 1232.116229] [c0000000fb2dbe30] [c00000000000bae0] ret_from_kernel_thread+0x5c/0x7c cheers -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f70.google.com (mail-wm0-f70.google.com [74.125.82.70]) by kanga.kvack.org (Postfix) with ESMTP id 1EE736B025E for ; Tue, 10 Oct 2017 08:27:31 -0400 (EDT) Received: by mail-wm0-f70.google.com with SMTP id b189so5493336wmd.9 for ; Tue, 10 Oct 2017 05:27:31 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id x2si9438412wrc.202.2017.10.10.05.27.29 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 10 Oct 2017 05:27:29 -0700 (PDT) Date: Tue, 10 Oct 2017 14:27:26 +0200 From: Michal Hocko Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early Message-ID: <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87bmlfw6mj.fsf@concordia.ellerman.id.au> Sender: owner-linux-mm@kvack.org List-ID: To: Michael Ellerman Cc: Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML , Vlastimil Babka On Tue 10-10-17 23:05:08, Michael Ellerman wrote: > Michal Hocko writes: > > > From: Michal Hocko > > > > Memory offlining can fail just too eagerly under a heavy memory pressure. > > > > [ 5410.336792] page:ffffea22a646bd00 count:255 mapcount:252 mapping:ffff88ff926c9f38 index:0x3 > > [ 5410.336809] flags: 0x9855fe40010048(uptodate|active|mappedtodisk) > > [ 5410.336811] page dumped because: isolation failed > > [ 5410.336813] page->mem_cgroup:ffff8801cd662000 > > [ 5420.655030] memory offlining [mem 0x18b580000000-0x18b5ffffffff] failed > > > > Isolation has failed here because the page is not on LRU. Most probably > > because it was on the pcp LRU cache or it has been removed from the LRU > > already but it hasn't been freed yet. In both cases the page doesn't look > > non-migrable so retrying more makes sense. > > This breaks offline for me. > > Prior to this commit: > /sys/devices/system/memory/memory0# time echo 0 > online > -bash: echo: write error: Device or resource busy > > real 0m0.001s > user 0m0.000s > sys 0m0.001s > > After: > /sys/devices/system/memory/memory0# time echo 0 > online > -bash: echo: write error: Device or resource busy > > real 2m0.009s > user 0m0.000s > sys 1m25.035s > > > There's no way that block can be removed, it contains the kernel text, > so it should instantly fail - which it used to. OK, that means that start_isolate_page_range should have failed but it hasn't for some reason. I strongly suspect has_unmovable_pages is doing something wrong. Is the kernel text marked somehow? E.g. PageReserved? In other words, does the diff below helps? diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3badcedf96a7..00d042052501 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7368,6 +7368,9 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, page = pfn_to_page(check); + if (PageReserved(page)) + return true; + /* * Hugepages are not in LRU lists, but they're movable. * We need not scan over tail pages bacause we don't > With commit 3aa2823fdf66 ("mm, memory_hotplug: remove timeout from > __offline_memory") also applied, it appears to just get stuck forever, > and I get lots of: > > [ 1232.112953] INFO: task kworker/3:0:4609 blocked for more than 120 seconds. > [ 1232.113067] Not tainted 4.14.0-rc4-gcc6-next-20171009-g49827b9 #1 > [ 1232.113183] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. > [ 1232.113319] kworker/3:0 D11984 4609 2 0x00000800 > [ 1232.113416] Workqueue: memcg_kmem_cache memcg_kmem_cache_create_func > [ 1232.113531] Call Trace: > [ 1232.113579] [c0000000fb2db7a0] [c0000000fb2db900] 0xc0000000fb2db900 (unreliable) > [ 1232.113717] [c0000000fb2db970] [c00000000001c964] __switch_to+0x304/0x6e0 > [ 1232.113840] [c0000000fb2dba10] [c000000000a408c0] __schedule+0x2e0/0xa80 > [ 1232.113978] [c0000000fb2dbae0] [c000000000a410a8] schedule+0x48/0xc0 > [ 1232.114113] [c0000000fb2dbb10] [c000000000a44d88] rwsem_down_read_failed+0x128/0x1b0 > [ 1232.114269] [c0000000fb2dbb70] [c0000000001696a8] __percpu_down_read+0x108/0x110 > [ 1232.114426] [c0000000fb2dbba0] [c00000000032e498] get_online_mems+0x68/0x80 > [ 1232.115487] [c0000000fb2dbbc0] [c0000000002c82ec] memcg_create_kmem_cache+0x4c/0x190 > [ 1232.115651] [c0000000fb2dbc60] [c0000000003483b8] memcg_kmem_cache_create_func+0x38/0xf0 > [ 1232.115809] [c0000000fb2dbc90] [c000000000121594] process_one_work+0x2b4/0x590 > [ 1232.115964] [c0000000fb2dbd20] [c000000000121908] worker_thread+0x98/0x5d0 > [ 1232.116095] [c0000000fb2dbdc0] [c00000000012a134] kthread+0x164/0x1b0 > [ 1232.116229] [c0000000fb2dbe30] [c00000000000bae0] ret_from_kernel_thread+0x5c/0x7c I do not see how this is related to the offline path. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg0-f72.google.com (mail-pg0-f72.google.com [74.125.83.72]) by kanga.kvack.org (Postfix) with ESMTP id 9C26E6B0253 for ; Tue, 10 Oct 2017 22:37:55 -0400 (EDT) Received: by mail-pg0-f72.google.com with SMTP id u23so999865pgo.7 for ; Tue, 10 Oct 2017 19:37:55 -0700 (PDT) Received: from ozlabs.org (ozlabs.org. [103.22.144.67]) by mx.google.com with ESMTPS id o15si2945041pgq.475.2017.10.10.19.37.53 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Oct 2017 19:37:54 -0700 (PDT) From: Michael Ellerman Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early In-Reply-To: <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> Date: Wed, 11 Oct 2017 13:37:50 +1100 Message-ID: <87infmz9xd.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML , Vlastimil Babka Michal Hocko writes: > On Tue 10-10-17 23:05:08, Michael Ellerman wrote: >> Michal Hocko writes: >> >> > From: Michal Hocko >> > >> > Memory offlining can fail just too eagerly under a heavy memory pressure. >> > >> > [ 5410.336792] page:ffffea22a646bd00 count:255 mapcount:252 mapping:ffff88ff926c9f38 index:0x3 >> > [ 5410.336809] flags: 0x9855fe40010048(uptodate|active|mappedtodisk) >> > [ 5410.336811] page dumped because: isolation failed >> > [ 5410.336813] page->mem_cgroup:ffff8801cd662000 >> > [ 5420.655030] memory offlining [mem 0x18b580000000-0x18b5ffffffff] failed >> > >> > Isolation has failed here because the page is not on LRU. Most probably >> > because it was on the pcp LRU cache or it has been removed from the LRU >> > already but it hasn't been freed yet. In both cases the page doesn't look >> > non-migrable so retrying more makes sense. >> >> This breaks offline for me. >> >> Prior to this commit: >> /sys/devices/system/memory/memory0# time echo 0 > online >> -bash: echo: write error: Device or resource busy >> >> real 0m0.001s >> user 0m0.000s >> sys 0m0.001s >> >> After: >> /sys/devices/system/memory/memory0# time echo 0 > online >> -bash: echo: write error: Device or resource busy >> >> real 2m0.009s >> user 0m0.000s >> sys 1m25.035s >> >> >> There's no way that block can be removed, it contains the kernel text, >> so it should instantly fail - which it used to. > > OK, that means that start_isolate_page_range should have failed but it > hasn't for some reason. I strongly suspect has_unmovable_pages is doing > something wrong. Is the kernel text marked somehow? E.g. PageReserved? I'm not sure how the text is marked, will have to dig into that. > In other words, does the diff below helps? No that doesn't help. > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 3badcedf96a7..00d042052501 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -7368,6 +7368,9 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, > > page = pfn_to_page(check); > > + if (PageReserved(page)) > + return true; > + > /* > * Hugepages are not in LRU lists, but they're movable. > * We need not scan over tail pages bacause we don't > > >> With commit 3aa2823fdf66 ("mm, memory_hotplug: remove timeout from >> __offline_memory") also applied, it appears to just get stuck forever, >> and I get lots of: >> >> [ 1232.112953] INFO: task kworker/3:0:4609 blocked for more than 120 seconds. >> [ 1232.113067] Not tainted 4.14.0-rc4-gcc6-next-20171009-g49827b9 #1 >> [ 1232.113183] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. >> [ 1232.113319] kworker/3:0 D11984 4609 2 0x00000800 >> [ 1232.113416] Workqueue: memcg_kmem_cache memcg_kmem_cache_create_func >> [ 1232.113531] Call Trace: >> [ 1232.113579] [c0000000fb2db7a0] [c0000000fb2db900] 0xc0000000fb2db900 (unreliable) >> [ 1232.113717] [c0000000fb2db970] [c00000000001c964] __switch_to+0x304/0x6e0 >> [ 1232.113840] [c0000000fb2dba10] [c000000000a408c0] __schedule+0x2e0/0xa80 >> [ 1232.113978] [c0000000fb2dbae0] [c000000000a410a8] schedule+0x48/0xc0 >> [ 1232.114113] [c0000000fb2dbb10] [c000000000a44d88] rwsem_down_read_failed+0x128/0x1b0 >> [ 1232.114269] [c0000000fb2dbb70] [c0000000001696a8] __percpu_down_read+0x108/0x110 >> [ 1232.114426] [c0000000fb2dbba0] [c00000000032e498] get_online_mems+0x68/0x80 >> [ 1232.115487] [c0000000fb2dbbc0] [c0000000002c82ec] memcg_create_kmem_cache+0x4c/0x190 >> [ 1232.115651] [c0000000fb2dbc60] [c0000000003483b8] memcg_kmem_cache_create_func+0x38/0xf0 >> [ 1232.115809] [c0000000fb2dbc90] [c000000000121594] process_one_work+0x2b4/0x590 >> [ 1232.115964] [c0000000fb2dbd20] [c000000000121908] worker_thread+0x98/0x5d0 >> [ 1232.116095] [c0000000fb2dbdc0] [c00000000012a134] kthread+0x164/0x1b0 >> [ 1232.116229] [c0000000fb2dbe30] [c00000000000bae0] ret_from_kernel_thread+0x5c/0x7c > > I do not see how this is related to the offline path. It's blocked doing get_online_mems(). So it's unrelated to the offline, but it can't proceed until the offline finishes, which it never does, IIUIC. cheers -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f197.google.com (mail-pf0-f197.google.com [209.85.192.197]) by kanga.kvack.org (Postfix) with ESMTP id 6F8F56B0253 for ; Wed, 11 Oct 2017 01:19:11 -0400 (EDT) Received: by mail-pf0-f197.google.com with SMTP id j64so2314964pfj.6 for ; Tue, 10 Oct 2017 22:19:11 -0700 (PDT) Received: from ozlabs.org (ozlabs.org. [2401:3900:2:1::2]) by mx.google.com with ESMTPS id h68si9324908pgc.508.2017.10.10.22.19.08 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 10 Oct 2017 22:19:09 -0700 (PDT) From: Michael Ellerman Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early In-Reply-To: <87infmz9xd.fsf@concordia.ellerman.id.au> References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> <87infmz9xd.fsf@concordia.ellerman.id.au> Date: Wed, 11 Oct 2017 16:19:05 +1100 Message-ID: <87a80yz2gm.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML , Vlastimil Babka Michael Ellerman writes: > Michal Hocko writes: >> On Tue 10-10-17 23:05:08, Michael Ellerman wrote: >>> Michal Hocko writes: >>> > From: Michal Hocko >>> > Memory offlining can fail just too eagerly under a heavy memory pressure. >>> > >>> > [ 5410.336792] page:ffffea22a646bd00 count:255 mapcount:252 mapping:ffff88ff926c9f38 index:0x3 >>> > [ 5410.336809] flags: 0x9855fe40010048(uptodate|active|mappedtodisk) >>> > [ 5410.336811] page dumped because: isolation failed >>> > [ 5410.336813] page->mem_cgroup:ffff8801cd662000 >>> > [ 5420.655030] memory offlining [mem 0x18b580000000-0x18b5ffffffff] failed >>> > >>> > Isolation has failed here because the page is not on LRU. Most probably >>> > because it was on the pcp LRU cache or it has been removed from the LRU >>> > already but it hasn't been freed yet. In both cases the page doesn't look >>> > non-migrable so retrying more makes sense. >>> >>> This breaks offline for me. >>> >>> Prior to this commit: >>> /sys/devices/system/memory/memory0# time echo 0 > online >>> -bash: echo: write error: Device or resource busy >>> >>> real 0m0.001s >>> user 0m0.000s >>> sys 0m0.001s >>> >>> After: >>> /sys/devices/system/memory/memory0# time echo 0 > online >>> -bash: echo: write error: Device or resource busy >>> >>> real 2m0.009s >>> user 0m0.000s >>> sys 1m25.035s >>> >>> There's no way that block can be removed, it contains the kernel text, >>> so it should instantly fail - which it used to. >> >> OK, that means that start_isolate_page_range should have failed but it >> hasn't for some reason. I strongly suspect has_unmovable_pages is doing >> something wrong. Is the kernel text marked somehow? E.g. PageReserved? > > I'm not sure how the text is marked, will have to dig into that. Yeah it's reserved: $ grep __init_begin /proc/kallsyms c000000000d70000 T __init_begin $ ./page-types -r -a 0x0,0xd7 flags page-count MB symbolic-flags long-symbolic-flags 0x0000000100000000 215 13 __________________________r_______________ reserved total 215 13 I added some printks, we're getting EBUSY from do_migrate_range(pfn, end_pfn). So we seem to just have an infinite loop: repeat: /* start memory hot removal */ ret = -EINTR; if (signal_pending(current)) goto failed_removal; cond_resched(); lru_add_drain_all_cpuslocked(); drain_all_pages(zone); pfn = scan_movable_pages(start_pfn, end_pfn); if (pfn) { /* We have movable pages */ ret = do_migrate_range(pfn, end_pfn); printk_ratelimited("memory-hotplug: migrate range returned %ld\n", ret); goto repeat; } eg: memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 __offline_pages: 354031 callbacks suppressed memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 memory-hotplug: migrate range returned -16 __offline_pages: 355794 callbacks suppressed cheers -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f199.google.com (mail-wr0-f199.google.com [209.85.128.199]) by kanga.kvack.org (Postfix) with ESMTP id 35A416B0253 for ; Wed, 11 Oct 2017 02:51:30 -0400 (EDT) Received: by mail-wr0-f199.google.com with SMTP id 22so349407wrb.9 for ; Tue, 10 Oct 2017 23:51:30 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id v21si11701643wra.442.2017.10.10.23.51.28 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 10 Oct 2017 23:51:28 -0700 (PDT) Date: Wed, 11 Oct 2017 08:51:23 +0200 From: Michal Hocko Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early Message-ID: <20171011065123.e7jvoftmtso3vcha@dhcp22.suse.cz> References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> <87infmz9xd.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87infmz9xd.fsf@concordia.ellerman.id.au> Sender: owner-linux-mm@kvack.org List-ID: To: Michael Ellerman Cc: Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML , Vlastimil Babka On Wed 11-10-17 13:37:50, Michael Ellerman wrote: > Michal Hocko writes: > > > On Tue 10-10-17 23:05:08, Michael Ellerman wrote: > >> Michal Hocko writes: > >> > >> > From: Michal Hocko > >> > > >> > Memory offlining can fail just too eagerly under a heavy memory pressure. > >> > > >> > [ 5410.336792] page:ffffea22a646bd00 count:255 mapcount:252 mapping:ffff88ff926c9f38 index:0x3 > >> > [ 5410.336809] flags: 0x9855fe40010048(uptodate|active|mappedtodisk) > >> > [ 5410.336811] page dumped because: isolation failed > >> > [ 5410.336813] page->mem_cgroup:ffff8801cd662000 > >> > [ 5420.655030] memory offlining [mem 0x18b580000000-0x18b5ffffffff] failed > >> > > >> > Isolation has failed here because the page is not on LRU. Most probably > >> > because it was on the pcp LRU cache or it has been removed from the LRU > >> > already but it hasn't been freed yet. In both cases the page doesn't look > >> > non-migrable so retrying more makes sense. > >> > >> This breaks offline for me. > >> > >> Prior to this commit: > >> /sys/devices/system/memory/memory0# time echo 0 > online > >> -bash: echo: write error: Device or resource busy > >> > >> real 0m0.001s > >> user 0m0.000s > >> sys 0m0.001s > >> > >> After: > >> /sys/devices/system/memory/memory0# time echo 0 > online > >> -bash: echo: write error: Device or resource busy > >> > >> real 2m0.009s > >> user 0m0.000s > >> sys 1m25.035s > >> > >> > >> There's no way that block can be removed, it contains the kernel text, > >> so it should instantly fail - which it used to. > > > > OK, that means that start_isolate_page_range should have failed but it > > hasn't for some reason. I strongly suspect has_unmovable_pages is doing > > something wrong. Is the kernel text marked somehow? E.g. PageReserved? > > I'm not sure how the text is marked, will have to dig into that. > > > In other words, does the diff below helps? > > No that doesn't help. This is really strange! As you write in other email the page is reserved. That means that some of the earlier checks if (zone_idx(zone) == ZONE_MOVABLE) return false; mt = get_pageblock_migratetype(page); if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) return false; has bailed out early. I would be quite surprised if the kernel text was sitting in the zone movable. The migrate type check is more fishy AFAICS. I can imagine that the kernel text can share the movable or CMA mt block. I am not really familiar with this function but it looks suspicious. So does it help to remove this check? --- diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3badcedf96a7..5b4d85ae445c 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7355,9 +7355,6 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, */ if (zone_idx(zone) == ZONE_MOVABLE) return false; - mt = get_pageblock_migratetype(page); - if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) - return false; pfn = page_to_pfn(page); for (found = 0, iter = 0; iter < pageblock_nr_pages; iter++) { @@ -7368,6 +7365,9 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, page = pfn_to_page(check); + if (PageReserved(page)) + return true; + /* * Hugepages are not in LRU lists, but they're movable. * We need not scan over tail pages bacause we don't -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f200.google.com (mail-pf0-f200.google.com [209.85.192.200]) by kanga.kvack.org (Postfix) with ESMTP id 32AE26B0260 for ; Wed, 11 Oct 2017 04:04:45 -0400 (EDT) Received: by mail-pf0-f200.google.com with SMTP id z80so3191725pff.1 for ; Wed, 11 Oct 2017 01:04:45 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id 36si10938776ple.401.2017.10.11.01.04.43 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 11 Oct 2017 01:04:43 -0700 (PDT) Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> <87infmz9xd.fsf@concordia.ellerman.id.au> <20171011065123.e7jvoftmtso3vcha@dhcp22.suse.cz> From: Vlastimil Babka Message-ID: Date: Wed, 11 Oct 2017 10:04:39 +0200 MIME-Version: 1.0 In-Reply-To: <20171011065123.e7jvoftmtso3vcha@dhcp22.suse.cz> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko , Michael Ellerman Cc: Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML On 10/11/2017 08:51 AM, Michal Hocko wrote: > On Wed 11-10-17 13:37:50, Michael Ellerman wrote: >> Michal Hocko writes: >> >>> On Tue 10-10-17 23:05:08, Michael Ellerman wrote: >>>> Michal Hocko writes: >>>> >>>>> From: Michal Hocko >>>>> >>>>> Memory offlining can fail just too eagerly under a heavy memory pressure. >>>>> >>>>> [ 5410.336792] page:ffffea22a646bd00 count:255 mapcount:252 mapping:ffff88ff926c9f38 index:0x3 >>>>> [ 5410.336809] flags: 0x9855fe40010048(uptodate|active|mappedtodisk) >>>>> [ 5410.336811] page dumped because: isolation failed >>>>> [ 5410.336813] page->mem_cgroup:ffff8801cd662000 >>>>> [ 5420.655030] memory offlining [mem 0x18b580000000-0x18b5ffffffff] failed >>>>> >>>>> Isolation has failed here because the page is not on LRU. Most probably >>>>> because it was on the pcp LRU cache or it has been removed from the LRU >>>>> already but it hasn't been freed yet. In both cases the page doesn't look >>>>> non-migrable so retrying more makes sense. >>>> >>>> This breaks offline for me. >>>> >>>> Prior to this commit: >>>> /sys/devices/system/memory/memory0# time echo 0 > online >>>> -bash: echo: write error: Device or resource busy Well, that means offline didn't actually work for that block even before this patch, right? Is it even a movable_node block? I guess not? >>>> real 0m0.001s >>>> user 0m0.000s >>>> sys 0m0.001s >>>> >>>> After: >>>> /sys/devices/system/memory/memory0# time echo 0 > online >>>> -bash: echo: write error: Device or resource busy >>>> >>>> real 2m0.009s >>>> user 0m0.000s >>>> sys 1m25.035s >>>> >>>> >>>> There's no way that block can be removed, it contains the kernel text, >>>> so it should instantly fail - which it used to. Ah, right. So your complain is really about that the failure is not instant anymore for blocks that can't be offlined. >>> OK, that means that start_isolate_page_range should have failed but it >>> hasn't for some reason. I strongly suspect has_unmovable_pages is doing >>> something wrong. Is the kernel text marked somehow? E.g. PageReserved? >> >> I'm not sure how the text is marked, will have to dig into that. >> >>> In other words, does the diff below helps? >> >> No that doesn't help. > > This is really strange! As you write in other email the page is > reserved. That means that some of the earlier checks > if (zone_idx(zone) == ZONE_MOVABLE) > return false; > mt = get_pageblock_migratetype(page); > if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) The MIGRATE_MOVABLE check is indeed bogus, because that doesn't guarantee there are no unmovable pages in the block (CMA block OTOH should be a guarantee). > return false; > has bailed out early. I would be quite surprised if the kernel text was > sitting in the zone movable. The migrate type check is more fishy > AFAICS. I can imagine that the kernel text can share the movable or CMA > mt block. I am not really familiar with this function but it looks > suspicious. So does it help to remove this check? > --- > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index 3badcedf96a7..5b4d85ae445c 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -7355,9 +7355,6 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, > */ > if (zone_idx(zone) == ZONE_MOVABLE) > return false; > - mt = get_pageblock_migratetype(page); > - if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) > - return false; > > pfn = page_to_pfn(page); > for (found = 0, iter = 0; iter < pageblock_nr_pages; iter++) { > @@ -7368,6 +7365,9 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, > > page = pfn_to_page(check); > > + if (PageReserved(page)) > + return true; > + > /* > * Hugepages are not in LRU lists, but they're movable. > * We need not scan over tail pages bacause we don't > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f71.google.com (mail-wm0-f71.google.com [74.125.82.71]) by kanga.kvack.org (Postfix) with ESMTP id 9988B6B025F for ; Wed, 11 Oct 2017 04:13:21 -0400 (EDT) Received: by mail-wm0-f71.google.com with SMTP id m72so497509wmc.0 for ; Wed, 11 Oct 2017 01:13:21 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id m28si10175521wmc.149.2017.10.11.01.13.20 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 11 Oct 2017 01:13:20 -0700 (PDT) Date: Wed, 11 Oct 2017 10:13:17 +0200 From: Michal Hocko Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early Message-ID: <20171011081317.b2cdnhjdyzgdo3up@dhcp22.suse.cz> References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> <87infmz9xd.fsf@concordia.ellerman.id.au> <20171011065123.e7jvoftmtso3vcha@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Vlastimil Babka Cc: Michael Ellerman , Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML On Wed 11-10-17 10:04:39, Vlastimil Babka wrote: > On 10/11/2017 08:51 AM, Michal Hocko wrote: [...] > > This is really strange! As you write in other email the page is > > reserved. That means that some of the earlier checks > > if (zone_idx(zone) == ZONE_MOVABLE) > > return false; > > mt = get_pageblock_migratetype(page); > > if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) > > The MIGRATE_MOVABLE check is indeed bogus, because that doesn't > guarantee there are no unmovable pages in the block (CMA block OTOH > should be a guarantee). OK, thanks for confirmation. I will remove the MIGRATE_MOVABLE check here. Do you think it is worth removing CMA check as well? This is merely an optimization AFAIU because we do not have to check the full pageblockworth of pfns. Anyway, let's way for Michael to confirm it really helps. If yes I will post a full patch and ask Andrew to add it as a prerequisite for this patch when sending to Linus to prevent the regression. -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f199.google.com (mail-pf0-f199.google.com [209.85.192.199]) by kanga.kvack.org (Postfix) with ESMTP id 9FB766B0260 for ; Wed, 11 Oct 2017 07:17:18 -0400 (EDT) Received: by mail-pf0-f199.google.com with SMTP id z80so4001739pff.1 for ; Wed, 11 Oct 2017 04:17:18 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id w187si10955349pfb.175.2017.10.11.04.17.17 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 11 Oct 2017 04:17:17 -0700 (PDT) Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> <87infmz9xd.fsf@concordia.ellerman.id.au> <20171011065123.e7jvoftmtso3vcha@dhcp22.suse.cz> <20171011081317.b2cdnhjdyzgdo3up@dhcp22.suse.cz> From: Vlastimil Babka Message-ID: Date: Wed, 11 Oct 2017 13:17:13 +0200 MIME-Version: 1.0 In-Reply-To: <20171011081317.b2cdnhjdyzgdo3up@dhcp22.suse.cz> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko Cc: Michael Ellerman , Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML On 10/11/2017 10:13 AM, Michal Hocko wrote: > On Wed 11-10-17 10:04:39, Vlastimil Babka wrote: >> On 10/11/2017 08:51 AM, Michal Hocko wrote: > [...] >>> This is really strange! As you write in other email the page is >>> reserved. That means that some of the earlier checks >>> if (zone_idx(zone) == ZONE_MOVABLE) >>> return false; >>> mt = get_pageblock_migratetype(page); >>> if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) >> >> The MIGRATE_MOVABLE check is indeed bogus, because that doesn't >> guarantee there are no unmovable pages in the block (CMA block OTOH >> should be a guarantee). > > OK, thanks for confirmation. I will remove the MIGRATE_MOVABLE check > here. Do you think it is worth removing CMA check as well? This is > merely an optimization AFAIU because we do not have to check the full > pageblockworth of pfns. Actually, we should remove the CMA part as well. It's true that MIGRATE_CMA does guarantee that the *buddy allocator* won't allocate non-MOVABLE pages from the pageblock. But if the memory got allocated as an actual CMA allocation (alloc_contig...) it will almost certainly not be movable. > Anyway, let's way for Michael to confirm it really helps. If yes I will > post a full patch and ask Andrew to add it as a prerequisite for this > patch when sending to Linus to prevent the regression. > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f71.google.com (mail-wm0-f71.google.com [74.125.82.71]) by kanga.kvack.org (Postfix) with ESMTP id 0159C6B0268 for ; Wed, 11 Oct 2017 07:24:18 -0400 (EDT) Received: by mail-wm0-f71.google.com with SMTP id 136so2328756wmu.3 for ; Wed, 11 Oct 2017 04:24:17 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id 129si9856664wme.241.2017.10.11.04.24.16 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 11 Oct 2017 04:24:16 -0700 (PDT) Date: Wed, 11 Oct 2017 13:24:15 +0200 From: Michal Hocko Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early Message-ID: <20171011112415.6hsudsrqhyuwjffx@dhcp22.suse.cz> References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> <87infmz9xd.fsf@concordia.ellerman.id.au> <20171011065123.e7jvoftmtso3vcha@dhcp22.suse.cz> <20171011081317.b2cdnhjdyzgdo3up@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Vlastimil Babka Cc: Michael Ellerman , Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML On Wed 11-10-17 13:17:13, Vlastimil Babka wrote: > On 10/11/2017 10:13 AM, Michal Hocko wrote: > > On Wed 11-10-17 10:04:39, Vlastimil Babka wrote: > >> On 10/11/2017 08:51 AM, Michal Hocko wrote: > > [...] > >>> This is really strange! As you write in other email the page is > >>> reserved. That means that some of the earlier checks > >>> if (zone_idx(zone) == ZONE_MOVABLE) > >>> return false; > >>> mt = get_pageblock_migratetype(page); > >>> if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) > >> > >> The MIGRATE_MOVABLE check is indeed bogus, because that doesn't > >> guarantee there are no unmovable pages in the block (CMA block OTOH > >> should be a guarantee). > > > > OK, thanks for confirmation. I will remove the MIGRATE_MOVABLE check > > here. Do you think it is worth removing CMA check as well? This is > > merely an optimization AFAIU because we do not have to check the full > > pageblockworth of pfns. > > Actually, we should remove the CMA part as well. It's true that > MIGRATE_CMA does guarantee that the *buddy allocator* won't allocate > non-MOVABLE pages from the pageblock. But if the memory got allocated as > an actual CMA allocation (alloc_contig...) it will almost certainly not > be movable. That was my suspicious. Thanks! -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f71.google.com (mail-wm0-f71.google.com [74.125.82.71]) by kanga.kvack.org (Postfix) with ESMTP id 27A866B025F for ; Wed, 11 Oct 2017 10:05:23 -0400 (EDT) Received: by mail-wm0-f71.google.com with SMTP id r202so2983132wmd.1 for ; Wed, 11 Oct 2017 07:05:23 -0700 (PDT) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com. [148.163.158.5]) by mx.google.com with ESMTPS id 65si268353edj.513.2017.10.11.07.05.21 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Wed, 11 Oct 2017 07:05:21 -0700 (PDT) Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.21/8.16.0.21) with SMTP id v9BDwhYI143649 for ; Wed, 11 Oct 2017 10:05:20 -0400 Received: from e06smtp11.uk.ibm.com (e06smtp11.uk.ibm.com [195.75.94.107]) by mx0b-001b2d01.pphosted.com with ESMTP id 2dhhqbvkhd-1 (version=TLSv1.2 cipher=AES256-SHA bits=256 verify=NOT) for ; Wed, 11 Oct 2017 10:05:17 -0400 Received: from localhost by e06smtp11.uk.ibm.com with IBM ESMTP SMTP Gateway: Authorized Use Only! Violators will be prosecuted for from ; Wed, 11 Oct 2017 15:05:15 +0100 Received: from d23av03.au.ibm.com (d23av03.au.ibm.com [9.190.234.97]) by b06cxnps4076.portsmouth.uk.ibm.com (8.14.9/8.14.9/NCO v10.0) with ESMTP id v9BE5CCP24576214 for ; Wed, 11 Oct 2017 14:05:13 GMT Received: from d23av03.au.ibm.com (localhost [127.0.0.1]) by d23av03.au.ibm.com (8.14.4/8.14.4/NCO v10.0 AVout) with ESMTP id v9BE54Ej010525 for ; Thu, 12 Oct 2017 01:05:04 +1100 Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> <87infmz9xd.fsf@concordia.ellerman.id.au> <87a80yz2gm.fsf@concordia.ellerman.id.au> From: Anshuman Khandual Date: Wed, 11 Oct 2017 19:35:04 +0530 MIME-Version: 1.0 In-Reply-To: <87a80yz2gm.fsf@concordia.ellerman.id.au> Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 7bit Message-Id: Sender: owner-linux-mm@kvack.org List-ID: To: Michael Ellerman , Michal Hocko Cc: Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML , Vlastimil Babka On 10/11/2017 10:49 AM, Michael Ellerman wrote: > Michael Ellerman writes: >> Michal Hocko writes: >>> On Tue 10-10-17 23:05:08, Michael Ellerman wrote: >>>> Michal Hocko writes: >>>>> From: Michal Hocko >>>>> Memory offlining can fail just too eagerly under a heavy memory pressure. >>>>> >>>>> [ 5410.336792] page:ffffea22a646bd00 count:255 mapcount:252 mapping:ffff88ff926c9f38 index:0x3 >>>>> [ 5410.336809] flags: 0x9855fe40010048(uptodate|active|mappedtodisk) >>>>> [ 5410.336811] page dumped because: isolation failed >>>>> [ 5410.336813] page->mem_cgroup:ffff8801cd662000 >>>>> [ 5420.655030] memory offlining [mem 0x18b580000000-0x18b5ffffffff] failed >>>>> >>>>> Isolation has failed here because the page is not on LRU. Most probably >>>>> because it was on the pcp LRU cache or it has been removed from the LRU >>>>> already but it hasn't been freed yet. In both cases the page doesn't look >>>>> non-migrable so retrying more makes sense. >>>> This breaks offline for me. >>>> >>>> Prior to this commit: >>>> /sys/devices/system/memory/memory0# time echo 0 > online >>>> -bash: echo: write error: Device or resource busy >>>> >>>> real 0m0.001s >>>> user 0m0.000s >>>> sys 0m0.001s >>>> >>>> After: >>>> /sys/devices/system/memory/memory0# time echo 0 > online >>>> -bash: echo: write error: Device or resource busy >>>> >>>> real 2m0.009s >>>> user 0m0.000s >>>> sys 1m25.035s >>>> >>>> There's no way that block can be removed, it contains the kernel text, >>>> so it should instantly fail - which it used to. >>> OK, that means that start_isolate_page_range should have failed but it >>> hasn't for some reason. I strongly suspect has_unmovable_pages is doing >>> something wrong. Is the kernel text marked somehow? E.g. PageReserved? >> I'm not sure how the text is marked, will have to dig into that. > Yeah it's reserved: > > $ grep __init_begin /proc/kallsyms > c000000000d70000 T __init_begin > $ ./page-types -r -a 0x0,0xd7 > flags page-count MB symbolic-flags long-symbolic-flags > 0x0000000100000000 215 13 __________________________r_______________ reserved > total 215 13 Hey Michael, What tool is this 'page-types' ? -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f69.google.com (mail-wm0-f69.google.com [74.125.82.69]) by kanga.kvack.org (Postfix) with ESMTP id 5DBBA6B0268 for ; Wed, 11 Oct 2017 10:16:20 -0400 (EDT) Received: by mail-wm0-f69.google.com with SMTP id r68so3009240wmr.6 for ; Wed, 11 Oct 2017 07:16:20 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id m28si10586245wmc.149.2017.10.11.07.16.19 for (version=TLS1 cipher=AES128-SHA bits=128/128); Wed, 11 Oct 2017 07:16:19 -0700 (PDT) Date: Wed, 11 Oct 2017 16:16:16 +0200 From: Michal Hocko Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early Message-ID: <20171011141616.7mo6g7qnd3df7ab5@dhcp22.suse.cz> References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> <87infmz9xd.fsf@concordia.ellerman.id.au> <87a80yz2gm.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Anshuman Khandual Cc: Michael Ellerman , Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML , Vlastimil Babka On Wed 11-10-17 19:35:04, Anshuman Khandual wrote: [...] > > $ grep __init_begin /proc/kallsyms > > c000000000d70000 T __init_begin > > $ ./page-types -r -a 0x0,0xd7 > > flags page-count MB symbolic-flags long-symbolic-flags > > 0x0000000100000000 215 13 __________________________r_______________ reserved > > total 215 13 > > Hey Michael, > > What tool is this 'page-types' ? tools/vm/page-types.c -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pf0-f200.google.com (mail-pf0-f200.google.com [209.85.192.200]) by kanga.kvack.org (Postfix) with ESMTP id 4CB706B0033 for ; Fri, 13 Oct 2017 07:42:57 -0400 (EDT) Received: by mail-pf0-f200.google.com with SMTP id e64so8259432pfk.0 for ; Fri, 13 Oct 2017 04:42:57 -0700 (PDT) Received: from ozlabs.org (ozlabs.org. [103.22.144.67]) by mx.google.com with ESMTPS id 69si494423pfh.27.2017.10.13.04.42.54 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Fri, 13 Oct 2017 04:42:54 -0700 (PDT) From: Michael Ellerman Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early In-Reply-To: References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> <87infmz9xd.fsf@concordia.ellerman.id.au> <20171011065123.e7jvoftmtso3vcha@dhcp22.suse.cz> Date: Fri, 13 Oct 2017 22:42:46 +1100 Message-ID: <87bmlbtgsp.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain Sender: owner-linux-mm@kvack.org List-ID: To: Vlastimil Babka , Michal Hocko Cc: Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML Vlastimil Babka writes: > On 10/11/2017 08:51 AM, Michal Hocko wrote: >> On Wed 11-10-17 13:37:50, Michael Ellerman wrote: >>> Michal Hocko writes: >>>> On Tue 10-10-17 23:05:08, Michael Ellerman wrote: >>>>> Michal Hocko writes: >>>>>> From: Michal Hocko >>>>>> >>>>>> Memory offlining can fail just too eagerly under a heavy memory pressure. ... >>>>> >>>>> This breaks offline for me. >>>>> >>>>> Prior to this commit: >>>>> /sys/devices/system/memory/memory0# time echo 0 > online >>>>> -bash: echo: write error: Device or resource busy > > Well, that means offline didn't actually work for that block even before > this patch, right? Is it even a movable_node block? I guess not? Correct. It should fail. >>>>> After: >>>>> /sys/devices/system/memory/memory0# time echo 0 > online >>>>> -bash: echo: write error: Device or resource busy >>>>> >>>>> real 2m0.009s >>>>> user 0m0.000s >>>>> sys 1m25.035s >>>>> >>>>> There's no way that block can be removed, it contains the kernel text, >>>>> so it should instantly fail - which it used to. > > Ah, right. So your complain is really about that the failure is not > instant anymore for blocks that can't be offlined. Yes. Previously it failed instantly, now it doesn't fail, and loops infinitely (once the 2 minute limit is removed). >> This is really strange! As you write in other email the page is >> reserved. That means that some of the earlier checks >> if (zone_idx(zone) == ZONE_MOVABLE) >> return false; >> mt = get_pageblock_migratetype(page); >> if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) > > The MIGRATE_MOVABLE check is indeed bogus, because that doesn't > guarantee there are no unmovable pages in the block (CMA block OTOH > should be a guarantee). OK I'll try that and get back to you. cheers >> diff --git a/mm/page_alloc.c b/mm/page_alloc.c >> index 3badcedf96a7..5b4d85ae445c 100644 >> --- a/mm/page_alloc.c >> +++ b/mm/page_alloc.c >> @@ -7355,9 +7355,6 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, >> */ >> if (zone_idx(zone) == ZONE_MOVABLE) >> return false; >> - mt = get_pageblock_migratetype(page); >> - if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) >> - return false; >> >> pfn = page_to_pfn(page); >> for (found = 0, iter = 0; iter < pageblock_nr_pages; iter++) { >> @@ -7368,6 +7365,9 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, >> >> page = pfn_to_page(check); >> >> + if (PageReserved(page)) >> + return true; >> + >> /* >> * Hugepages are not in LRU lists, but they're movable. >> * We need not scan over tail pages bacause we don't >> -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f71.google.com (mail-wm0-f71.google.com [74.125.82.71]) by kanga.kvack.org (Postfix) with ESMTP id 461066B0033 for ; Fri, 13 Oct 2017 07:58:39 -0400 (EDT) Received: by mail-wm0-f71.google.com with SMTP id b189so5891404wmd.9 for ; Fri, 13 Oct 2017 04:58:39 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id f134si826502wmd.165.2017.10.13.04.58.37 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 13 Oct 2017 04:58:37 -0700 (PDT) Date: Fri, 13 Oct 2017 13:58:35 +0200 From: Michal Hocko Subject: Re: [PATCH 1/2] mm, memory_hotplug: do not fail offlining too early Message-ID: <20171013115835.zaehapuucuzl2vlv@dhcp22.suse.cz> References: <20170918070834.13083-1-mhocko@kernel.org> <20170918070834.13083-2-mhocko@kernel.org> <87bmlfw6mj.fsf@concordia.ellerman.id.au> <20171010122726.6jrfdzkscwge6gez@dhcp22.suse.cz> <87infmz9xd.fsf@concordia.ellerman.id.au> <20171011065123.e7jvoftmtso3vcha@dhcp22.suse.cz> <87bmlbtgsp.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <87bmlbtgsp.fsf@concordia.ellerman.id.au> Sender: owner-linux-mm@kvack.org List-ID: To: Michael Ellerman Cc: Vlastimil Babka , Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , linux-mm@kvack.org, LKML On Fri 13-10-17 22:42:46, Michael Ellerman wrote: > Vlastimil Babka writes: > > On 10/11/2017 08:51 AM, Michal Hocko wrote: > >> On Wed 11-10-17 13:37:50, Michael Ellerman wrote: > >>> Michal Hocko writes: > >>>> On Tue 10-10-17 23:05:08, Michael Ellerman wrote: > >>>>> Michal Hocko writes: > >>>>>> From: Michal Hocko > >>>>>> > >>>>>> Memory offlining can fail just too eagerly under a heavy memory pressure. > ... > >>>>> > >>>>> This breaks offline for me. > >>>>> > >>>>> Prior to this commit: > >>>>> /sys/devices/system/memory/memory0# time echo 0 > online > >>>>> -bash: echo: write error: Device or resource busy > > > > Well, that means offline didn't actually work for that block even before > > this patch, right? Is it even a movable_node block? I guess not? > > Correct. It should fail. > > >>>>> After: > >>>>> /sys/devices/system/memory/memory0# time echo 0 > online > >>>>> -bash: echo: write error: Device or resource busy > >>>>> > >>>>> real 2m0.009s > >>>>> user 0m0.000s > >>>>> sys 1m25.035s > >>>>> > >>>>> There's no way that block can be removed, it contains the kernel text, > >>>>> so it should instantly fail - which it used to. > > > > Ah, right. So your complain is really about that the failure is not > > instant anymore for blocks that can't be offlined. > > Yes. Previously it failed instantly, now it doesn't fail, and loops > infinitely (once the 2 minute limit is removed). Yeah it failed only because the migration code retried few times and we bailed out which is wrong as well. I will send two patches as a reply to this email. > >> This is really strange! As you write in other email the page is > >> reserved. That means that some of the earlier checks > >> if (zone_idx(zone) == ZONE_MOVABLE) > >> return false; > >> mt = get_pageblock_migratetype(page); > >> if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) > > > > The MIGRATE_MOVABLE check is indeed bogus, because that doesn't > > guarantee there are no unmovable pages in the block (CMA block OTOH > > should be a guarantee). > > OK I'll try that and get back to you. Thanks! -- Michal Hocko SUSE Labs -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f72.google.com (mail-wm0-f72.google.com [74.125.82.72]) by kanga.kvack.org (Postfix) with ESMTP id C30706B0038 for ; Fri, 13 Oct 2017 08:00:24 -0400 (EDT) Received: by mail-wm0-f72.google.com with SMTP id k4so5874941wmc.20 for ; Fri, 13 Oct 2017 05:00:24 -0700 (PDT) Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id 38sor391509wrw.67.2017.10.13.05.00.23 for (Google Transport Security); Fri, 13 Oct 2017 05:00:23 -0700 (PDT) From: Michal Hocko Subject: [PATCH 1/2] mm: drop migrate type checks from has_unmovable_pages Date: Fri, 13 Oct 2017 14:00:12 +0200 Message-Id: <20171013120013.698-1-mhocko@kernel.org> In-Reply-To: <20171013115835.zaehapuucuzl2vlv@dhcp22.suse.cz> References: <20171013115835.zaehapuucuzl2vlv@dhcp22.suse.cz> Sender: owner-linux-mm@kvack.org List-ID: To: linux-mm@kvack.org Cc: Michael Ellerman , Vlastimil Babka , Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , LKML , Michal Hocko From: Michal Hocko Michael has noticed that the memory offline tries to migrate kernel code pages when doing echo 0 > /sys/devices/system/memory/memory0/online The current implementation will fail the operation after several failed page migration attempts but we shouldn't even attempt to migrate that memory and fail right away because this memory is clearly not migrateable. This will become a real problem when we drop the retry loop counter resp. timeout. The real problem is in has_unmovable_pages in fact. We should fail if there are any non migrateable pages in the area. In orther to guarantee that remove the migrate type checks because MIGRATE_MOVABLE is not guaranteed to contain only migrateable pages. It is merely a heuristic. Similarly MIGRATE_CMA does guarantee that the page allocator doesn't allocate any non-migrateable pages from the block but CMA allocations themselves are unlikely to migrateable. Therefore remove both checks. Reported-by: Michael Ellerman Signed-off-by: Michal Hocko --- mm/page_alloc.c | 3 --- 1 file changed, 3 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 3badcedf96a7..ad0294ab3e4f 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7355,9 +7355,6 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, */ if (zone_idx(zone) == ZONE_MOVABLE) return false; - mt = get_pageblock_migratetype(page); - if (mt == MIGRATE_MOVABLE || is_migrate_cma(mt)) - return false; pfn = page_to_pfn(page); for (found = 0, iter = 0; iter < pageblock_nr_pages; iter++) { -- 2.14.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f71.google.com (mail-wm0-f71.google.com [74.125.82.71]) by kanga.kvack.org (Postfix) with ESMTP id 643286B025E for ; Fri, 13 Oct 2017 08:00:25 -0400 (EDT) Received: by mail-wm0-f71.google.com with SMTP id u138so9384160wmu.2 for ; Fri, 13 Oct 2017 05:00:25 -0700 (PDT) Received: from mail-sor-f65.google.com (mail-sor-f65.google.com. [209.85.220.65]) by mx.google.com with SMTPS id n128sor315958wma.79.2017.10.13.05.00.24 for (Google Transport Security); Fri, 13 Oct 2017 05:00:24 -0700 (PDT) From: Michal Hocko Subject: [PATCH 2/2] mm, page_alloc: fail has_unmovable_pages when seeing reserved pages Date: Fri, 13 Oct 2017 14:00:13 +0200 Message-Id: <20171013120013.698-2-mhocko@kernel.org> In-Reply-To: <20171013120013.698-1-mhocko@kernel.org> References: <20171013115835.zaehapuucuzl2vlv@dhcp22.suse.cz> <20171013120013.698-1-mhocko@kernel.org> Sender: owner-linux-mm@kvack.org List-ID: To: linux-mm@kvack.org Cc: Michael Ellerman , Vlastimil Babka , Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , LKML , Michal Hocko From: Michal Hocko Reserved pages should be completely ignored by the core mm because they have a special meaning for their owners. has_unmovable_pages doesn't check those so we rely on other tests (reference count, or PageLRU) to fail on such pages. Althought this happens to work it is safer to simply check for those explicitly and do not rely on the owner of the page to abuse those fields for special purposes. Please note that this is more of a further fortification of the code rahter than a fix of an existing issue. Signed-off-by: Michal Hocko --- mm/page_alloc.c | 3 +++ 1 file changed, 3 insertions(+) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index ad0294ab3e4f..a8800b0a5619 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -7365,6 +7365,9 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, page = pfn_to_page(check); + if (PageReferenced(page)) + return true; + /* * Hugepages are not in LRU lists, but they're movable. * We need not scan over tail pages bacause we don't -- 2.14.2 -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f69.google.com (mail-wm0-f69.google.com [74.125.82.69]) by kanga.kvack.org (Postfix) with ESMTP id 5D6EC6B0038 for ; Fri, 13 Oct 2017 08:04:11 -0400 (EDT) Received: by mail-wm0-f69.google.com with SMTP id p186so5778604wmd.11 for ; Fri, 13 Oct 2017 05:04:11 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id e200si955466wmf.111.2017.10.13.05.04.09 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 13 Oct 2017 05:04:09 -0700 (PDT) Subject: Re: [PATCH 2/2] mm, page_alloc: fail has_unmovable_pages when seeing reserved pages References: <20171013115835.zaehapuucuzl2vlv@dhcp22.suse.cz> <20171013120013.698-1-mhocko@kernel.org> <20171013120013.698-2-mhocko@kernel.org> From: Vlastimil Babka Message-ID: Date: Fri, 13 Oct 2017 14:04:08 +0200 MIME-Version: 1.0 In-Reply-To: <20171013120013.698-2-mhocko@kernel.org> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 7bit Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko , linux-mm@kvack.org Cc: Michael Ellerman , Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , LKML , Michal Hocko On 10/13/2017 02:00 PM, Michal Hocko wrote: > From: Michal Hocko > > Reserved pages should be completely ignored by the core mm because they > have a special meaning for their owners. has_unmovable_pages doesn't > check those so we rely on other tests (reference count, or PageLRU) to > fail on such pages. Althought this happens to work it is safer to simply > check for those explicitly and do not rely on the owner of the page > to abuse those fields for special purposes. > > Please note that this is more of a further fortification of the code > rahter than a fix of an existing issue. > > Signed-off-by: Michal Hocko > --- > mm/page_alloc.c | 3 +++ > 1 file changed, 3 insertions(+) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index ad0294ab3e4f..a8800b0a5619 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -7365,6 +7365,9 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, > > page = pfn_to_page(check); > > + if (PageReferenced(page)) "Referenced" != "Reserved" > + return true; > + > /* > * Hugepages are not in LRU lists, but they're movable. > * We need not scan over tail pages bacause we don't > -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wm0-f69.google.com (mail-wm0-f69.google.com [74.125.82.69]) by kanga.kvack.org (Postfix) with ESMTP id 28ABF6B0038 for ; Fri, 13 Oct 2017 08:07:58 -0400 (EDT) Received: by mail-wm0-f69.google.com with SMTP id r68so9393031wmr.6 for ; Fri, 13 Oct 2017 05:07:58 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id b191si840502wma.214.2017.10.13.05.07.56 for (version=TLS1 cipher=AES128-SHA bits=128/128); Fri, 13 Oct 2017 05:07:57 -0700 (PDT) Date: Fri, 13 Oct 2017 14:07:56 +0200 From: Michal Hocko Subject: Re: [PATCH 2/2] mm, page_alloc: fail has_unmovable_pages when seeing reserved pages Message-ID: <20171013120756.jeopthigbmm3c7bl@dhcp22.suse.cz> References: <20171013115835.zaehapuucuzl2vlv@dhcp22.suse.cz> <20171013120013.698-1-mhocko@kernel.org> <20171013120013.698-2-mhocko@kernel.org> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: Sender: owner-linux-mm@kvack.org List-ID: To: Vlastimil Babka Cc: linux-mm@kvack.org, Michael Ellerman , Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , LKML On Fri 13-10-17 14:04:08, Vlastimil Babka wrote: > On 10/13/2017 02:00 PM, Michal Hocko wrote: > > From: Michal Hocko > > > > Reserved pages should be completely ignored by the core mm because they > > have a special meaning for their owners. has_unmovable_pages doesn't > > check those so we rely on other tests (reference count, or PageLRU) to > > fail on such pages. Althought this happens to work it is safer to simply > > check for those explicitly and do not rely on the owner of the page > > to abuse those fields for special purposes. > > > > Please note that this is more of a further fortification of the code > > rahter than a fix of an existing issue. > > > > Signed-off-by: Michal Hocko > > --- > > mm/page_alloc.c | 3 +++ > > 1 file changed, 3 insertions(+) > > > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > > index ad0294ab3e4f..a8800b0a5619 100644 > > --- a/mm/page_alloc.c > > +++ b/mm/page_alloc.c > > @@ -7365,6 +7365,9 @@ bool has_unmovable_pages(struct zone *zone, struct page *page, int count, > > > > page = pfn_to_page(check); > > > > + if (PageReferenced(page)) > > "Referenced" != "Reserved" Dohh, you are right of course. I blame auto-completion ;) but I am lame in fact... --- From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-pg0-f72.google.com (mail-pg0-f72.google.com [74.125.83.72]) by kanga.kvack.org (Postfix) with ESMTP id 350356B0069 for ; Tue, 17 Oct 2017 07:41:13 -0400 (EDT) Received: by mail-pg0-f72.google.com with SMTP id l24so1305713pgu.17 for ; Tue, 17 Oct 2017 04:41:13 -0700 (PDT) Received: from ozlabs.org (ozlabs.org. [2401:3900:2:1::2]) by mx.google.com with ESMTPS id h1si6062072pln.150.2017.10.17.04.41.11 for (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); Tue, 17 Oct 2017 04:41:12 -0700 (PDT) From: Michael Ellerman Subject: Re: [PATCH 1/2] mm: drop migrate type checks from has_unmovable_pages In-Reply-To: <20171013120013.698-1-mhocko@kernel.org> References: <20171013115835.zaehapuucuzl2vlv@dhcp22.suse.cz> <20171013120013.698-1-mhocko@kernel.org> Date: Tue, 17 Oct 2017 22:41:08 +1100 Message-ID: <871sm2j92j.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain Sender: owner-linux-mm@kvack.org List-ID: To: Michal Hocko , linux-mm@kvack.org Cc: Vlastimil Babka , Andrew Morton , KAMEZAWA Hiroyuki , Reza Arbab , Yasuaki Ishimatsu , qiuxishi@huawei.com, Igor Mammedov , Vitaly Kuznetsov , LKML , Michal Hocko Michal Hocko writes: > From: Michal Hocko > > Michael has noticed that the memory offline tries to migrate kernel code > pages when doing > echo 0 > /sys/devices/system/memory/memory0/online > > The current implementation will fail the operation after several failed > page migration attempts but we shouldn't even attempt to migrate > that memory and fail right away because this memory is clearly not > migrateable. This will become a real problem when we drop the retry loop > counter resp. timeout. > > The real problem is in has_unmovable_pages in fact. We should fail if > there are any non migrateable pages in the area. In orther to guarantee > that remove the migrate type checks because MIGRATE_MOVABLE is not > guaranteed to contain only migrateable pages. It is merely a heuristic. > Similarly MIGRATE_CMA does guarantee that the page allocator doesn't > allocate any non-migrateable pages from the block but CMA allocations > themselves are unlikely to migrateable. Therefore remove both checks. > > Reported-by: Michael Ellerman > Signed-off-by: Michal Hocko Thanks, that works for me. Tested-by: Michael Ellerman cheers -- To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to majordomo@kvack.org. For more info on Linux MM, see: http://www.linux-mm.org/ . Don't email: email@kvack.org From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-wr0-f199.google.com (mail-wr0-f199.google.com [209.85.128.199]) by kanga.kvack.org (Postfix) with ESMTP id 76F206B0038 for ; Tue, 17 Oct 2017 08:03:21 -0400 (EDT) Received: by mail-wr0-f199.google.com with SMTP id y39so708089wrd.17 for ; Tue, 17 Oct 2017 05:03:21 -0700 (PDT) Received: from mx2.suse.de (mx2.suse.de. [195.135.220.15]) by mx.google.com with ESMTPS id o17si6647071wme.34.2017.10.17.05.03.20 for (version=TLS1 cipher=AES128-SHA bits=128/128); Tue, 17 Oct 2017 05:03:20 -0700 (PDT) Date: Tue, 17 Oct 2017 14:03:18 +0200 From: Michal Hocko Subject: Re: [PATCH 1/2] mm: drop migrate type checks from has_unmovable_pages Message-ID: <20171017120318.rers7zga3cnirt4i@dhcp22.suse.cz> References: <20171013115835.zaehapuucuzl2vlv@dhcp22.suse.cz> <20171013120013.698-1-mhocko@kernel.org> <871sm2j92j.fsf@concordia.ellerman.id.au> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <871sm2j92j.fsf@concordia.ellerman.id.au> Sender: owner-linux-mm@kvack.org List-ID: To: Michael Ellerman